Ylem documentation
  • 🗒️General information
    • Introduction to Ylem
    • Quick start guide
    • Release notes
  • 🔬Open-source edition
    • Installation
    • Usage of Apache Kafka
    • Task processing architecture
    • Configuring integrations with .env variables
  • 💡Integrations
    • Connecting an integration
    • Library of integrations
      • Amazon Redshift
      • Apache Kafka
      • APIs
      • Atlassian Jira
      • AWS Lambda
      • AWS RDS
      • AWS S3
      • ClickHouse
      • ElasticSearch
      • E-mail
      • Google Big Query
      • Google Cloud SQL
      • Google Pub/Sub
      • Google Sheets
      • Immuta
      • Incident.io
      • Jenkins
      • Hubspot
      • Microsoft Azure SQL
      • MySQL
      • OpenAI ChatGPT
      • Opsgenie
      • PostgreSQL
      • PlanetScale
      • RabbitMQ
      • Salesforce
      • Slack
      • Snowflake
      • Tableau
      • Twilio. SMS
      • WhatsApp (through Twilio)
    • Initial demo data source
  • 🚡Pipelines
    • Pipeline management
    • Tasks
      • Aggregator
      • API Call
      • Code
      • Condition
      • External trigger
      • Filter
      • For each
      • GPT
      • Merge
      • Notification
      • Query
      • Pipeline runner
      • Processor
      • Transformer
    • Running and scheduling pipelines
    • Library of templates
    • Environment variables
    • Mathematical functions and operations
    • Formatting of messages
  • 📈Statistics and profiling
    • Statistics of runs
    • Slow tasks
  • 📊Metrics
    • Metric management
    • Using previous values of a metric
  • 💼Use cases, patterns, templates, examples
    • Use cases
    • Messaging patterns
      • Datatype Channel
      • Message Dispatcher
      • Messaging Bridge
      • Message Bus
      • Message Filter
      • Message Router
      • Point-to-Point Channel
      • Publish-Subscribe Channel
      • Pull-Push
    • Functional use cases
      • Streaming from Apache Kafka and messaging queues
      • Streaming from APIs
      • Streaming from databases
      • Data orchestration, transformation and processing
      • Usage of Python and Pandas
      • KPI Monitoring
      • OKRs and custom metrics
      • Data Issues & Incidents
      • Reporting
      • Other functional use cases
    • Industry-specific use cases
      • Finance and Payments
      • E-commerce & Logistics
      • Customer Success
      • Security, Risk, and Anti-Fraud
      • Anti-Money Laundering (AML)
  • 🔌API
    • OAuth clients
    • API Reference
  • 👁️‍🗨️Other resources
    • FAQ
    • Our blog on Medium
Powered by GitBook
On this page
  • Usage
  • Configuration

Was this helpful?

Edit on GitHub
  1. Pipelines
  2. Tasks

GPT

PreviousFor eachNextMerge

Last updated 10 months ago

Was this helpful?

This task is currently only available in the on-premise and open-source editions but not in the cloud version

"GPT" is a task that allows you to enrich or transform your data with AI using OpenAI's models.

Usage

For example, you have such a pipeline consisting of two tasks:

The first task returns a JSON with the list of items, containing an item ID and amount:

[
    {
        "id": 1,
        "amount": 33
    },
    {
        "id": 2,
        "amount": 150
    },
    {
        "id": 3,
        "amount": 4000
    }
]

Now you can ask chat GPT to transform this data. For example, multiply all amounts by 3:

And this is what the task returns:

Of course, such a simple multiplication can be done by other tasks without the use of AI, but that's just an example and you can use ChatGPT for much more. Some other ideas:

  • Generate missing postal codes for the list of addresses

  • Clean up organization names in the list

  • Convert amounts from one currency to another

  • Rewrite texts on the fly

  • And so on and so on

For example, if you have a list of addresses missing postal codes like this:

[
  {
    "country": "United States",
    "state": "Michigan",
    "city": "Detroit",
    "address": "2645 Woodward Ave"
  },
  {
    "country": "Canada",
    "province": "Alberta",
    "city": "Edmonton",
    "address": "104 Ave NW"
  },
  {
    "country": "United States",
    "state": "Florida",
    "city": "Sunrise",
    "address": "1 Panther Pkwy"
  }
]

With the workflow as above and that message to chat GPT:

You will get the perfect result;

Configuration

To be able to use this task, you need to configure your API Secret Key and choose a model you want to use:

TASK_RUNNER_OPENAI_GPT_KEY=%%YOUR_KEY_IS_HERE%%
TASK_RUNNER_OPENAI_MODEL=gpt-4o-mini

The following two variables need to be placed in the file ./processor/taskrunner/.env.

Read more about and in the official OpenAI reference guide.

🚡
API Keys
available models
⚠️
chat completion