Ylem documentation
  • 🗒️General information
    • Introduction to Ylem
    • Quick start guide
    • Release notes
  • 🔬Open-source edition
    • Installation
    • Usage of Apache Kafka
    • Task processing architecture
    • Configuring integrations with .env variables
  • 💡Integrations
    • Connecting an integration
    • Library of integrations
      • Amazon Redshift
      • Apache Kafka
      • APIs
      • Atlassian Jira
      • AWS Lambda
      • AWS RDS
      • AWS S3
      • ClickHouse
      • ElasticSearch
      • E-mail
      • Google Big Query
      • Google Cloud SQL
      • Google Pub/Sub
      • Google Sheets
      • Immuta
      • Incident.io
      • Jenkins
      • Hubspot
      • Microsoft Azure SQL
      • MySQL
      • OpenAI ChatGPT
      • Opsgenie
      • PostgreSQL
      • PlanetScale
      • RabbitMQ
      • Salesforce
      • Slack
      • Snowflake
      • Tableau
      • Twilio. SMS
      • WhatsApp (through Twilio)
    • Initial demo data source
  • 🚡Pipelines
    • Pipeline management
    • Tasks
      • Aggregator
      • API Call
      • Code
      • Condition
      • External trigger
      • Filter
      • For each
      • GPT
      • Merge
      • Notification
      • Query
      • Pipeline runner
      • Processor
      • Transformer
    • Running and scheduling pipelines
    • Library of templates
    • Environment variables
    • Mathematical functions and operations
    • Formatting of messages
  • 📈Statistics and profiling
    • Statistics of runs
    • Slow tasks
  • 📊Metrics
    • Metric management
    • Using previous values of a metric
  • 💼Use cases, patterns, templates, examples
    • Use cases
    • Messaging patterns
      • Datatype Channel
      • Message Dispatcher
      • Messaging Bridge
      • Message Bus
      • Message Filter
      • Message Router
      • Point-to-Point Channel
      • Publish-Subscribe Channel
      • Pull-Push
    • Functional use cases
      • Streaming from Apache Kafka and messaging queues
      • Streaming from APIs
      • Streaming from databases
      • Data orchestration, transformation and processing
      • Usage of Python and Pandas
      • KPI Monitoring
      • OKRs and custom metrics
      • Data Issues & Incidents
      • Reporting
      • Other functional use cases
    • Industry-specific use cases
      • Finance and Payments
      • E-commerce & Logistics
      • Customer Success
      • Security, Risk, and Anti-Fraud
      • Anti-Money Laundering (AML)
  • 🔌API
    • OAuth clients
    • API Reference
  • 👁️‍🗨️Other resources
    • FAQ
    • Our blog on Medium
Powered by GitBook
On this page
  • Filtering
  • Aggregation
  • Transformation
  • Use Pandas
  • Limitations
  • Currently supported Python-modules

Was this helpful?

Edit on GitHub
  1. Pipelines
  2. Tasks

Code

PreviousAPI CallNextCondition

Last updated 8 months ago

Was this helpful?

Ylem is a no-code solution but as with every modern no-code solution, you still can write code if you want to:-)

From time to time you need to resort to complex data transformations. Say a single filter is not enough, so you can write custom Python 3.9 code with a full range of features over your data work.

This is what a "Code" task is for. It has an input as JSON, with which you can do whatever you want.

Let's say that we have the following dataset and see what we do with it.

Filtering

Let's filter out rows of a type "invoice" that have an "amount" less than 2000:

result = []

for i in input:
  if i["amount"] is not None and i["amount"] < 2000 and i["type"] == "invoice":
    result.append(i)

input = result

The result of the workflow above

Aggregation

Let's calculate the total amount:

total = 0

for i in input:
  if i["amount"] is not None:
    total += i["amount"]

input = {
  "total": total
}

Transformation

total = 0

for i in input:
  if i["amount"] is not None:
    total += i["amount"]

input = {
  "items": input,
  "total": total
}

Use Pandas

Limitations

Currently supported Python-modules

  • json

  • re

  • time

  • datetime

  • pandas

The simplest example can be achieved by using the "" block. But what if we need more complex filtering?

Yes, the "Code" supports framework. This is how you can work with it and the input dataset:

Our Python processor is based on , therefore the "Code" task supports many standard Python libraries but not all of them. Let us know if you want to use something that is missing and we'll be happy to add it.

🚡
Filter
Pandas
RestrictedPython