Ylem documentation
  • 🗒️General information
    • Introduction to Ylem
    • Quick start guide
    • Release notes
  • 🔬Open-source edition
    • Installation
    • Usage of Apache Kafka
    • Task processing architecture
    • Configuring integrations with .env variables
  • 💡Integrations
    • Connecting an integration
    • Library of integrations
      • Amazon Redshift
      • Apache Kafka
      • APIs
      • Atlassian Jira
      • AWS Lambda
      • AWS RDS
      • AWS S3
      • ClickHouse
      • ElasticSearch
      • E-mail
      • Google Big Query
      • Google Cloud SQL
      • Google Pub/Sub
      • Google Sheets
      • Immuta
      • Incident.io
      • Jenkins
      • Hubspot
      • Microsoft Azure SQL
      • MySQL
      • OpenAI ChatGPT
      • Opsgenie
      • PostgreSQL
      • PlanetScale
      • RabbitMQ
      • Salesforce
      • Slack
      • Snowflake
      • Tableau
      • Twilio. SMS
      • WhatsApp (through Twilio)
    • Initial demo data source
  • 🚡Pipelines
    • Pipeline management
    • Tasks
      • Aggregator
      • API Call
      • Code
      • Condition
      • External trigger
      • Filter
      • For each
      • GPT
      • Merge
      • Notification
      • Query
      • Pipeline runner
      • Processor
      • Transformer
    • Running and scheduling pipelines
    • Library of templates
    • Environment variables
    • Mathematical functions and operations
    • Formatting of messages
  • 📈Statistics and profiling
    • Statistics of runs
    • Slow tasks
  • 📊Metrics
    • Metric management
    • Using previous values of a metric
  • 💼Use cases, patterns, templates, examples
    • Use cases
    • Messaging patterns
      • Datatype Channel
      • Message Dispatcher
      • Messaging Bridge
      • Message Bus
      • Message Filter
      • Message Router
      • Point-to-Point Channel
      • Publish-Subscribe Channel
      • Pull-Push
    • Functional use cases
      • Streaming from Apache Kafka and messaging queues
      • Streaming from APIs
      • Streaming from databases
      • Data orchestration, transformation and processing
      • Usage of Python and Pandas
      • KPI Monitoring
      • OKRs and custom metrics
      • Data Issues & Incidents
      • Reporting
      • Other functional use cases
    • Industry-specific use cases
      • Finance and Payments
      • E-commerce & Logistics
      • Customer Success
      • Security, Risk, and Anti-Fraud
      • Anti-Money Laundering (AML)
  • 🔌API
    • OAuth clients
    • API Reference
  • 👁️‍🗨️Other resources
    • FAQ
    • Our blog on Medium
Powered by GitBook
On this page
  • Mathematical functions
  • Array functions
  • Single number functions
  • Other functions
  • Metric functions
  • Arithmetical and logical operations
  • Arithmetic
  • Comparison
  • Boolean operators
  • Bitmasking
  • Regular expression comparison
  • Formatting of dates

Was this helpful?

Edit on GitHub
  1. 🚡Pipelines

Mathematical functions and operations

PreviousEnvironment variablesNextFormatting of messages

Last updated 11 months ago

Was this helpful?

Several tasks such as Aggregator, Condition, and Notification are empowered by an extended mathematical editor that supports a powerful set of mathematical functions and operations.

Mathematical functions

Most probably you are already familiar with them and used these ones in SQL databases, Microsoft Excel, or Google Spreadsheets.

Array functions

Formula
Meaning
Example

AVG(field_name)

Average value in the input data set

AVG(amount)

SUM(field_name)

A sum of values in the input data set

SUM(amount)

COUNT()

Number of rows in the input data set

FIRST(field_name)

First value in the input data set

LAST(field_name)

Last value in the input data set

MAX(field_name)

Max value in the input data set

MAX(amount)

MIN(field_name)

Min value in the input data set

MIN(amount)

Single number functions

Formula
Meaning
Example

ROUND(field_name, precision, "floor|ceil")

Round a number up or down with a certain precision

ROUND(tax, 2, "floor")

ABS(field_name)

The absolute value of a number

NEG(field_name)

Inverts the sign of a number. From "-" to "+" and other way around

SIGN(field_name)

Returns:

-1 if d <  0
 0 if d == 0
+1 if d >  0

STRING(field_name)

Converts number to string

INT(field_name)

Returns an integer part of a decimal number

Other functions

Formula
Meaning
Example

ENV()

Use environment variable in a format of ENV_variable_name

ENV_REVENUE_THRESHOLD

NOW()

Current timestamp

INPUT()

The entire input data set as JSON

Metric functions

Also, there's an additional set of functions available for Metrics only:

Formula
Meaning
Example

METRIC_AVG("period", duration)

This function returns an average value of this metric within a selected period of time

METRIC_AVG("day", 5) or METRIC_AVG("month", 3) etc.

METRIC_QUANTILE(level, "period", duration)

This function returns a metric quantile within a selected period of time

METRIC_QUANTILE(0.25, "day", 7)

METRIC_MEDIAN("period", duration)

This function returns a median metric value within a selected period of time.

Alias for METRIC_QUANTILE(0.5, "period", duration)

METRIC_MEDIAN("day", 5) or METRIC_MEDIAN("month", 3) etc.

Arithmetical and logical operations

The following list is supported:

Arithmetic

+ - / * % **

Comparison

== === != !== > < >= <= >== <==

Boolean operators

|| && !

Bitmasking

| & ^ << >>

Regular expression comparison

The following two operations are supported for searching in strings with regular expressions

=~ // means a regular expression match is found !~ // means a regular expression match is not found

For example, the following string placed in the Condition or Aggregator will return true

"Mountaineering is the best sport in the world. Or engineering?" =~ "[(Engin|Mountain)]eering\\s"

Supported syntax for regular expressions: https://pkg.go.dev/regexp/syntax.

Formatting of dates

Comparison operations can also be used for comparing dates. The following formats are supported:

  • time.UnixDate,

  • time.RubyDate,

  • time.Kitchen,

  • time.RFC3339,

  • time.RFC3339Nano,

  • "2006-01-02", // RFC 3339

  • "2006-01-02 15:04", // RFC 3339 with minutes

  • "2006-01-02 15:04:05", // RFC 3339 with seconds

  • "2006-01-02 15:04:05-07:00", // RFC 3339 with seconds and timezone

  • "2006-01-02T15Z", // ISO8601 with hour without UTC offset

  • "2006-01-02T15:04Z", // ISO8601 with minutes without UTC offset

  • "2006-01-02T15Z0700", // ISO8601 with hour

  • "2006-01-02T15:04:05Z", // ISO8601 with seconds without UTC offset

  • "2006-01-02T15:04Z0700", // ISO8601 with minutes

  • "2006-01-02T15:04:05Z0700", // ISO8601 with seconds

  • "2006-01-02T15:04:05.999999999Z", // ISO8601 with nanoseconds without UTC offset

  • "2006-01-02T15:04:05.999999999Z0700", // ISO8601 with nanoseconds