Ylem documentation
  • 🗒️General information
    • Introduction to Ylem
    • Quick start guide
    • Release notes
  • 🔬Open-source edition
    • Installation
    • Usage of Apache Kafka
    • Task processing architecture
    • Configuring integrations with .env variables
  • 💡Integrations
    • Connecting an integration
    • Library of integrations
      • Amazon Redshift
      • Apache Kafka
      • APIs
      • Atlassian Jira
      • AWS Lambda
      • AWS RDS
      • AWS S3
      • ClickHouse
      • ElasticSearch
      • E-mail
      • Google Big Query
      • Google Cloud SQL
      • Google Pub/Sub
      • Google Sheets
      • Immuta
      • Incident.io
      • Jenkins
      • Hubspot
      • Microsoft Azure SQL
      • MySQL
      • OpenAI ChatGPT
      • Opsgenie
      • PostgreSQL
      • PlanetScale
      • RabbitMQ
      • Salesforce
      • Slack
      • Snowflake
      • Tableau
      • Twilio. SMS
      • WhatsApp (through Twilio)
    • Initial demo data source
  • 🚡Pipelines
    • Pipeline management
    • Tasks
      • Aggregator
      • API Call
      • Code
      • Condition
      • External trigger
      • Filter
      • For each
      • GPT
      • Merge
      • Notification
      • Query
      • Pipeline runner
      • Processor
      • Transformer
    • Running and scheduling pipelines
    • Library of templates
    • Environment variables
    • Mathematical functions and operations
    • Formatting of messages
  • 📈Statistics and profiling
    • Statistics of runs
    • Slow tasks
  • 📊Metrics
    • Metric management
    • Using previous values of a metric
  • 💼Use cases, patterns, templates, examples
    • Use cases
    • Messaging patterns
      • Datatype Channel
      • Message Dispatcher
      • Messaging Bridge
      • Message Bus
      • Message Filter
      • Message Router
      • Point-to-Point Channel
      • Publish-Subscribe Channel
      • Pull-Push
    • Functional use cases
      • Streaming from Apache Kafka and messaging queues
      • Streaming from APIs
      • Streaming from databases
      • Data orchestration, transformation and processing
      • Usage of Python and Pandas
      • KPI Monitoring
      • OKRs and custom metrics
      • Data Issues & Incidents
      • Reporting
      • Other functional use cases
    • Industry-specific use cases
      • Finance and Payments
      • E-commerce & Logistics
      • Customer Success
      • Security, Risk, and Anti-Fraud
      • Anti-Money Laundering (AML)
  • 🔌API
    • OAuth clients
    • API Reference
  • 👁️‍🗨️Other resources
    • FAQ
    • Our blog on Medium
Powered by GitBook
On this page
  • Triggering pipeline from outside of Ylem
  • Streaming from Apache Kafka
  • Streaming from RabbitMQ
  • Streaming from Google Pub/Sub
  • Streaming from AWS Lambda
  • Streaming from other messaging platforms

Was this helpful?

Edit on GitHub
  1. Use cases, patterns, templates, examples
  2. Functional use cases

Streaming from Apache Kafka and messaging queues

Since Ylem is a real-time streaming platform, our goal is to provide you with a tool with which you can deliver your data from one place to another as fast as possible.

PreviousFunctional use casesNextStreaming from APIs

Last updated 9 months ago

Was this helpful?

The fastest way of doing it is to stream data to Ylem directly from such software as , , , Amazon SQS, , etc.

Triggering pipeline from outside of Ylem

No matter which software you want to stream data from, you need to start your pipeline with the "" task.

After this task you can create any pipeline of any logic you want, using the input data that you are planning to send to Ylem from your messaging platforms.

With the following pipeline you can stream from the external messaging platforms to your API:

Or to your database:

Streaming from Apache Kafka

Streaming from RabbitMQ

RabbitMQ allows you to listen to various events in your infrastructure. At the same time, Ylem can become one of the consumers of RabbitMQ messages.

Streaming from Google Pub/Sub

Streaming from Google Pub/Sub to Ylem's pipelines does not even require any additional library. It is as easy as just triggering pipelines via API.

Streaming from AWS Lambda

We are building an open-source AWS Lambda function for you and it is coming live soon.

Streaming from other messaging platforms

This task doesn't require any additional configuration but allows a pipeline to be triggered , which is used by all of the open-source libraries we developed for external integration.

To stream from Apache Kafka, you can use our open-source library, which triggers pipelines when a new message arrives to a topic and forwards this message as input to the pipeline:

For this integration, you can use our .

Read the detailed .

The procedure of how to trigger pipelines via API is described where you can also get the Endpoint URL.

AWS Lambda allows you to listen to various events in the AWS universe. Therefore these data can immediately be streamed to Ylem by triggering our .

If you want to stream from other messaging platforms, that are not mentioned here, you can either develop an integration yourself , or contact us and request the new integration.

💼
https://github.com/ylem-co/ylem-kafka-trigger
Apache Kafka integration
open-source RabbitMQ consumer
documentation about how to configure it here
Google Pub/Sub integration
AWS Lambda integration
using our API
Apache Kafka
RabbitMQ
Google Pub/Sub
AWS Lambda
External trigger
through the API endpoint
here in our documentation
API for pipelines
Apache Kafka example in the intro guide