Streaming from Apache Kafka and messaging queues
Since Ylem is a real-time streaming platform, our goal is to provide you with a tool with which you can deliver your data from one place to another as fast as possible.
Last updated
Since Ylem is a real-time streaming platform, our goal is to provide you with a tool with which you can deliver your data from one place to another as fast as possible.
Last updated
The fastest way of doing it is to stream data to Ylem directly from such software as Apache Kafka, RabbitMQ, Google Pub/Sub, Amazon SQS, AWS Lambda, etc.
No matter which software you want to stream data from, you need to start your pipeline with the "External trigger" task.
This task doesn't require any additional configuration but allows a pipeline to be triggered through the API endpoint, which is used by all of the open-source libraries we developed for external integration.
After this task you can create any pipeline of any logic you want, using the input data that you are planning to send to Ylem from your messaging platforms.
With the following pipeline you can stream from the external messaging platforms to your API:
Or to your database:
To stream from Apache Kafka, you can use our open-source library, which triggers pipelines when a new message arrives to a topic and forwards this message as input to the pipeline: https://github.com/ylem-co/ylem-kafka-trigger
RabbitMQ allows you to listen to various events in your infrastructure. At the same time, Ylem can become one of the consumers of RabbitMQ messages.
For this integration, you can use our open-source RabbitMQ consumer.
Read the detailed documentation about how to configure it here.
Streaming from Google Pub/Sub to Ylem's pipelines does not even require any additional library. It is as easy as just triggering pipelines via API.
The procedure of how to trigger pipelines via API is described here in our documentation where you can also get the Endpoint URL.
AWS Lambda allows you to listen to various events in the AWS universe. Therefore these data can immediately be streamed to Ylem by triggering our API for pipelines.
We are building an open-source AWS Lambda function for you and it is coming live soon.
If you want to stream from other messaging platforms, that are not mentioned here, you can either develop an integration yourself using our API, or contact us and request the new integration.
