Streaming from Apache Kafka and messaging queues
Since Ylem is a real-time streaming platform, our goal is to provide you with a tool with which you can deliver your data from one place to another as fast as possible.
Last updated
Was this helpful?
Since Ylem is a real-time streaming platform, our goal is to provide you with a tool with which you can deliver your data from one place to another as fast as possible.
Last updated
Was this helpful?
The fastest way of doing it is to stream data to Ylem directly from such software as , , , Amazon SQS, , etc.
No matter which software you want to stream data from, you need to start your pipeline with the "" task.
After this task you can create any pipeline of any logic you want, using the input data that you are planning to send to Ylem from your messaging platforms.
With the following pipeline you can stream from the external messaging platforms to your API:
Or to your database:
RabbitMQ allows you to listen to various events in your infrastructure. At the same time, Ylem can become one of the consumers of RabbitMQ messages.
Streaming from Google Pub/Sub to Ylem's pipelines does not even require any additional library. It is as easy as just triggering pipelines via API.
We are building an open-source AWS Lambda function for you and it is coming live soon.
This task doesn't require any additional configuration but allows a pipeline to be triggered , which is used by all of the open-source libraries we developed for external integration.
To stream from Apache Kafka, you can use our open-source library, which triggers pipelines when a new message arrives to a topic and forwards this message as input to the pipeline:
For this integration, you can use our .
Read the detailed .
The procedure of how to trigger pipelines via API is described where you can also get the Endpoint URL.
AWS Lambda allows you to listen to various events in the AWS universe. Therefore these data can immediately be streamed to Ylem by triggering our .
If you want to stream from other messaging platforms, that are not mentioned here, you can either develop an integration yourself , or contact us and request the new integration.
