Usage of Apache Kafka

Ylem uses Apache Kafka to exchange messages for processing pipelines and tasks. By default Ylem already comes with the pre-configured Apache Kafka container, however, you might already have an Apache Kafka cluster in your infrastructure and might want to reuse it.

In this case, you need to take the following steps:

1. Create Apache Kafka topics

To be able to work correctly, Ylem requires the following Apache Kafka topics to be created:

• task_runs

• task_runs_load_balanced

• task_run_results

• query_task_run_results

• notification_task_run_results

2. Remove the Apache Kafka container from docker-compose.yml

Open docker-compose.yml in the root folder, and remove or comment out the following lines to exclude it from the network.

Also don't forget to remove these and these dependency checks.

3. Add your Apache Kafka host to the .env parameters

Open .env and replace the following value with your Kafka host:

YLEM_KAFKA_BOOTSTRAP_SERVERS=ylem_kafka_broker:39092

Now if you run docker compose up and everything is set up correctly Ylem should be able to work with your own Kafka cluster.