Klāvs Priedītis

Full Stack Web Developer

Apache Kafka quick start guide

Apache Kafka is quite a popular distributed messaging system. It has been designed with a lot of thoughts invested in it. After reading such a good documentation you problably want to try it out to see how it behaves in practice. What could be a better solution to try it out than Docker?

Let's get down to business. First of all, Kafka is dependent on Zookeeper, another Apache project, so you'll need to get that running also. Running multiple applications on the same system just to test them out is a breeze if you are using Docker.

Begin by starting up a Zookeeper daemon. Assign a name to it so that it can be referenced later.

docker run -d --name zookeeper jplock/zookeeper:3.4.6

Then do the same for Kafka.

docker run -d --name kafka --link zookeeper:zookeeper ches/kafka

Now that you have Zookeeper and Kafka running, create a test topic 

docker run --rm --link zookeeper:zookeeper ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181

Now you will need to open two terminal instances. One for producer and the other one for consumer. Let's start with a producer. Do not close this one.

docker run --rm --interactive --link kafka:kafka ches/kafka kafka-console-producer.sh --topic test --broker-list kafka:9092

If you'd like to, you can now enter some random text terminated by a newline in the producer terminal.

Finally, it is time to start up a consumer terminal in the second terminal instance.

docker run --rm --link zookeeper:zookeeper ches/kafka kafka-console-consumer.sh --topic test --from-beginning --zookeeper zookeeper:2181

You may have noticed that the consumer has received the random text you wrote into the producer earlier. Now you can go wild and start sending any messages you'd like. Try starting more producer and consumer instances to see what happens with messages then.

These instructions are based on instructions provided by Ches Martin. I've made some improvments, by linking producer to Kafka instance and consumer to Zookeeper instance using Docker's built in linking instead of IP provided in environment variables.

OK, that's about it! Now you know how to quickly start Apache Kafka on your machine.


Be sure to share this article if you feel like it could be helpful to someone!