Kafka Consumer CLI - Consume/Read from Kafka Topic

In this lesson, you will learn how to consume messages or how to read messages from Apache Kafka topic using Kafka Consumer Command Line Interface(CLI).

To learn how to produce or send messages to Kafka topic, please read the following tutorial: Kafka Producer CLI – Produce/Send message to Kafka topic.

If you are interested in video lessons then check my video course Apache Kafka for Event-Driven Spring Boot Microservices.

Consuming Messages From the Beginning

To consume or read all messages from the Apache Kafka topic using Kafka CLI, you will need to first open a terminal window and change the directory to the Kafka folder. Once you are in the Kafka folder, type the following command:

On Windows

bin/windows/kafka-console-consumer.bat --topic my-topic --from-beginning --bootstrap-server localhost:9092

On Mac or Linux

bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092

Let’s break down the command:

bin/kafka-console-consumer.sh: This part of the command specifies the Kafka command-line consumer script that you want to use.
--topic my-topic: This part specifies the Kafka topic from which you want to consume messages. Replace “my-topic” with the actual name of the topic you’re interested in.
--from-beginning: This flag instructs the consumer to start consuming messages from the beginning of the topic’s log. In other words, it will read all the messages in the topic, starting with the earliest available message, and continue to consume new messages as they are produced to the topic.
--bootstrap-server localhost:9092 with this flag, you are providing the consumer script with the initial Kafka bootstrap server. If you have multiple servers running in your Kafka cluster, then, with this parameter, you will provide the address of one of those servers as an initial server to connect to. Once the consumer script connects to one server, it will be able to discover other servers as well. If needed, you can provide two servers here separating them with commas. This way if one server is not available, Kafka consumer will use a second server to connect to the cluster.

Important: Every time you run the consumer script with --from-beginning parameter, it will read messages from the beginning. Even if you run it multiple times. It will always consume messages from Kafka topic from the beginning. In the following section, you will learn how to read recent messages only.

To learn about other command-line arguments that Kafka Console Consumer can accept, check this documentation page.

You can start more consumers and run the same command to consume messages from the same topic.

Each consumer will read all messages from the topic, regardless of whether they have been consumed by other consumers or not. This is because Kafka does not delete messages from a topic after they are consumed. It keeps them in the topic for a configurable period of time, so that other consumers can read them as well.

To stop the consumer script, press Ctrl+C on the terminal. This will terminate the script and disconnect from the Kafka cluster.

Reading New Messages Only

To read new Kafka messages from the Kafka topic, you simply do not use the --from-beginning parameter. So your new command will look like this:

bin/kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092

By simply omitting --from-beginning, you’re telling Kafka, “I’m interested in this topic, but I only want to read new messages, not the old ones that were sent before I started reading.”

How to Display Message Key?

Messages in Kafka topic are stored as a key:value pairs. To display both a message and a message key, you can use the following two parameters:

--property print.key=true this tells the consumer to print the message keys,
--property print.value=true this tells the consumer to print the message.

So your consumer console command will look like this:

bin/kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --property print.key=true --property print.value=true

How to Stop Kafka Consumer CLI Script?

To stop the consumer CLI script, simply press Ctrl + C in your terminal. This will halt the consumer and allow you to exit gracefully.

How to Create Kafka Consumer Group in CLI?

To make your Kafka Console Consumer join a consumer group you can use the --group parameter.

The --group option is used to specify the name of the consumer group that the consumer belongs to. When you run a consumer, it will automatically join a consumer group with a default name. However, you can override this default name by specifying your own group name using the --group option.

For example:

./kafka-console-consumer.sh --topic product-created-topic --bootstrap-server localhost:9092 --from-beginning --property print.key=true --property print.value=true --group my-created-consumer-group

The consumer will join a consumer group named my-created-consumer-group. This is useful when you want to have multiple consumers that belong to different groups.