Kafka CLI: Produce and Consume Messages in Order

In this tutorial, you will learn how to produce and consume messages in order using the Kafka command-line interface (CLI). You will also learn how to use a key to group and identify the messages, and how to specify a separator between the key and the value.

If you are interested in video lessons then check my video course Apache Kafka for Event-Driven Spring Boot Microservices.

What is Kafka CLI?

Kafka CLI is a set of scripts that allow you to interact with Kafka from the terminal. You can use Kafka CLI to create topics, produce messages, consume messages, and perform other operations on Kafka.

In this tutorial, I will use Kafka CLI to send and read messages from Kafka topic.

What are key:value pair messages?

A “key:value pair message” is a message that has two parts: a key and a value, separated by a delimiter. The key and the value can be any string of characters. The key can be used to identify or partition the message, while the value can be the actual content of the message.

To be able to read messages in order, these messages must be sent to Kafka topic as a key:value pair.

Why use key:value pair messages?

One reason to use key:value pair messages is to preserve the order of the messages. Kafka stores messages in topics, which are split into multiple partitions. Kafka uses a hash of the message key to determine which partition to store the message in. Messages with the same key will be stored in the same partition and will be read in order. This is useful when you want to maintain the sequence of events or transactions in your messages.

Important: To make Kafka consumer read messages from Kafka topic in order, these messages must have the same message key.

How to Produce and Consume Messages in Order?

To be able to consume messages in the order they were sent, these messages must be sent as a key:value pair and have the same message key.

To produce and consume key:value pair messages, you need to use the kafka-console-producer and kafka-console-consumer scripts. These scripts allow you to send and receive messages from a Kafka topic and display them on the terminal.

Prerequisites

Before you start, you need to have the following:

Apache Kafka installed on your machine. You can download it from here.
A Kafka topic with some messages in it. You can create a topic and produce some messages using the kafka-topics and kafka-console-producer scripts, as explained in the previous sections of this tutorial.
A terminal window open and ready to run commands.

Send Message with the Same Key

To be able to read messages in order, each message you send must have the same key. To produce message as a key:value pair messages, follow these steps:

Navigate to the bin folder where the Kafka CLI scripts are located. Run this command:
```
cd /Users/sergeykargopolov/kafka
```
Navigate to the Kafka folder where you installed Kafka. For example, if you installed Kafka in /Users/sergeykargopolov/kafka, run this command:
```
cd bin
```
To produce key:value pair messages, run the kafka-console-producer script with the following parameters:For example, to produce key:value pair messages with a colon (:) as the separator to a topic called messages-order, run this command:
- --topic: The name of the topic you want to send messages to. For example, messages-order.
- --bootstrap-server: The address of one or more Kafka servers that you want to connect to. For example, localhost:9092.
- --property parse.key=true: A flag that tells the producer to enable the key:value pair support.
- --property key.separator=:: A parameter that specifies the separator between the key and the value. You can use any character as the separator, but make sure it is not part of the key or the value.

For example, to produce key:value pair messages with a colon (:) as the separator to a topic called messages-order, run this command:

./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic messages-order --property parse.key=true --property key.separator=:

Press Enter to run the command. You are ready to start sending messages. Type a message in the format key:value and press Enter to send it. For example, to send a message with the key 1 and the value First message, type this:

1:First message

Send more messages with the same key. For example:

1:Second message
1:Third message
1:Fourth message
1:Fifth message
1:Sixth message

Notice that you are using the same key for all the messages. This means that they will be stored in the same partition and will be read in order.

The key can be any string of characters, such as a user ID, a product ID, a transaction ID, etc. The key can also be null, which means that the message has no key.

The format of the key is binary, which means that it is composed of bits (0s and 1s). However, you can use other formats, such as strings or numbers, and convert them to binary using a serializer. A serializer is a function that transforms a value into a binary format. For example, you can use a string serializer to convert a string key into a binary key.

Consume Key:Value Pair Messages

To consume key:value pair messages, open another terminal window and run the kafka-console-consumer script with the following parameters:

--topic: The name of the topic you want to read messages from. For example, messages-order.
--from-beginning: A flag that tells the consumer to read all messages from the topic, starting from the first one. If you omit this flag, the consumer will only read new messages that arrive after you start the script.
--bootstrap-server: The address of one or more Kafka servers that you want to connect to. For example, localhost:9092.
--property print.key=true: A flag that tells the consumer to print the key part of the message.
--property print.value=true: A flag that tells the consumer to print the value part of the message.

For example, to consume key:value pair messages from a topic called messages-order from the beginning and print both the key and the value, run this command:

./kafka-console-consumer.sh --topic messages-order --from-beginning --bootstrap-server localhost:9092 --property print.key=true --property print.value=true

Press Enter to run the command. You should see the messages from the topic displayed on the terminal, one per line, with the key and the value separated by a tab. For example:

1	First message
1	Second message
1	Third message
1	Fourth message
1	Fifth message
1	Sixth message

Notice that the messages are displayed in the same order they were sent. This is because they have the same key and are stored in the same partition.

To stop the producer or the consumer script, press Ctrl+C on the terminal. This will terminate the script and disconnect from the Kafka cluster.

Conclusion

In this tutorial, you learned how to produce and consume messages in order using the Kafka CLI. You also learned how to use a message key to group and identify the messages, and how to specify a separator between the key and the value. You can use this technique to preserve the sequence of events or transactions in your messages.

To learn more about Apache Kafka, please check my other Apache Kafka tutorials for beginners.

Happy learning!