How to Start Apache Kafka Server with KRaft

In this tutorial, you will learn how to start Apache Kafka Server with KRaft, a new consensus protocol that replaces ZooKeeper for metadata management. KRaft simplifies Kafka’s architecture and improves its performance and stability. You can read more about KRaft here and here.

Step 1: Generate a Cluster UUID

Before you can start Apache Kafka server on your computer, you need to generate a UUID for your Kafka cluster. A cluster UUID is a unique identifier for your Kafka cluster. You can generate one by running the following command in your terminal:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

This will store the cluster UUID in a variable called KAFKA_CLUSTER_ID. You can print it by running:

echo $KAFKA_CLUSTER_ID

Note: All the commands in this tutorial assume that you are in the root directory of the Kafka folder that you downloaded and extracted. If you are in a different location, you may need to adjust the paths accordingly.

Step 2: Format Log Directories

Now that you have generated a unique identified for your Kafka cluster, you can use it to format the log directories where Kafka will store its data. You can do this by running the following command:

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

This will use the cluster UUID and the configuration file config/kraft/server.properties to format the log directories. You can modify the configuration file to change the default settings, such as the log directory path, the number of controllers, and the ports.

What are log directories?

Log directories are the folders where Kafka stores its data as ordered sequences of messages called logs. Each log is divided into smaller files called segments, which contain the actual messages and some index files to help locate them. Log directories are organized by topics and partitions, which are logical groups of logs that belong to a specific use case and provide parallelism and fault tolerance.

Why do we need to format log directories?

We need to format log directories before we can start Kafka in KRaft mode. Formatting log directories means assigning a unique identifier to the cluster and initializing the metadata for the partitions. This prevents accidental configuration changes and ensures compatibility with KRaft mode.

What is server.properties file?

The server.properties is a configuration file that specifies various settings for the Kafka server, such as the broker id, the listeners, the log directories, the number of controllers, the ports, and the log retention policies. You can modify this file to change the default values according to your needs and preferences. You can also override some of the properties by passing them as command-line arguments when starting the Kafka server.

Step 3: Start the Kafka Server

Now you are ready to start Apache Kafka server in KRaft mode. You can do this by running the following command:

bin/kafka-server-start.sh config/kraft/server.properties

This will start the Kafka server as both a broker and a controller in a single process. This is suitable for local development and testing, but not for production environments. For production, you should run separate processes for brokers and controllers, and use multiple nodes for fault tolerance.

Note: By default, when starting a single Kafka server on your computer, it will listen on localhost:9092 for incoming connections from clients. You can change this by modifying the listeners property in the server.properties file.

Step 4: Test the Kafka Server

To test if the Kafka server is working properly, you can use the Kafka command-line tools to create a topic, produce some messages, and consume them. For example, you can run the following commands in separate terminals:

# Create a topic named product-created-events-topic with one partition and one replica
bin/kafka-topics.sh --create --topic product-created-events-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# Produce some messages to the topic
bin/kafka-console-producer.sh --topic product-created-events-topic --bootstrap-server localhost:9092

# Consume the messages from the topic
bin/kafka-console-consumer.sh --topic product-created-events-topic --from-beginning --bootstrap-server localhost:9092

You should see the messages you typed in the producer terminal appear in the consumer terminal.

Conclusion

I hope this tutorial is helpful to you. To learn more about Apache Kafka, please check my other Apache Kafka tutorials for beginners. You can also find more resources and documentation on the official Kafka website.

Happy learning!