Updating Kafka Topic Configuration

In this tutorial, I want to share with you how to add or update the configuration of an existing Kafka topic.

You may need to update the configuration of an existing Kafka topic when you want to change the behaviour of the topic or when you want to optimize the performance of your Kafka cluster.

For example, you may want to increase the minimum number of in-sync replicas required for a producer to receive an acknowledgment of a write operation, or you may want to increase the maximum size of a message that can be written to the Kafka topic.

If you are interested in video lessons then check my video course Apache Kafka for Event-Driven Spring Boot Microservices.

Before Updating Kafka Topic Configuration

Before updating the configuration of an Apache Kafka topic, it is generally recommended to follow these steps:

Consult your team members about the changes you are going to make,
Stop producers and consumers that are using the target topic(s). This ensures that no data is being written to or read from the topic during the configuration update process.

Once the topic configuration is updated, you can start Kafka producers and consumers to test if your application works as expected.

It’s important to note that these steps may vary depending on your specific Kafka setup and deployment environment.

Updating Kafka Topic Configuration

Let’s assume that we need to update the Kafka topic configuration to change the minimum number of in-sync replicas. To do that, we will use the kafka-configs.sh script, which you will find in your Apache Kafka’s “bin” folder.

On Windows

windows/kafka-configs.bat --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name topic-insync --add-config min.insync.replicas=2

On MacOS or Linux

./bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name topic-insync --add-config min.insync.replicas=2

Where:

./bin/kafka-configs.sh: This is the path to the kafka-configs.sh script that you need to run in order to update the configuration of a Kafka topic.
--bootstrap-server localhost:9092: This specifies the Kafka broker(s) that you want to connect to. In this case, I am connecting to a broker running on localhost on port 9092.
--alter: This tells the script that you want to alter the configuration of an existing Kafka topic.
--entity-type topics: This specifies that you want to alter the configuration of a Kafka topic.
--entity-name topic-insync: This specifies the name of the Kafka topic that you want to update. In my case, I am updating the topic with the name “topic-insync”,
--add-config min.insync.replicas=2: This specifies the configuration property that you want to update and its new value. In this case, I am updating the min.insync.replicas property to have a value of 2.

What topic configuration can be updated?

The kafka-configs.sh script cannot be used to update just any information about the topic. For example, it cannot update the topic name.

Below is a list of some of the configuration properties that you can update using the kafka-configs.sh script:

min.insync.replicas: The minimum number of in-sync replicas required for a producer to receive an acknowledgment of a write operation,
retention.ms: The time duration for which a message will be retained in the Kafka topic,
max.message.bytes: The maximum size of a message that can be written to the Kafka topic,
cleanup.policy: The policy used to determine when messages should be deleted from the Kafka topic,
segment.bytes: The size of each log segment file,
Other properties.

Please note that this is not a complete list, and there are many other configuration properties that you can update using the kafka-configs.sh script. For a complete list of topic configurations, I have included a link to Confluent documentation called “Other properties”.

How do we update the Apache Kafka topic name?

At the moment of writing this tutorial, renaming an existing Apache Kafka topic is not supported. You cannot use kafka-configs.sh to change the name of an existing Kafka topic. But there is a way around it.

Basically, you need to create a new topic and copy all messages from the old topic to a new one.

Follow these steps:

Create a new topic with a new name,
Use a Kafka consumer to read all messages from the old topic,
Use a Kafka producer to write all messages to the new topic,
Once all messages have been copied to a new topic, you can delete the old topic.

But before doing it, consider the amount of messages that need to be transferred from an old topic to a new one. If there are too many messages, the transfer can consume additional resources and maybe keeping an old topic name is a better option.