Apache Kafka Idempotent Producer in Spring Boot

In this tutorial, you will learn how to implement an Apache Kafka Idempotent Producer in a Spring Boot application.

Idempotence, in the context of Kafka, refers to the ability of the Kafka producer to prevent sending duplicate messages, even in the case of network errors, retries, or broker failures. This is important because it ensures data consistency and reliability, which are essential in systems where data accuracy is critical.

Imagine you’re running an e-commerce platform, and your system sends a message to Kafka every time a purchase is made. Now, suppose there’s a network glitch, and your producer doesn’t get an acknowledgment from Kafka, so it retries and sends the same message again. Without idempotence, Kafka might record that purchase twice. This could lead to issues like double-charging a customer or incorrect inventory tracking. Enabling idempotence in your Kafka producer prevents such scenarios, ensuring each transaction is recorded exactly once, no matter how many times the producer sends it.

In this tutorial, I’ll show you how to enable idempotence for your Kafka Producer in a Spring Boot Microservice.

Kafka Producer Configuration Basics

Before I show you how to set up an Idempotent Producer in Apache Kafka for a Spring Boot application, you need to know the basics of Kafka Producer configuration. To help you with this, I have already written a comprehensive tutorial. It’s clear and detailed and will provide you with the necessary background knowledge.

Please read this tutorial first: Apache Kafka Producer in a Spring Boot Microservice. It will help you create a basic Apache Kafka Producer in a Spring Boot Microservice. Once you’re familiar with the basics, we can proceed to the next part of this current tutorial.

Initial Configuration(no idempotence)

To start, let’s assume you have a Kafka Producer configuration in your Spring Boot application that currently doesn’t have idempotence enabled. It might look something like this:

// Kafka Producer configuration without idempotence
public Map<String, Object> producerConfigs() {
    Map<String, Object> props = new HashMap<>();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, keySerializer);
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, valueSerializer);
    // Other configurations...
    return props;
}

In Apache Kafka, enabling idempotence in the producer configuration requires careful attention to avoid conflicting settings. As per the official Confluent documentation, if idempotence is explicitly enabled but conflicting configurations are set, a ConfigException will be thrown.

Let’s go through the process of correctly enabling idempotence in your Kafka producer configuration.

Enabling Idempotence for Kafka Producer

To enable idempotence, use the following configuration property enable.idempotence and set it to true. By the way, the default value of this property is true. So, by default, idempotence is enabled unless conflicting configuration properties are set. More about it below in this tutorial.

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

If you are using application.properties file, then you can enable idempotence this way:

spring.kafka.producer.properties.enable.idempotence=true

This tells Kafka to prevent the production of duplicate messages.

Also, by setting the enable.idempotence property to true, Kafka automatically manages unique Producer IDs and Sequence Numbers for each message. This ensures that messages are delivered exactly once, without any manual configuration needed for these IDs and numbers.

Ensuring Compatible Configuration Settings

When idempotence is enabled, ensure the following settings are compatible to avoid a ConfigException.

• ACKs: Must be set to all. This ensures that the producer waits for acknowledgments from all in-sync replicas. Read more about Producer Acknowledgements in Spring Boot Microservice.
• Retries: Must be greater than 0(zero). The default value of this configuration property is very high. It is equal to 2147483647, which is equal to Integer.MAX_VALUE. Read more about Producer Retries in a Spring Boot Microservice.
• Max.in.flight.requests.per.connection: This should be set to 5 or lower to maintain the order of message delivery.

These configurations work together to ensure that each message is delivered exactly once, avoiding duplicates even in case of network issues or broker failures.

If these configurations are not set correctly and enable.idempotence property to true, Kafka will throw a ConfigException, indicating a conflict in the settings.

Kafka Producer Configuration Java Code with Idempotence

Here’s how your Kafka Producer configuration should look with idempotence enabled:

// Kafka Producer configuration with idempotence enabled
public Map<String, Object> producerConfigs() {
    Map<String, Object> props = new HashMap<>();
    // Existing configurations...
    // Enable idempotence
    props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
    props.put(ProducerConfig.ACKS_CONFIG, "all");
    props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
    props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

    return props;
}

Idempotence Configuration in application.properties file

If you use application.properties file to configure idempotence in your Spring Boot application, then your configuration properties will be as below:

spring.kafka.producer.properties.enable.idempotence=true
spring.kafka.producer.acks=all
spring.kafka.producer.retries=2147483647
spring.kafka.producer.properties.max.in.flight.requests.per.connection=5

Why use enable.idempotence property?

You might wonder why explicitly set the value of enable.idempotence to true if its default value is already true. I think there can be two reasons for this:

• First, enabling idempotence separately makes your code more explicit and clear about your intention. It also avoids any confusion or ambiguity if you change other configurations later that might conflict with idempotence. For example, if you set acks=1 or max.in.flight.requests.per.connection=10, idempotence will be disabled automatically unless you explicitly enable it.
• Second, enabling idempotence separately gives you more control and flexibility over the producer’s behaviour. You can choose to enable or disable idempotence for different topics or partitions, depending on your use case and requirements. You can also adjust the number of retries and the in-flight requests per connection to optimize the performance and reliability of your producer.

Conclusion

I hope this tutorial was helpful to you.

You’ve just learned how to enable idempotence in your Kafka producer in a Spring Boot application. Remember, setting enable.idempotence to true is key to avoiding duplicate messages. Make sure your max.in.flight.requests.per.connection is 5 or less, retries is more than 0, and acks is set to all. Watch out for those ConfigException errors if the settings conflict.

If you want to learn more about Apache Kafka with Spring Boot, I have more tutorials for you. Check out the tutorials on my page: Apache Kafka Tutorials. They’re straightforward and packed with useful insights.

Thanks for sticking with me in this tutorial. Keep exploring, and happy coding!