blog

Kafka Rebalancing and Improving Message Processing Bottlenecks

NEXTREE.

27 8월 2026 • 7 min read

1. Problem Occurrence

We received an inquiry that Kafka messages were not being consumed properly in production.

After checking the logs of the server, we found that Consumer Group rebalancing was occurring repeatedly at approximately 10-minute intervals.

To identify the cause of the rebalancing, we checked both the Consumer settings and the actual message processing time.

At the time, max.poll.interval.ms was set to 10 minutes, and processing a single Kafka message was taking more than 10 minutes.

max.poll.interval.ms is the maximum interval allowed between the time a Consumer calls poll() and the time it calls poll() again. Therefore, we determined that the Consumer was unable to maintain its normal processing cycle because the message processing time exceeded this setting, resulting in rebalancing.

The settings at the time the problem was identified were as follows.

max.poll.interval.ms=600000

By reviewing the logs and settings together, we confirmed that this problem was not an issue with Kafka receiving messages itself, but rather an issue caused by the message processing time becoming too long and exceeding the Consumer's processing interval limit.

2. Initial Operational Measures

After identifying the cause, it was first necessary to mitigate the situation in which rebalancing was repeatedly occurring in the Consumer running in production.

Accordingly, we changed max.poll.interval.ms from the existing 10 minutes to 30 minutes.

max.poll.interval.ms=1800000

This configuration change ensured that the Consumer had enough time to process messages even if the current processing time took more than 10 minutes.

However, we did not consider this measure to be a fundamental solution.

Because the current processing time was 10 minutes, changing the setting to 30 minutes could mitigate the immediate problem. However, the same problem could occur again if processing time exceeded 30 minutes in the future due to data growth or increased DB processing time.

Therefore, in the production environment, we first changed the configuration so that the Consumer could process messages normally, and then additionally investigated the cause of the long message processing time.

3. Analysis of the Cause of Message Processing Delays

After reviewing the message processing logic, we found that approximately 46,000 records were being stored in MSSQL while processing a single Kafka message.

The problem was not the number of records itself—46,000—but rather that multiple DB storage operations were being performed to process a single message, causing the total processing time to increase to more than 10 minutes.

From the Kafka Consumer's perspective, this was a task for processing a single message, but internally, the application was performing multiple DB storage operations to process that message.

Therefore, we determined that it was necessary to reduce the message processing time itself rather than further adjusting the Consumer settings.

We traced the message processing logic and examined each processing stage, confirming that the DB storage section accounted for a significant portion of the total processing time.

Accordingly, we proceeded with improving this DB storage section in order to reduce the Consumer's processing time.

The cause of and solution direction for this problem can be summarized as follows.

Processing a single Kafka message took more than 10 minutes
max.poll.interval.ms was set to 10 minutes
Rebalancing occurred repeatedly during message processing
The DB storage section consumed a significant amount of time during message processing
It was decided to reduce the total message processing time by improving the DB storage method

4. Changing the DB Storage Method

After reviewing the message processing logic, we found that storing approximately 46,000 records in MSSQL was taking a significant amount of time.

The existing application was using JPA. For this improvement, it was necessary to change only the DB storage section of the specific message processing flow to a Batch-based method.

Because JPA's Batch settings can affect Hibernate operations that use those settings, we kept the existing JPA settings and applied JdbcTemplate.batchUpdate() only to the relevant storage logic.

jdbcTemplate.batchUpdate(
   sql,
   data,
   batchSize,
   (PreparedStatement ps, T obj) -> {
      // 데이터 바인딩
   }
 );

Previously, DB storage operations were repeated while storing multiple records. After the change, the records were grouped into Batch units and inserted together.

This improved the processing time of the problematic DB storage section without affecting other storage logic that uses the existing JPA.

5. Maintaining the Existing ID Generation Method

While changing the DB storage method, we also reviewed the existing ID generation method.

The Entity was using Hibernate's guid strategy, and in the existing environment, IDs were generated using MSSQL's NEWID().

When performing inserts directly using JdbcTemplate, the ID generation process previously handled by JPA/Hibernate must be handled directly. Therefore, it was necessary to configure the process so that IDs would be generated in the same way as before.

It was also possible to generate and pass a UUID in Java, but for this task, we prioritized not changing the existing system's ID generation method.

Therefore, we configured Batch Insert to use MSSQL's NEWID() as well.

INSERT INTO TABLE_NAME ( ID, COLUMN_A, COLUMN_B)
VALUES (NEWID() , ?, ?)

This allowed us to maintain the ID generation method used by the existing system while changing only the DB storage method to Batch Insert.

In this task, an important criterion was not only improving processing performance but also avoiding unnecessary changes to the existing data generation method.

6. Improvement Results

After applying Batch Insert, the time required for the DB processing section that stored approximately 46,000 records decreased, which also improved the total processing time for Kafka messages.

When the problem occurred, rebalancing was happening repeatedly because the message processing time exceeded max.poll.interval.ms.

We first changed max.poll.interval.ms to 30 minutes to mitigate the situation in the production environment, and then improved the DB storage section—the cause of the increased processing time—by applying the Batch Insert method.

As a result, the time required for DB storage decreased, reducing the total message processing time as well, and the recurring rebalancing phenomenon was also mitigated.

The important point in this improvement was that we did not simply avoid the problem by increasing max.poll.interval.ms to 30 minutes. Instead, we approached it by separating the operational mitigation achieved through the configuration change from the code improvement intended to reduce the actual processing time.

The configuration change was a measure to ensure that the Consumer running in production had enough time to process messages, while applying Batch Insert was an improvement intended to reduce the actual message processing time.

This allowed us to first mitigate the operational incident while also addressing the actual cause of the processing delay.

7. What We Confirmed Through This Issue

While investigating this issue, we confirmed that Rebalancing occurring in a Kafka Consumer should not be approached simply as a Kafka configuration problem.

Initially, the issue was reported as "Kafka messages are not being consumed," but after confirming that Rebalancing was occurring repeatedly in the logs, we were able to narrow down the cause by comparing the Consumer configuration with the actual message processing time.

In particular, max.poll.interval.ms and the actual message processing time were at similar levels. By confirming that saving data to the DB was taking a considerable amount of time during message processing, we determined that the phenomenon that appeared to be a Consumer problem was actually connected to the processing time of the message-handling logic.

We also confirmed that increasing max.poll.interval.ms alone cannot resolve the problem.

Increasing the setting allows the Consumer to secure more time to process messages, but if the actual processing time continues to increase, the same problem may eventually occur again.

Therefore, when a similar problem occurs in production, we believe it is important to look beyond specific configuration values and examine the point at which Rebalancing occurred together with the actual message processing flow to identify where processing time is increasing.

In this case, the DB saving stage was identified as the primary bottleneck, but even if the same phenomenon occurs, the actual cause may not be the DB.

What is important is not to prejudge a specific technology or setting as the cause, but to narrow down the cause step by step by examining the logs, configuration, and actual processing flow together.

8. Conclusion

This issue initially appeared as a problem in which Kafka messages were not being consumed properly, but by examining the logs, Consumer configuration, and actual processing time together, we were able to narrow down the cause of the Rebalancing.

Rather than ending the investigation by simply adjusting max.poll.interval.ms, we also identified the DB saving stage, where processing took a long time during the actual message processing flow, and improved the processing method.

In particular, we confirmed that when an operational incident occurs, it is necessary to distinguish between quickly mitigating the current symptoms and analyzing the actual processing flow to improve the root cause.

If a similar problem occurs in the future, rather than simply checking the point where the error occurred or the configuration values, we intend to respond by examining the logs and actual processing flow together, narrowing down the cause step by step, and improving the identified bottleneck.