NATS JetStream Incident Analysis

1. Overview

Applications based on MSA (Microservice Architecture) are structured by dividing one large system into multiple independent services. Each service has different functions and responsibilities, but data transfer and state sharing between services must continue in order to provide actual services. Messaging is one of the communication methods used between services, and a message broker can be utilized when asynchronous data delivery is required.

Project A uses NATS JetStream, a persistent messaging system based on NATS, to reliably and efficiently deliver sensor data and event data between multiple services onboard a vessel. NATS JetStream supports not only asynchronous message delivery between services, but also message storage, Consumer state management, and redelivery, enabling the reliable processing of continuously generated vessel data.

However, various situations occur during actual development and operation, such as forced shutdowns of the vessel server and abnormal application termination, and we frequently experienced unexpected failures related to NATS JetStream in these environments. In this article, based on failure cases that occurred while operating NATS JetStream, we analyze the primary causes and organize a diagnostic procedure for identifying and responding to similar failures in the future based on the Consumer state values.

2. Occurrence of Message Processing Failures

When the server was rebooted after an abnormal shutdown, such as a forced shutdown, a phenomenon occurred in which the Durable Consumer could not properly consume messages even though messages continued to be Published to the Stream.

When starting, this application checks whether an existing Durable Consumer is present. If a Consumer already exists, it reuses the existing Consumer instead of creating a new one. A Durable Consumer continuously maintains not only subscription information, but also consumption state, including the Delivered Sequence, which indicates how far messages have been delivered; the Ack Floor, which indicates the point up to which acknowledgments have been completed; the state of messages that have not yet been acknowledged; and messages targeted for redelivery. Therefore, if the server is forcibly terminated while processing messages, Ack processing for some messages may not be completed, or the Consumer's delivery and processing state may remain unfinished.

After the server reboots, the application continues to reuse the existing Durable Consumer, and therefore also inherits the message processing state that remained immediately before the abnormal termination. Under normal circumstances, unprocessed messages should be delivered again according to AckWait and the redelivery policy, after which message consumption should continue. However, depending on various cases, such as accumulated Ack Pending messages or the processing state between the Delivered Sequence and Ack Floor not continuing properly, message delivery by the Consumer may be delayed or stopped.

3. How to Analyze Consumer State Values

When a failure occurs, first check the Consumer state information and determine at which stage the failure occurred.

Information for Consumer “stream” > “consumer”

Configuration:
    Name: “consumer”
    Pull Mode: true
    Filter Subject: subject.>
    Deliver Policy: All
    Ack Policy: Explicit
    Ack Wait: 30s
    Maximum Deliveries: 3
    Maximum Ack Pending: 1000
State:
    Last Delivered Message:
        Consumer sequence: 15230
        Stream sequence: 185400

    Acknowledgment Floor:
        Consumer sequence: 14230
        Stream sequence: 184400

    OutStanding Acks: 1000
    Redelivered Messages: 15
    Unprocessed Messages: 3270
    Waiting Pulls: 1

• Table 1. Key Consumer State Values and Their Meanings

1) When OutStanding Acks Reaches MaxAckPending

OutStanding Acks refers to the number of messages that the Consumer has delivered to the application but has not yet received an Ack for. When this value reaches the MaxAckPending setting, JetStream may limit the delivery of new messages until Acks are processed and capacity becomes available.

Maximum Ack Pending:  1000
OutStanding Acks:     1000
Unprocessed Messages: 3270

In the example above, 3270 messages still remain in the Stream, but the Consumer has already delivered messages up to the permitted Ack Pending limit. Therefore, if Acks are not processed by the application, no additional messages will be delivered, which may result in a situation where Publish from outside appears normal but the Consumer receives no messages.

2) When Pull Requests Do Not Occur Properly

A Pull Consumer delivers messages only when the application sends Pull requests using fetch() or consume(). Therefore, if messages remain to be delivered but Waiting Pulls remains at 0, the application's Pull processing logic can be suspected rather than the Consumer itself.

Unprocessed Messages: 3270
OutStanding Acks:      0
Waiting Pulls:         0

In the state above, message delivery is not blocked by Acks, but there are no Pull requests requesting messages. After the server reboots, check whether the Pull processing Thread or consume() loop started properly and whether the Subscription was properly restored after the NATS reconnection.

3) When Last Delivered and Ack Floor Do Not Progress for an Extended Period

Last Delivered indicates how far the Consumer has delivered messages, while Ack Floor indicates how far acknowledgments have been completed continuously. Rather than focusing on the difference between the two values themselves, it is important to check whether neither value progresses at all over time.

Last Delivered Stream Sequence: 185400
Ack Floor Stream Sequence:      185380
OutStanding Acks:               20
Unprocessed Messages:           3270

\n If Last Delivered remains at 185400 for several seconds or minutes while Unprocessed Messages continues to increase, the Consumer's consumption flow may be stalled. In this case, check Waiting Pulls, Redelivered Messages, and the application logs together to distinguish whether the issue lies with the Consumer or with application processing.

In these failure situations, message consumption proceeded normally after the existing Durable Consumer was deleted and the application was restarted to create a new Consumer. This confirmed that, rather than being caused by the Publisher or the Stream itself, the state maintained by the existing Durable Consumer from before the abnormal termination had affected message consumption.

Afterward, we created a recovery script that deletes and recreates the existing Consumer when the server restarts, thereby initializing the state of the messaging system. This helped reduce failures related to NATS JetStream onboard the vessel.

4. Conclusion

In actual operating environments, message processing in NATS JetStream may fail due to unexpected situations such as forced server shutdowns or abnormal application termination. In particular, message processing is a core function linked to data transfer between services, so it is important to quickly identify the cause and take appropriate action when a failure occurs.

Therefore, when similar failures occur in the future, it is necessary to quickly identify and respond to their causes based on the key Consumer state values and case-by-case analysis methods described in this article, thereby minimizing service downtime and maintaining stable operations.

deeeneee