A Case Study of Migrating Legacy Integrations to Asynchronous Processing Using the Inbox Pattern

-Experience converting the legacy bulk dispatch information reception stage from synchronous processing to a save-then-exit structure-

1. Introduction — What problem did we face?

The adapter service I was responsible for serves as a gateway that receives survey dispatch information sent by the hospital's legacy medical information system (hereafter, the legacy system) and registers it with internal microservices (the account service and survey service). At first glance, it appears to be a simple integration that receives and registers a single message, but once it entered production, three problems emerged simultaneously.

• Processing time was directly equivalent to the legacy system's transaction occupancy time. To process a single received item, the adapter synchronously performs two calls to the legacy system, four to five calls to the account service, and around ten calls to the survey service—approximately 20 REST round trips in total. Because the legacy system makes consecutive calls for bulk items within its own transaction, it held the entire duration of the adapter's processing time. When I examined the traces, the HTTP response time per item was 1.4–2.2 seconds, and during periods when hundreds of items arrived at once, timeouts on the legacy-system side were an ever-present risk.

• Partial failures left orphaned resources. Since the adapter was designed without its own database (DB-less), the @Transactional attached to the entry point effectively did nothing. If an intermediate REST call failed, guest accounts or recipient information that had already been created in earlier steps remained without being rolled back.

• A single synchronous response was carrying too much responsibility. Because everything from the message parsing result to business validation (whether the form exists, whether the group exists, and whether the form has been distributed) was returned through the success/failure header of a single response, the response did not indicate at which stage or why the failure occurred, and there was no means of reprocessing.

This article summarizes the process of changing this reception stage to a "save immediately upon receipt and exit (Inbox pattern)" structure, while separating the actual registration logic into Kafka event-based asynchronous processing. It also records the basis for the design decisions, decisions that were reversed along the way, and the results confirmed through actual measurements.

2. Background to the Technology Choices — The Process of Eliminating Alternatives

2.1 Choosing the Asynchronous Processing Method

The direction of "respond first and process later" was decided quickly, but there were several possible ways to implement it. The reasons for eliminating each option were as follows.

• Asynchronous processing with an internal adapter thread pool — This requires the fewest code changes, but because the adapter has no database, it cannot record received items anywhere. If the pod restarts, items for which a receipt response was sent disappear without a trace. Since the legacy system does not resend items after receiving a success response, this directly results in lost dispatches.

• Adding a database to the adapter — This would allow receipt records to be retained, but it would violate the service's identity that "the adapter does not own data." Once it owns data, new synchronization issues with the source service arise.

• Directly loading the original messages into a message broker — This prevents loss, but provides no way to query the status of individual items (retry count, failure reason, and whether processing is complete) or for operators to inspect them visually. We determined that a structure unable to answer "How many items are currently backed up?" during a large-scale failure would be difficult to operate.

The final choice was the Inbox pattern + event trigger. The original message is stored in the survey service, which already owns a database, while the survey service treats it only as an uninterpreted original message (opaque payload) and processing metadata. The boundary that only the adapter understands the meaning of the message is preserved, while responsibility for infrastructure concerns such as storage and status management is delegated to the database-owning service.

2.2 Processing Trigger — Events Instead of a Feign Callback

Who would trigger the actual processing after storage was another point of debate. If the survey service directly called the adapter (Feign), the wiring would be simple, but a reverse synchronous dependency would be newly introduced into what had previously been a one-way adapter → survey dependency. Once the communication matrix becomes circular, corresponding failure-propagation paths also emerge.

Therefore, after the storage commit, the survey service publishes only a domain event and the adapter receives it as a consumer and processes it. As a result, the inter-service contract gained only one line: "the adapter subscribes to survey events," and no reverse synchronous call was introduced. Specifying the partition key as the survey form number also provided, at no additional cost, the benefit of ensuring that events for the same target are published and consumed in order within the same partition.

2.3 Applied Technology Stack

• Apache Kafka — Publishes and subscribes to storage-complete events. The partition key provides the infrastructure basis for guaranteeing order.

• PostgreSQL & JPA — Inbox persistence. The original message is managed as JSON text, while the status, retry count, and failure reason are managed as normalized columns.

• ShedLock — A distributed lock was applied so that the fallback batch would not execute redundantly across multiple instances.

• Redis — Used for caching external-system lookups (§4.4) and as a layer for retaining dispatch metadata.

• Spring Cloud OpenFeign — Calls the Inbox registration, acquisition, and status-update APIs. The contract was separated into a dedicated client module and distributed through the company's internal repository.

3. Implementation Process

3.1 Reception Stage — What to Keep and What to Remove

The first step in the migration was to minimize the work performed at the time of receipt. However, we did not eliminate all validation. A message that is malformed cannot be recovered even if it is stored, so such messages are not stored; instead, a synchronous failure response is returned immediately. All other business validation was moved to the asynchronous path.

// 전환 전: 진입점이 등록 전체를 동기로 수행 (@Transactional은 DB가 없어 무효)
@Transactional
public void registerSurveySendInfo(SurveySendInfo info) {
    // 레거시 조회 2회 + 계정 4~5회 + 설문 10여 회 ... 총 20여 회 REST 왕복
    // 중간 실패 시 앞 단계 산출물(게스트, 수신자정보 등)이 롤백 없이 잔존
}

// 전환 후: 저장-후-종료
public void acceptSurveySendInfo(SrvySendInfoDto ipd) {
    validateRequired(ipd); // srvyDrawupNo, patno, formCd 존재만 확인
    smsMsgInfoAction.saveMsgInfo(ipd);  // 발송 메타데이터 보존(Redis)
    surveySendInfoInboxProxy.register( // 원문 그대로 저장 + 커밋 후 이벤트 발행
            SurveySendInfoInboxCdo.from(ipd));
}

[Code 1] Before and after the change to the reception entry point. The contract with the legacy system (URL, message format, and response headers) was not changed.

What we actually encountered here was finalizing the list of required fieldsAt first, we also treated the dispatch date and time as required, but staging measurements confirmed that messages with an empty dispatch date and time actually existed in production. Since the existing logic already fell back to the receipt time when parsing failed, we corrected this by excluding it from validation, storing the empty value as is, and leaving the fallback to processing time. This was a case where strengthening validation nearly blocked traffic that had been processing normally.

3.2 State Machine and a Single Retry Ledger

Stored records move through five states: RECEIVED → PROCESSING → COMPLETED; when processing fails, they are recorded as FAILED, and once the retry limit is exceeded, they transition to DEAD and become subject to manual intervention.

The principle we paid the most attention to in the design was to maintain a single retry ledgerWhen retries in the messaging layer (10 attempts by default, followed by DLT) overlap with application retries, no one can explain the actual number of attempts. Therefore, the consumer does not throw exceptions outward; it catches everything and only records it in Inbox, while retry decisions are handled exclusively by Inbox's attempt count and the fallback batch.

3.3 Atomic Claim — The Only Gateway That Prevents Duplicate Processing

The most dangerous scenario in the asynchronous transition is that duplicate event consumption and batch republishing race with each other, causing the same record to be registered twice. We prevented this with a single conditional UPDATE statement. Only the instance that succeeds in the transition receives the original message and proceeds with processing; the others terminate quietly.

@Modifying
@Query("UPDATE SurveySendInfoInboxJpo i " +
       " SET i.status = 'PROCESSING', i.modifiedOn = :now " +
       " WHERE i.id = :id AND i.status IN ('RECEIVED', 'FAILED')")
int claim(@Param("id") Long id, @Param("now") LocalDateTime now);

// 갱신 행 수가 1이면 점유 성공 → 원문 반환, 0이면 이미 점유/종결된 건 → null
public SurveySendInfoInboxRdo claimAndLoad(Long id) {
    if (repository.claim(id, LocalDateTime.now()) == 0) {
        return null;
    }
    return SurveySendInfoInboxRdo.from(repository.getById(id));
}

[Code 2] Claiming must be a conditional single UPDATE, not a read followed by an update. If the read and update are separated, the interval between them becomes a race condition.

Early in the design, we separated the claim API and the original-message retrieval API, but during implementation we combined them so that the original message is returned along with a successful claimSplitting them into two calls would not only increase the number of calls; it would also create an intermediate state in which claiming succeeds but retrieval fails.

3.4 Consumer — Do Not Throw Exceptions Outward

@KafkaListener(topics = SurveyTopic.SURVEY_SEND_INFO_RECEIVED, groupId = "...")
public void handle(SurveySendInfoReceivedDomainEvent event) {
    SurveySendInfoInboxRdo rdo = inboxProxy.claim(event.getInboxId());
    if (rdo == null) {
        return;   // 이미 점유·종결됐거나 선행 미종결 건이 있어 보류된 경우
    }
    try {
        surveyFlow.registerSurveySendInfo(rdo.toSurveySendInfo());
        inboxProxy.complete(rdo.getId());
    } catch (PromsBizException e) { // 업무 예외 → 메시지를 그대로 사유로
        handleFailure(rdo, e.getMessage());
    } catch (Exception e) { // 시스템 예외 → 고정 문구
        log.error("inbox 처리 실패. id={}", rdo.getId(), e);
        handleFailure(rdo, "시스템 오류");
    }
}

private void handleFailure(SurveySendInfoInboxRdo rdo, String reason) {
    String status = InboxProxy.fail(rdo.getId(), reason);  // FAILED 또는 DEAD 반환
    if ("DEAD".equals(status)) {
        amcFlowProxy.sendResult(rdo.getSrvyDrawupNo(), "N", reason); // 레거시 회신
    }
}

[Code 3] The failure-recording API determines and returns whether to transition to DEAD, while the adapter controls the response trigger.

In the retry policy, we decided not to distinguish between business exceptions and system exceptions. Even a business error such as an unregistered form can be resolved if the person in charge registers the form, and the retry interval is one minute, so the cost is almost negligible. The difference between the two appears only in the reason message returned to the legacy system.

3.5 Fallback Batch — Republish Only the Event Instead of Processing Directly

We added a batch to compensate for lost events or consumer outages, but made sure that the batch does not call the registration logic directly. Republishing only the event makes the normal and recovery paths converge on exactly the same codeThis was because. We also considered that recovery-specific code paths are not executed during normal operation, making their behavior difficult to trust precisely when they are needed.

@Scheduled(cron = "0 * * * * *")                       // 1분 주기
@SchedulerLock(name = "surveySendInfoInboxRedeliver")  // 다중 인스턴스 중복 방지
public void redeliver() {
    // ① 이벤트 유실 의심: RECEIVED 이면서 생성 후 2분 경과
    // ② 재시도 대상: FAILED 이면서 retryCount < 5
    // ③ 처리중 고착: PROCESSING 이면서 15분간 갱신 없음 → RECEIVED 복귀 후 재발행
    for (SurveySendInfoInbox target : InboxStore.findRedeliverTargets()) {
        try {
            eventPublisher.publish(SurveySendInfoReceivedDomainEvent.of(target));
        } catch (Exception e) { // 한 건 실패가 라운드 전체를 막지 않도록
            log.warn("재발행 실패. id={}", target.getId(), e);
        }
    }
}

[Code 4] One batch absorbs three types of stalled situations. It is safer to attach a per-record try-catch as a matter of habit when writing a batch.

4. Lessons Learned from Problem Solving

4.1 Retry Safety — Atomize Instead of Scattering Guards

To introduce asynchronous retries, the registration logic must remain safe even if it runs multiple times. A step-by-step review revealed duplication risks because recipient information, scheduling, assignment, and task registration were all unconditionally created as new records. In particular, there had already been real incidents in which duplicate schedules and assignments left active dispatch records without tasks, causing subsequent batches to fail permanently.

At first, we considered adding an API to the adapter to "query backward for whether an existing record exists" and attaching a guard at each step. However, this approach scatters guards across four places and leaves room to repeat the same mistake every time a new step is added.

Instead, we arranged for the survey service to create and call a bundle API that combines the four steps into a single transactionIf the bundle fails, no survey-side data remains at all, making retries safe automatically. Since the entry point was already filtering out completed records into a resend branch, this single act of atomization made the adapter-side idempotency guards entirely unnecessary. The key was moving responsibility for atomicity to the service that owns the data.

4.2 Ordering Guarantees and Remaining Risks

Because the legacy system does not distinguish between batch and individual messages, we made the explicit decision to regard arrival order as business orderThe partition key supports this at the infrastructure level, but one area remains where the premise can be disrupted by retries. If the initial dispatch message fails and is awaiting a retry, a submission message for the same subject may arrive first, reversing the order.

We blocked this at the point just before claiming with a guard that terminates without processing the current record if an earlier unfinished record for the same subject existsA held record is republished by the batch in order, so it is ultimately processed in arrival order. It was a low-cost guard that could be resolved with a single index, and it continued to serve as a safety mechanism when we later added the policy of "interpreting receipt of the same message again as a resend request."

4.3 The Cost of Reducing the Synchronous Contract — Creating a Path to Report Failures

As the meaning of the response changed from "registration complete" to "receipt complete," the ability to notify the legacy system of business validation failures disappeared. To fill this gap, we expanded the one-way result notification interface that happened to be under separate design at the time so that it could also be used as a failure-response channel.

• Final-failure (DEAD) records — Notify the recipient, including the reason for failure, that dispatch could not be completed.

• Notification of omitted sending — We notify you, along with the reason, that this request only generates the questionnaire and does not send an AlimTalk message.

• Records that have already been submitted and cannot be resent — Notify the reason why resending is not possible.

The survey service, which owns the data, makes the determination and reports it in the response, while the adapter, which understands the legacy message format, handles the response itself. However, the lack of a means to send a response again when the response fails remains an issue.

4.4 Do Not Guess at Bottlenecks—Instrument Them — Caching External Lookups

Even after the transition, throughput did not increase as much as expected, so we cross-checked several traces using a monitoring tool. As a result, of the two legacy lookups generated for each registration, One consent-form lookup consistently took 760–984 millisecondswhile another took only 50–150 milliseconds. Since the two calls looked nearly identical in terms of code structure, without instrumentation we likely would have assumed they were equally slow and worked on the wrong area.

After confirming that the legacy side could not be improved, we absorbed the load with a short-TTL cache in the adapter. Because multiple messages for the same survey arrive together as part of the business workflow, the total number of lookups is reduced from the "number of messages" to the "number of distinct surveys."

public List<AmcSurveyConsent> findAmcSurveyConsent(String srvyNo) {
    String key = "proms:amc:survey-consents:" + srvyNo;
    try {
        String cached = redisTemplate.opsForValue().get(key);
        if (cached != null) {
            return objectMapper.readValue(cached, new TypeReference<>() {});
        }
    } catch (Exception e) {
        log.warn("캐시 조회 실패 — 원 호출로 폴백. srvyNo={}", srvyNo, e);
    }

    List<AmcSurveyConsent> result = amcProxy.findSurveyConsent(srvyNo);
    // 동의서는 빈 목록도 캐시한다 — 동의서 없는 설문이 일반적이라 여기가 절감의 핵심
    cachePut(key, result, Duration.ofMinutes(5));
    return result;
}

[Code 5] To ensure that a cache failure does not become a functional failure, exceptions in the cache layer are logged as warnings, after which the system falls back to the original call.

We applied different caching policies depending on the nature of the data. Results indicating that a configuration does not exist are not cached. If the old result remains for up to five minutes even after the person in charge corrects the configuration, it leads to inquiries asking why the correction is not taking effect. Conversely, empty consent-form lists are cached. Since surveys without consent forms are actually the norm, this case accounts for most of the savings. As a trade-off, we announced to operations that changes to legacy configurations may take up to five minutes to take effect.

4.5 Exceptions and Transaction Boundaries — Why We Decided Not to Use noRollbackFor

When there is a requirement that "processing may fail, but the failure record must remain," the easiest solution is to configure the exception to be excluded from rollback. However, this approach requires maintaining a list each time the types of exceptions increase, and it is impossible to tell which states remain and which disappear without reading through all the code.

Therefore, we established the principle of resolving the issue solely by separating transaction boundaries. Occupancy, completion, and failure recording are each implemented as independent REST calls and independent transactions. Even if the main processing fails, the failure record is in a separate transaction and is therefore not rolled back, so the attempt count and reason remain intact.

In the worst case, where even recording the failure fails, we rely not on rollback but on recovery over time. If a state remains as processing, a batch job reverts it after 15 minutes, and the atomicity of occupancy prevents duplicate processing, ensuring safety. However, we explicitly documented that the attempt count does not increase on this path.

During the full verification process, we also found a gap in the shared library. An internal business-exception class was missing from the consumer's retry-exclusion list, meaning that a business exception that should have gone straight to the DLT was reaching the DLT only after 10 retries. The Inbox path was unaffected because the consumer does not throw an exception, but this issue directly affected consumers in other services, so we registered it as an improvement item for the shared library.

4.6 Including the Rollback Method in the Design

As we organized the rollback procedure in advance in case problems arose after deployment, we discovered that this transition had a constraint that prevented a conventional image rollback from working. The legacy system does not resend messages because it has already received the acknowledgment response, while the old-version adapter has no consumer, leaving any outstanding unfinished items permanently unprocessed. In other words, confirming that all unfinished items have been drained must precede any rollback path.

Therefore, we chose a configuration toggle rather than an image rollback as the primary measure. If only the receiving path is reverted to the old synchronous method while the consumer and batch job remain active, new arrivals are processed synchronously while the remaining items continue to be consumed, allowing the drain to complete naturally.

// 롤백 토글: false면 Inbox를 거치지 않고 구 동기 경로로 즉시 처리한다.
// off 상태에서도 컨슈머·fallback은 유지되어 잔여 inbox 건은 계속 소화된다.
@Value("${proms.survey-send-info.inbox-enabled:true}")
private boolean inboxEnabled;

if (inboxEnabled) {
    surveySendInfoInboxFlow.acceptSurveySendInfo(ipd); // 저장-후-종료
} else {
    surveyFlow.registerSurveySendInfo(new SurveySendInfo(ipd)); // 구 동기 경로
}

[Code 6] We made it possible to restore the receiving path through a configuration change and restart alone, without redeploying the image.

There was also a point we nearly overlooked while designing the toggle. We had changed the lookup of sending metadata to prioritize Inbox, but when the toggle is turned off, re-received items update only Redis, resulting in a reversal in which responses use values from old Inbox rows. We also shared the same toggle at the lookup point so that, when it is off, only Redis is consulted. We learned that whenever a toggle is introduced, every point that references it must be reviewed together.

5. Verification Results

We performed four rounds of real-traffic verification in the development environment and one in staging, measuring the figures directly with logging and tracing tools.

• Receiving response time — Reduced from the previous 1.4–2.2 seconds to a median of 33 milliseconds, an approximately 50-fold improvement. The legacy transaction occupancy issue was fundamentally resolved.

• Full completion — All 111 messages were completed in 38 seconds, and all 273 messages in 94 seconds, with zero failures and zero fallback interventions. The throughput was approximately three messages per second, and we also confirmed that partitioning and consumer concurrency are the levers to use if scaling out becomes necessary.

• Reduction in external lookups — After introducing the cache, legacy lookups decreased by approximately 98% relative to the number of messages (approximately five cache-miss lookups when processing 111 messages).

• Validation of the failure-notification path — We verified through real traffic that 29 cases with unregistered forms transitioned to DEAD after five retries and were returned to the legacy system together with the reason.

• Validation of automatic failure recovery — As it happened, the staging verification coincided with a brief interruption to the legacy database. All 273 messages were initially accepted successfully, but every message failed in the first round. After the database recovered, batch re-publication resulted in all 273 messages recovering automatically within 3.5 minutes (zero final failures). Since the intended design worked during an actual failure rather than only in a planned scenario, this was personally the most meaningful result of the transition.

6. Retrospective — What to Carry Forward

• When the meaning of a response changes, we must trace its impact all the way through. The change from "Registration complete" to "Reception complete" involved only a few lines of code, but it gave rise to two major requirements: establishing a failure-notification path and ensuring that draining precedes rollback. I developed the habit of asking first whether the meaninghas changed, rather than focusing on the wording of the contract.

• Idempotency is better achieved by moving boundaries than by adding more guards.Instead of attaching guards in four places, we atomized the operation in the data-owning service. As a result, all the guards disappeared, and the system remained safe even when new steps were added.

• The recovery path must converge on the same code as the normal path to be reliable.Because we decided to have the batch republish only events instead of processing them directly, the recovery path worked without any additional validation even when it was first executed during an actual incident.

• Bottlenecks should be addressed only after they have been instrumented.Without looking at the traces, we would not have known that one of two queries that appeared to have the same shape was 15 times slower.

• Including rollback procedures in the design reveals flaws in the design.While writing the rollback procedure, we discovered the constraint that "cases for which an acknowledgment response had already been sent had no way to be processed in the old version." This enabled us to prepare a better first-line measure: a toggle.

There are still some remaining tasks. There is no way to resend a response when the reply fails, and because the original text contains personal information in plain text, we need to finalize the retention period for closed cases and the cleanup batch interval. We are managing both items as follow-up work.

conley