1. Introduction: Challenges of Backfill Architecture for Large Data Transfers
In modern distributed system architecture, the reliability of data pipelines is directly linked to the survival of services. In particular, time series data generated in IoT, finance, or real-time monitoring systems often requires throughput of tens of thousands of records per second. In such environments, the backfill process of resending failed data without omission due to system failures, network outages, or temporary paralysis of downstream databases is a very challenging task for backend engineers.
One common mistake many developers make is simply re-sending failed data by looping it or directly pushing it into a public message queue with real-time traffic flowing. However, in a large-scale operational environment, such an approach can trigger cascading failures that paralyze real-time normal services.
This technical document details the experience and optimization strategies of seamlessly combining the ultra-fast messaging infrastructure NATS, the high-performance non-blocking data structure ConcurrentLinkedQueue in Java, and Sping's custom asynchronous thread pool (@Async) to reliably and efficiently resend millions of accumulated time series data while perfectly protecting real-time traffic.
2. Limitations of Existing Architecture and Technical Pain Points
The initial system before enhancement was a simple structure that handled real-time traffic and backfill traffic simultaneously within a single messaging pipeline. This structure did not reveal its flaws in small-scale test environments, but it caused three critical bottlenecks the moment large-scale backfill operations were triggered in operational environments.
2.1. Real-time Service Paralysis Due to Resource Exhaustion
Real-time user requests and backfill requests shared resources from a public HTTP thread pool and a single message queue. As millions of backfill data flooded in, the backfill worker threads monopolized the entire CPU, memory, and thread resources. Consequently, critical requests that needed to be processed in real-time were left waiting indefinitely without getting thread allocations, resulting in catastrophic connection timeouts.
Figure 1. Resource exhaustion issue arising when real-time requests and backfill requests share the same resources
2.2. Load Spike from Time-based Scheduling
When loading backfill data piled in the queue into the downstream database (PostgreSQL), an initial time-centric logic like "run the scheduler every 10 seconds to completely empty the queue" was used. This increases unpredictability when the volume of incoming data is irregular. If data explodes during a specific 10-second window, tens of thousands of queries could be dumped into the database all at once during the next scheduling time, causing the connection pool to crash and DB CPU utilization to hit 100%, frequently resulting in traffic spikes.
2.3. Resource Wastage Due to Thread Blocking
In the process of confirming the completion of backfill data resending and triggering the final statistics aggregation process, a synchronous blocking mechanism like Thread.sleep() or CountDownLatch was used under the guise of "waiting until all data has been processed." This caused the threads in a waiting state to hold onto memory, leading to frequent context switching and drastically reducing overall system resource efficiency.
3. Four Architectural Enhancement Strategies for Isolation and Optimization
To overcome the above three limitations, we introduced the Bulkhead pattern to completely separate the overall structure from the infrastructure layer to the application's data structures and thread models.
3.1. Infrastructure and Data Structure Layer Isolation: NATS and ConcurrentLinkedQueue
The first step was a thorough separation of traffic routing. We separated real-time data topics and backfill-only topics at the infrastructure (NATS) level, and registered a dedicated memory buffer queue as a @Bean inside the Java application to safely accommodate only backfill data.
@Configuration
public class QueueConfig {
@Bean
public Queue<TimeSeriesVo> backfillTimeSeriesVoQueue() {
// 다중 스레드 환경에서 Lock-Free 고성능을 보장하는 논블로킹 큐 선택
return new ConcurrentLinkedQueue<>();
}
}The key here is the choice of ConcurrentLinkedQueue. Conventional LinkedBlockingQueue or ArrayBlockingQueue requires producer (Producer) and consumer (Consumer) threads to acquire locks when accessing the queue, leading to wait bottlenecks when there is high contention. In contrast, ConcurrentLinkedQueue operates without locks, based on the CPU-level atomic operation of the CAS (Compare-And-Swap) algorithm. Even if the NATS consumer, with its extremely fast response time, pushes tens of thousands of data per second, it serves as a buffer zone that safely receives data into memory non-blocking, without colliding with worker threads.
Figure 2. How multiple NATS consumer threads load data into the queue without locks using the CAS algorithm
3.2. Thread Layer Isolation: @Async("backfillTimeSeriesPool")
Even if we safely receive data through NATS and memory queues, if the workers that extract it to perform actual business logic and load it into the DB share the main thread pool of the web server, the issue of resource depletion mentioned earlier will not be resolved.
Therefore, we utilized Spring's @Async mechanism to establish a dedicated custom Thread Pool specifically for backfill tasks.
@Configuration
@EnableAsync
public class AsyncConfig {
@Bean(name = "backfillTimeSeriesPool")
public Executor backfillTimeSeriesPool() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5); // 기본 유지할 스레드 개수
executor.setMaxPoolSize(10); // 트래픽 폭발 시 최대 확장 스레드 개수
executor.setQueueCapacity(500); // Task 대기 큐 용량
executor.setThreadNamePrefix("backfill-task-");
executor.setRejectedExecutionHandler(
new ThreadPoolExecutor.CallerRunsPolicy());
executor.initialize();
return executor;
}
}This isolated thread pool is declared and injected into the asynchronous worker component handling backfill.
@Component
public class BackfillWorker {
private final Queue<TimeSeriesVo> backfillTimeSeriesVoQueue;
private final TimeSeriesRepository timeSeriesRepository;
public BackfillWorker(
Queue<TimeSeriesVo> backfillTimeSeriesVoQueue,
TimeSeriesRepository timeSeriesRepository) {
this.backfillTimeSeriesVoQueue = backfillTimeSeriesVoQueue;
this.timeSeriesRepository = timeSeriesRepository;
}
@Async("backfillTimeSeriesPool")
public void processBackfill() {
// 공용 HTTP 스레드를 전혀 오염시키지 않고, 백필 전용 스레드(backfill-task-) 내에서만 안전하게 루프 구동
List<TimeSeriesVo> chunkList = new ArrayList<>();
while (!backfillTimeSeriesVoQueue.isEmpty()) {
TimeSeriesVo vo = backfillTimeSeriesVoQueue.poll();
if (vo != null) {
chunkList.add(vo);
// 3.3 섹션에서 서술할 크기 기반 청크 처리 로직이 여기에 위치함
if (chunkList.size() >= 1000) {
timeSeriesRepository.saveAll(chunkList);
chunkList.clear();
}
}
}
if (!chunkList.isEmpty()) {
timeSeriesRepository.saveAll(chunkList);
}
}
}Through this design, even if backfill tasks run heavily in the background for hours, the http-nio- threads processing real-time user requests remain completely isolated, maintaining a high-performance response time of under 0.1 seconds peacefully.
3.3. Backend Flow Control: Size-based Chunk Extraction
To solve the indiscriminate DB dropping problem of time-based schedulers, we changed the paradigm to Size-based Extraction to maintain an optimal speed within the system's capacity limits.
When a thread retrieves data from the queue, it relies solely on the number of data (Chunk Size) instead of the passage of time. We determine the optimal bulk unit that the database (PostgreSQL) can accept most reliably in terms of index updates and connection load, and as data fills the queue, we continuously push data into the DB in designated size chunks (poll()).
Through this flow control, even when backfill data surges in the hundreds of thousands, we prevent the worst-case scenario of the database freezing all at once, control the upper limit of the load within the system threshold, and secure structural stability to gradually complete bulk inserts.
Figure 3. Time-based scheduling causes load spikes, but size-based chunk extraction maintains consistent throughput
3.4. Asynchronous Resource Optimization: Control Flags and isEmpty() Non-blocking Completion Verification
After the backfill data has been fully retransmitted, the system must perform aggregation operations to ensure final data consistency. We stripped away the existing thread-blocking method and combined Control Flag and queue state verification to complete a perfect asynchronous flow.
When the end-of-stream flag indicating the end of data transmission reaches the consumer through the NATS protocol, the system switches the state flag to COMPLETING. Subsequently, the worker threads check the status of the main queue in a non-blocking manner without sleeping or waiting.
public void checkAndTriggerAggregation() {
// 스레드를 재우는 오버헤드 없이, 락 프리로 큐의 공백 상태를 체크
if (isEndOfStreamSignalReceived && backfillTimeSeriesVoQueue.isEmpty()) {
// 대기 시간 0초 만에 즉시 최종 집계 프로세스 비동기 트리거
aggregationService.triggerFinalAggregation();
}
}As soon as backfillTimeSeriesVoQueue.isEmpty() returns true, the final aggregation logic is triggered without any CPU delay or thread idle state.
In addition, to defend against some race conditions where the order of 'backfill data request' and 'successful response receipt' may be reversed due to network latency characteristic of a distributed asynchronous environment, we added logic to first register certain normal response events in an in-memory whitelist and cross-verify them, thus reducing the possibility of data loss.
4. Conclusion: The lesson from the harmony of infrastructure and software
The most valuable lesson learned from this advanced project is that "no matter how fast a high-performance distributed infrastructure (NATS) is introduced, if the data structure, thread model, and flow control strategy inside the backend application are insufficient, the bottleneck moves to another place and ultimately occurs again."
The chain of topic separation at the infrastructure level, @Async custom thread pool isolation through Spring configuration, lock-free buffering through ConcurrentLinkedQueue, and size-based chunk control only completed the true meaning of a large-scale data pipeline when they interconnected organically.
The experience of this architectural improvement, which dramatically reduced the system's resource consumption while perfectly ensuring data integrity in large-scale retransmission environments, serves as a foothold to fine-tune trade-offs amidst limited computing resources and secure robust system stability that remains unaffected even in extreme traffic situations.
Pancake Maker