Practical Application of Debezium CDC

1. Introduction

The problem I faced while participating in the Medical System MSA Transition Project was data duplication. The new MSA system (PostgreSQL) and the existing legacy system (Oracle) had to be operated side by side, and there was a requirement to maintain data consistency in real-time while both systems were simultaneously serving.

Batch synchronization caused delays of several minutes, making it unsuitable for the healthcare domain, the DB trigger method added operational burdens to the legacy Oracle, and application dual writes made it difficult to ensure transaction atomicity. After reviewing various methods, we decided to adopt the CDC (Change Data Capture) method, which directly reads the transaction logs from the source DB and propagates them asynchronously.

2. Architecture: Debezium and custom synchronization application

Debezium is an open-source CDC platform that directly subscribes to the WAL (Write-Ahead Log) of PostgreSQL, publishing INSERT/UPDATE/DELETE events in real-time to Kafka. Since it reads the WAL logs without triggering the source DB or changing the code, it did not require touching legacy systems, and the ability to continue from the last processed log position (LSN) even after the connector restarts aligned well with the project's requirements.

[Figure 1] Debezium Official Architecture — Source: debezium.io/documentation/reference/3.5 (Apache License 2.0)

There were three main reasons for directly developing a custom Spring Boot application (Data-Sync) instead of using the standard Sink Connector.

Schema transformation - The complex PK, non-standard column names, and type conversions (timestamptz → TIMESTAMP WITH LOCAL TIME ZONE, etc.) of legacy Oracle were difficult to handle with standard Connector settings alone.

Error handling - We needed to have different retry intervals and notification methods based on the type of error, and for this, the flexibility of custom code was necessary.

Consistency validation - There was also a requirement to asynchronously store the data hash after synchronization is complete in order to periodically verify consistency between the two systems.

3. Durability Verification: Fault Response Testing

After the design was completed, I wanted to check whether the system would not lose data even in the event of a failure before it was actually put into operation. I independently executed scenarios in a development environment Kubernetes cluster, verifying each time whether 20,000 records of data were input into PostgreSQL and successfully loaded into Oracle without any loss.

4. Conclusion: Limitations and Lessons Learned, and Future

◆ Limitations of Initial Design

The method of directly storing column information required for synchronization in the Data-Sync internal metadata table had clear limitations. Each time a column was added to a specific service, changes were needed in Data-Sync as well, compromising the service independence that MSA aims for in the data synchronization layer. To address this issue, we are considering the introduction of Confluent Schema Registry and Avro format. This is a feature already officially supported in Debezium 3.x, and by applying it, Debezium will automatically register the schema with the Schema Registry along with the events, allowing Data-Sync to interpret it dynamically, enabling each service to change the schema independently of Data-Sync.

◆ Key Changes in Debezium 3.x (2024-2025)

References

[1] Debezium official documentation, debezium.io/documentation/reference/3.4

[2] Debezium 3.1~3.4 Release Notes, debezium.io/blog

[3] Confluent Schema Registry, docs.confluent.io/platform/current/schema-registry

Juno