1. Overview
This article summarizes the optimization experience related to mass request processing in a first-come, first-served application system, such as course registration or reservation requests, centered around DB transactions. The system begins right after the application starts.Focus on tens of thousands of requests in a short timeIt has a characteristic that allows it to be. Users must repeatedly check, apply, and cancel while satisfying various conditions such as maximum credits, the number of applicants allowed, and reservation limits. Therefore, database transaction processing to maintain data integrity is very important.
In a real operational environment, approximately 20,000 users use the service simultaneously, 5 to 10 minutesat about 80,000 to 100,000A level of application and cancellation requests has occurred. In such an environment Application information generation and applicant count aggregation data updateThis was the core transaction processing target.
2. Initial Implementation and Performance Issues
After the project development is completed, use Apache JMeter under conditions similar to the production environmentPerforming load testingCompleted. In the initial implementation, both the application API and the cancellation API were active from the start to the end of the API.Transactions applied across the entire scopehave occurred.
However, there were many prerequisites to determine eligibility for application, and performance issues with the SQL migrated from the legacy system also existed. As a result, the API processing time increased, and the transaction retention time also grew. Consequently, the connection to the database was maintained for a long time, causing a shortage of connection poolto occur.
In situations where a large number of requests arise, the waiting time to acquire connections from DBCP increased, and some requests failed due to timeouts.There was an issue where the normal response rate did not meet expectationsThe ultimate goal is for all requests to receive a normal response.
3. Performance improvement process
In the first improvement phase, DB indexes were addedand SQL tuningwas carried out. This improved the lookup speed, and the API processing time decreased. However, the issue of connection pool shortage was still not resolved.
In the second phase, a service in an MSA environment Increase the number of PodsScale-out has been performed. Ultimately, about 30 Pods were operated.The throughput increasesHowever, There is a bottleneck in the database connectionThe request failure issue has not been completely resolved because it has not been done.
In the third step, Increase the number of connections in HikariCP from the default of 10 to 100As a result, the number of requests processed simultaneously in WAS increased, but the memory usage skyrocketed, leading to an Out Of Memory error.
In the fourth step, TomcatIncrease heap memory from 256MB to 4GBThe OOM issue has been resolved, but the connection shortage phenomenon has continued to occur. However, overall The failure rate has decreased somewhatperformed.
4. Balancing WAS and DB Resources
The essence of the problem has been confirmed to be not simply a lack of processing power but rather the way database connections are utilized. When the DB connection request becomes a bottleneck, k8s should switch the corresponding POD to an unavailable state to resolve the bottleneck; however, the Tomcat WAS was able to continuously receive requests, causing request failures.
Accordingly, processing requests concurrently in Tomcat WASChange the default number of request threads from 200 to 50was reduced to, The number of connections in DBCP is reduced from 100 to 50Re-adjusted.
Through these changes The issue of insufficient DB connections has been significantly alleviatedIt has been done. However, at the same time The number of requests that can be processed is limitedAs a result, another performance limitation has occurred.
Additionally, there were constraints in the integrated database environment. Since multiple systems were sharing a single physical database, the available resources were limited.The maximum number of DB connections is limitedTherefore, simply increasing the number of connections did not solve the problem.
5. Request Control and Transaction Optimization
To ensure operational stability, we are using the existing legacy system.Reintroducing the Request Waiting Solution (NetPanel)We were able to control the sudden traffic surge through this and address the issues that occurred during operation.The mass failure issue has disappeared.
After that, Minimize transaction scopeThe operation was carried out. Among the various conditions for determining eligibility, the data retrieval tasks that do not require real-time changes and are not subject to real-time changes have been moved outside the transaction scope. This has reduced the overall occupancy time of the DB connection.The number of requests that can be processed has increasedCompleted.
6. Deadlock problem and solutions
A new issue has arisen after transaction optimization. In the process of updating the applicant count aggregation information, the select for update query, that is pessimistic lockI was using it, but a deadlock occurred due to a surge of requests for the same record.
In particular, a single transaction A situation that requires additional DB connectionsIf the connection pool is insufficient, a deadlock may occur. To resolve this, it is necessary to analyze the maximum number of connections required per request and Determining the appropriate connection pool sizewas done.
DB Pool Size = T × (C - 1) + 1Here, T is the maximum number of threads in WAS, and C is the maximum number of database connections required for a single request. After adjusting the settings based on this formula,The deadlock phenomenon has disappeared.
7. Response to the issue of exceeding the number of applicants
In a bulk request environment, applyThe phenomenon of applications being completed after exceeding the limitIt has occurred. To resolve this, based on the moment immediately after the application is completedVerify once againThe operation has been performed. It was a method to check if the limit has been exceeded on a first-come, first-served basis, so it could be checked regardless of the timing.
Validation result limit number of peopleif exceededis the transactionRollback or cancel the applicationWe have maintained consistency in the way we do this. This has allowed us to ensure the accuracy of the final data.
8. Transitioning from pessimistic locking to optimistic locking
The performance analysis results show that pessimistic lock is key in a high-volume request environmentActing as a bottleneck elementwas doing. Accordingly, a pessimistic lock-based structure Changed to an Optimistic Lock-based structurehas been done.
To apply optimistic lockModification time attribute version informationIt has been utilized. The version is configured to increase every time the modification time of the entity changes, and when a conflict occurs, an OptimisticLockException is thrown.
AlsoSpring Retry featureusing In the event of a collisionException occurred as a conditionAutomatic Retryhas been configured to perform. The retry interval is set to less than 0.5 seconds, and it has been implemented to retry up to 10 times.
After this structural change, the efficiency of database connection usage has greatly improved. The deadlock issue has virtually disappeared, and the connection shortage has also been resolved. Ultimately, the number of WAS threads is 50, the number of DB connections is 20 it was possible to operate stably even at this level.
9. Conclusion
This case shows that simply increasing server capacity or connection count cannot solve the problem of handling large requests. It is important to accurately analyze the actual bottlenecks, minimize transaction scopes, and choose appropriate locking strategies.
In particular, converting from pessimistic locking to optimistic locking and adopting a retry strategy was one of the most effective improvement measures. Additionally, it was confirmed that balancing the database connection pool, the number of WAS threads, and the transaction scope is a key factor in ensuring the stability and performance of systems handling large requests.
Through this experience, important elements that can be applied to the design of various high-traffic systems such as course registration, reservation systems, event applications, and first-come-first-served sales were identified.
Daniel(K)