1. The Beginning of the Problem: Rising Storage Costs
As the service operation period increased, the MongoDB storage capacity also continuously expanded.
Due to business requirements, we couldn't delete the already created data, but keeping all the data in regular storage was becoming a significant cost burden.
In particular, the specific collections subject to the Online Archive application contained critical core service data, and as data accumulated continuously, Increased to about 1.2TB levelwas.
As a result, it was a situation where we had to reconsider not just the simple cost of storage, but the long-term data operation strategy itself.
Accordingly, we reviewed the plan to transfer data from a certain period to a separate storage, and decided to implement MongoDB Online Archive.
- Data that has been created for 365 days will be transferred to Online Archive. -
This criterion was more akin to an operational policy for optimizing storage costs rather than a value that was carefully designed based on query pattern analysis.
However, the Online Archive has the advantage of reducing storage costs, but it has characteristics that require considering a separate cost structure when accessing it.
That is,
What should be archived?
not only,
How to query the archived data?
It was indeed a situation that required thoughtful consideration together.
2. Online Archive design
The Archive policy is structured as follows.
Archive criterion: data that has elapsed 365 days based on the time field
Partition structure: dataId → time
The main query patterns of the actual service were as follows.
Full data retrieval for a specific dataId
Accordingly, to prioritize grouping the data of the same dataId, dataId was set as the preceding key and the time, which serves as the Archive criterion, was set as the trailing key.
This was aimed at aligning the actual query patterns with the Archive policy's consistency.
Moreover, in the Online Archive environment, considering the partition criteria for queries could potentially reduce Data Scanned, thus it was deemed an important design element from a cost optimization perspective.
3. Index strategy for Archive operation
In the Archive introduction process, not only was the Archive policy defined, but also the indexing was configured by considering the Archive operation process and the actual query patterns.
3-1. Index for identifying Archive targets
{ time: 1, dataId: 1 }Since it was necessary to continuously identify data that had passed 365 days after creation according to the Archive policy,
a separate index was created to efficiently identify Archive target data.
3-2. Index for Queries After Archive
The main query pattern of the service was full retrieval based on dataId.
The partition of the Online Archive was also structured as follows.
{ dataId: 1, time: 1 }Accordingly, to account for cases where Archive data is queried together through the unified endpoint, we created indexes in the same order as the partition structure.
This was done to maintain consistency between the actual query pattern and the Archive structure.
In summary, it is as follows.
|
Purpose |
Index |
|---|---|
|
Archive Target Identification |
{ time: 1, dataId: 1 } |
|
Unified Query After Archive |
{ dataId: 1, time: 1 } |
4. Existing Bulk Retrieval API
Our service had an API that returned all data for a specific identifier.
GET /data/{dataId}
This API was not a simple retrieval API.
It was necessary to perform data processing tasks that conform to the response specification, rather than simply returning the retrieved data.
MongoDB retrieval → response data processing → return to client
Also, the data distribution was not uniform.
|
Data size by dataId |
Count |
|---|---|
|
In general cases |
Dozens to thousands of cases |
|
Large volume cases |
Up to hundreds of thousands of cases |
In other words, even the same API required bulk data processing for specific dataIds.
5. Existing retrieval method
Previously, we were using an OFFSET-based iterative retrieval method.
For example, it had a structure like the following.
LIMIT 1000 OFFSET 0
LIMIT 1000 OFFSET 1000
LIMIT 1000 OFFSET 2000 ...There were no major issues in a typical MongoDB environment.
However, after the introduction of the Online Archive, it was necessary to reconsider whether the existing query method was appropriate in the Archive environment.
6. Reasons for re-evaluating OFFSET-based queries
In the Online Archive environment, the amount of Data Scanned can impact costs.
That is,
Increase in Data Scanned → Potential increase in query costs
This relationship had to be considered.
The existing OFFSET-based query repeatedly scans previous data to reach the desired position.
For example,
LIMIT 1000 OFFSET 90000A query like this requires skipping the previous 90,000 records to return results.
In a structure where large data is queried multiple times, such scanning costs occur repeatedly.
It was determined that in the Archive environment, such repeated scanning could lead to an increase in Data Scanned.
OFFSET → Repeated query → Occurrence of repeated scanning → Increase in Data Scanned → Potential increase in query costs
Ultimately, the existing OFFSET-based query strategy was re-evaluated.
7. Transition to Cursor-based queries
A Cursor-based query was chosen as the new query method.
I have configured the Stream feature of Spring Data MongoDB to consume data sequentially.
@Meta(cursorBatchSize = 3000)
Stream<DataDoc> streamByDataId(String dataId);I also applied cursorBatchSize together.
The reason for using batchSize in this case was to avoid loading large amounts of data into memory at once and to consume it in units of a certain size.
Cursor → batch unit query → sequential consumption → memory usage control
Compared to the previous structure, it is as follows.
OFFSET → repeated queries → repeated exploration Cursor → sequential queries → maintaining single query flow
In the Archive environment, I judged that this structure is more suitable from a cost optimization perspective.
8. Retrospective
Through this experience, I confirmed that the introduction of Online Archive is not just about reducing storage costs.
When the storage strategy changes, the query strategy should also be re-evaluated.
Especially in environments where the cost structure changes, the existing method cannot always be considered optimal.
In this case, Archive policy, index configuration, query patterns, and query strategies had to be considered together.
Storage cost optimization
→ Introduction of Online Archive
→ Partition design
→ Index composition
→ Consideration of query costs
→ Review of OFFSET
→ Transition to Cursor-based querying
Ultimately, this case was not just a story about applying Cursor.
The work that began for storage cost optimization led to considerations about query costs, and in that process, I was able to reevaluate existing query strategies.
The changes in infrastructure led to changes in the application's query strategies, allowing me to once again realize how closely storage strategies and query strategies are connected.
Reference
https://www.mongodb.com/docs/atlas/online-archive/
https://www.mongodb.com/docs/atlas/data-federation/billing/
https://www.mongodb.com/docs/manual/reference/method/cursor.skip/
yeop