1. Overview
When dealing with large amounts of data, how the retrieved data is processed and presented on the screen can sometimes have a greater impact on performance than retrieving the data itself.
In an actual project, I developed a screen that had to process tens of thousands of records and display them in a Grid. Rather than simply outputting the data received from the server as is, the screen had to connect separate sets of data based on specific criteria and display the result of processing multiple values in a single Cell.
During the early stages of development, performance issues were not particularly apparent because testing was performed with a relatively small amount of data. However, as the amount of actual data increased to tens of thousands of records or more, the processing time required to reflect the data in the Grid became longer, and the screen would sometimes freeze when other actions were performed while the data was loading.
At first, I thought the cause was the rendering performance of the Grid itself. However, after examining the processing flow, I found that the repeated data searches and processing performed to create Cell data were also incurring considerable costs.
In this article, based on that experience, I will summarize how I improved the performance issues that occurred while processing large amounts of data and how these issues relate to Main Thread Blocking and Long Task.
2. Requirements and Problem Situation
The requirements and problem situation were as follows.
Requirements
-
Display data using scrolling instead of pagination
-
Filtering functionality was required for more than several dozen columns (all data was required)
-
Display different background colors based on Cell values
Problem Situation
-
A long time was required to process and display the data after receiving it
-
The screen froze when other screen actions were performed while loading
3. Repeated Operations with Large Amounts of Data
In the initial implementation, Related Data, which was managed separately from Main Data, was combined at the Cell level.
In simplified form, it looked like this.
예시)
const columnDefs = [
{
headerName: 'Related Data',
valueGetter: params => {
const relatedItems = relatedData.filter(
item => item.targetId === params.data.id
);
return relatedItems
.map(item => item.name)
.join(', ');
},
},
];This was not a problem when there was little data, but as Main Data and Related Data both increased, the searches required to create Cell values also increased repeatedly.
In particular, in a Grid, Cell values may be recalculated depending on operations such as data updates, sorting, or filtering.
Ultimately, the problem was not simply that “there was a lot of data in the Grid.”
The problem was that as the amount of data increased, the searches and processing required to display each item on the screen were also being repeated.
4. Investigating the Problem from the Main Thread Perspective
In the browser's Main Thread, various tasks required not only for JavaScript execution but also for handling user events and rendering are performed.
Therefore, if JavaScript operations occupy the Main Thread for a long time, user input and subsequent rendering may also be affected.
At the time, the following tasks were required to receive the data and display it on the actual screen.
API Response
↓
Data search
↓
Data combination and transformation
↓
Grid data construction
↓
Grid Rendering
↓
Layout / Paint
Although each operation itself was not large, repeating them for tens of thousands of records increased the overall JavaScript execution cost.
This issue is also related to Long Task. In browsers, a Task that runs for more than 50ms on the Main Thread is classified as a Long Task.
If user interaction occurs while a Long Task is running, event handling may be delayed until that task is completed.
Therefore, performance issues with large amounts of data had to be considered not only from the perspective of reducing the total processing time, but also from the perspective of reducing the amount of work performed at once on the Main Thread.
5. Preprocessing Data
First, I sought to reduce the repeated data searches performed when generating Cell data.
Previously, every time a Cell value was calculated, filter() was used to find data related to the current Row in Related Data.
If Main Data and Related Data contain N and M records, respectively, Related Data is repeatedly traversed while calculating each Cell value, which can result in an overall search cost of O(N × M).
To improve this, I preprocessed Related Data based on targetId and constructed a Lookup structure.
예시)
const relatedNameMap = new Map<string, string[]>();
relatedData.forEach( item => {
const names = relatedNameMap.get(item.targetId) ?? [];
names.push(item.name);
relatedNameMap.set(item.targetId, names);
});6. Passing Processed Data to the Grid
After preprocessing Related Data, I changed the implementation to combine the result with Main Data and pass it to the Grid.
Previously, the required data was searched for and the values were combined when the Cell was created.
Cell Creation
↓
Related Data Exploration
↓
Data Combination and Transformation
↓
Cell Rendering
With this approach, Cell values could be recalculated whenever the Grid data was updated, sorted, or filtered.
After the improvement, we included the necessary values in the Main Data before passing it to the Grid, using the relatedNameMap created earlier.
예시)
const gridData = mainData.map( row => ({
...row,
relatedNames: relatedNameMap.get(row.id)?.join(“, ”) ?? “",
}));
const columnDefs = [ {
field: 'relatedNames',
headerName: 'Related Data',
},];This allowed us to eliminate the repeated data exploration and processing from the rendering process every time a Cell was created.
In particular, even when data updates, sorting, or filtering occurred, it was no longer necessary to explore the Related Data again to construct Cell values. The Grid could focus on displaying the already-calculated values on the screen.
As a result, the significance of this change was not simply that it improved Cell rendering speed, but that it separated repetitive operations from the rendering process and reused the calculation results.
7. Problems That Cannot Be Solved by Virtualization Alone
When displaying tens of thousands of data records in a Grid, creating every Row as an actual DOM element is inefficient. The AG Grid we used also rendered only the Rows needed around the current Viewport through Virtualization, so the range of elements created in the actual DOM was limited.
However, applying Virtualization does not solve every performance issue that occurs when processing large amounts of data. Virtualization primarily reduces the range of the DOM rendered on the screen; it does not reduce the JavaScript operations involved in constructing the data passed to the Grid or calculating the values displayed in Cells.
At the time, processing tens of thousands of data records required exploring and combining data that was separated across different sources. Therefore, the key area for improvement was reducing the repeated exploration and processing costs involved in constructing Grid data or calculating Cell values, rather than reducing the rendering range of the Grid itself.
8. Summary
Looking back on this experience from the perspective of the Main Thread, the improvements we applied were ultimately aimed at reducing the work that the Main Thread had to perform.
|
Problem |
Improvement Direction |
|---|---|
|
Repetitive data exploration |
Constructing a Lookup structure |
|
Data processing at the Cell level |
Precomputation |
|
Repeating the same calculations |
Reusing calculation results |
Even after applying these methods, if the calculations themselves are still heavy, you can also consider splitting the Task or using a Web Worker.
However, rather than applying a complex solution from the beginning, I thought it was important to first identify which tasks were being repeatedly performed on the current Main Thread.
This is because simply eliminating unnecessary exploration and duplicate calculations can reduce the overall amount of work that the Main Thread needs to process.
9. Conclusion
Through this experience, I learned that when handling large amounts of data, it is necessary to examine not only rendering performance but also which operations are being repeated before the data is displayed on the screen. I also learned that performance optimization is not just about reducing the execution time of a particular function; it is important to design the data processing flow so that unnecessary calculations are not repeated in the first place.
Jung