Overview
When working on a project, you may encounter OOM or OOMKilled. Especially in a Kubernetes environment, it is common to face situations where "the Pod was restarted due to OOMKilled."
OOMKilled may seem like a simple server restart, but if left unaddressed without identifying the cause, it can repeatedly occur in high service load situations, leading to critical failures.
This document summarizes the investigation and review of what OOMKilled is, why it occurs, and how to analyze and resolve it based on encounters during the project's progress.
Difference between OOM and OOMKilled
Both OOM and OOMKilled are issues related to memory allocation. However, they differ in terms of the entity that triggers them and their operational mechanisms.
OOM (Out Of Memory Error) occurs within the JVM. When the heap memory is full and no more objects can be allocated, the JVM throws a java.lang.OutOfMemoryError and terminates the process. In other words, the JVM recognizes its own limit and throws an error.
OOMKilled occurs outside of the JVM, in container environments like Kubernetes. The moment the memory limit set for a container is exceeded, Kubernetes forcefully terminates that process. Since it is killed externally before the JVM can recognize the error, it is harder to identify the cause as no separate error logs are left, unlike OOM.
|
Classification |
OOM |
OOMKilled |
|---|---|---|
|
Entity that triggers the occurrence |
Inside JVM |
Kubernetes (cgroup) |
|
Cause |
Heap memory exhaustion |
Container limit exceeded |
|
Error log |
OutOfMemoryError log retained |
Forced termination without log |
|
Exit code |
1 |
137 |
|
Target memory |
Heap area |
Total of Heap + Non-Heap + Off-Heap |
When an OOMKilled occurs, the Pod is automatically restarted according to Kubernetes' restart policy. If it happens during times of low traffic, it may not be a big issue. However, if OOMKilled occurs under heavy load conditions, the situation can become serious. Depending on the Pod's status, the service may be interrupted during restart, and if traffic spikes immediately after the restart, memory can quickly fill up, resulting in repeated OOMKilled incidents. Ultimately, as CPU and memory loads increase, it can lead to critical failures for the entire service.
Distinction of memory
When referring to memory related to OOMKilled, it is important to understand not only Heap memory allocation but also Container, JVM, and other more specific memory details.
Container memory
One important factor to consider in a Kubernetes environment is the range of memory settings. The memory limit set for a container applies not just to the JVM but to the entire container.
In other words, it is essential to recognize the relationship where the container memory is greater than the JVM memory.
Container 메모리
│
├── JVM 프로세스 메모리
│ ├── Heap
│ ├── Non-Heap
│ └── Off-Heap (Direct Buffer)
│
├── OS 및 시스템 라이브러리
└── 사이드카 Container (Istio sidecar 등) 등 기타…JVM memory
JVM memory is broadly categorized into three areas.
Heap area
This is the space where objects created by Java applications are stored. It is also the area targeted by GC (Garbage Collection).
|
Metrics |
Description |
|---|---|
|
Heap Used |
Memory currently in use |
|
Heap Committed |
Memory that the JVM has pre-allocated from the OS. It is not immediately released after GC. |
|
Heap Max |
The maximum heap size available for the JVM. |
The relationship between the three metrics is always Used ≤ Committed ≤ Max, and Committed is cleared during a Full GC.
Non-Heap Area
An area for JVM's own operation, not subject to GC.
|
Area |
Description |
|---|---|
|
Metaspace |
Stores class metadata and static variables. |
Metaspace can theoretically grow indefinitely if -XX:MaxMetaspaceSize is not set.
Off-Heap Area (Direct Buffer)
An area that is neither Heap nor Non-Heap, memory allocated directly by the JVM from the OS. It is primarily used for NIO-based file processing or network I/O.
Since it is not subject to GC, it can accumulate if not explicitly freed, and can grow indefinitely if a limit is not set with -XX:MaxDirectMemorySize.
Relationship between Container Memory and JVM Memory
It may seem fine if the JVM's Heap size is set lower than the Container Limit. However, since not only Heap but also Non-Heap and Off-Heap are managed together within the Container Limit, all three areas must be carefully calculated by summing them up. Especially, services that process many files depending on the system's nature may use Off-Heap a lot, while services that load many classes may use Non-Heap more, so it is essential to understand one's system characteristics thoroughly before designing memory, rather than just looking at Heap size.
In a system that uses 200MB for Non-Heap, 100MB for Off-Heap, and 50MB for other JVM-related memory, if the Container Limit is set to 1Gi and -XX:MaxRAMPercentage=80, the following situation occurs.
Heap Max (80%) : 819MB ← JVM이 사용 가능하다고 인식하는 Heap 한도
Non-Heap : 200MB
Off-Heap : 100MB
JVM 외 기타 : 50MB
─────────────────────────
Heap 외 합계 : 350MB
실질적으로 Heap에 허용되는 메모리 : 1,024MB - 350MB = 674MBThe JVM recognizes it can use up to 819MB of Heap and continues to allocate memory without a Full GC. However, since Kubernetes sums up Heap + Non-Heap + Off-Heap + others and compares it to the Container Limit, the moment Heap usage exceeds 674MB, it surpasses the Container Limit of 1Gi, and Kubernetes forcibly terminates the process before the JVM realizes it. This is what is known as OOMKilled.
JVM Settings
Below is a brief definition of the JVM options related to OOMKilled.
MaxRAMPercentage
-
Sets the maximum size of the heap as a percentage of the container limit. Setting it too low can waste memory, and setting it too high can lead to exceeding the container limit when combined with Non-Heap and Off-Heap consumption.
MaxMetaspaceSize
-
Limits the maximum size of Metaspace in the Non-Heap area. If not set, Metaspace can theoretically increase without limit. It is recommended to set a cap with some buffer based on current usage.
MaxDirectMemorySize
-
Limits the maximum size of Off-Heap (Direct Buffer). In services that heavily utilize NIO-based file processing or network I/O, if not set, the Direct Buffer can grow without limit and exceed the container limit. Setting this option also provides the additional benefit of logging OutOfMemoryError: Direct buffer memory when the limit is exceeded, making it easier to identify the cause.
OOMKilled server analysis case
Below is a summary of the memory status of the service where OOMKilled occurred, analyzed by AI based on the input. The OOMKilled issue was resolved after applying the final JVM option modifications.
Original notion link :
https://app.notion.com/p/vizend/Pod-OOMKilled-38b35bc54c13806699e8e3ae329bf38f
Memory status analysis
1. Container resource settings
|
Item |
Value |
|---|---|
|
Memory Limit |
1Gi (1,073 MB) |
|
Memory Request |
256Mi |
|
QoS Class |
Burstable |
2. JVM Memory Status (Current Time)
|
Area |
Value |
MB Conversion |
|---|---|---|
|
Heap Used |
376,595,248 bytes |
~359 MB |
|
Heap Committed |
512,229,376 bytes |
~488 MB |
|
Heap Max |
859,308,032 bytes |
~819 MB |
|
Non-Heap Used |
210,018,744 bytes |
~200 MB |
|
└ Metaspace |
127,576,696 bytes |
~122 MB |
3. Core Issue Analysis
Container Limit : 1,073 MB (1Gi)
────────────────────────────────────────
Heap Max (JVM 설정) : 819 MB
Non-Heap Used (현재) : 200 MB
────────────────────────────────────────
합산 최대 사용 가능 : 1,019 MB ← 한계에 매우 근접
나머지 여유 : ~54 MB ← 위험 수준Heap Max + Non-Heap of JVM alone accounts for about 95% of the Container Limit.
When adding the internal overhead of the JVM (Thread Stack, GC metadata, etc.), it exceeds 1Gi, resulting in OOMKilled.
4. Check GC Method
The Actuator response shows PS Eden Space, PS Old Gen, PS Survivor Space.
We recommend using G1GC.
5. Cause Summary
|
Cause |
Description |
|---|---|
|
Excessive Heap Max |
Exceeding limit just before 1,019MB with 819MB Heap + 200MB Non-Heap |
|
Inappropriate GC Method |
Parallel GC → Insufficient recognition of container memory |
|
JVM Option Not Set |
Assuming lack of container recognition options like -XX:MaxRAMPercentage |
|
QoS Burstable |
Request ≠ Limit → No memory guarantee, OOM priority |
MaxRAMPercentage Simulation Result
Compared and analyzed MaxRAMPercentage from the current 80% to 60%.
Heap Usage
|
Item |
80% |
70% |
65% |
60% |
|---|---|---|---|---|
|
Container Limit |
1,073MB |
1,073MB |
1,073MB |
1,073MB |
|
Heap Max |
858MB |
~751MB |
~698MB |
~644MB |
|
Non-Heap Measurement |
~200MB |
~200MB |
~200MB |
~200MB |
|
Overhead including Thread Stack |
~50~80MB |
~50~80MB |
~50~80MB |
~50~80MB |
|
Total Estimate |
~1,108~1,131MB |
~1,001~1,031MB |
~948~978MB |
~894MB |
|
Free |
~42~72MB |
~95~125MB |
~179MB |
Comparison by option
|
Settings |
Heap Max |
Free |
Evaluation |
Comparison |
|---|---|---|---|---|
|
Current 80% |
~858MB |
~15MB |
Risk |
Issues when using Non-Heap, Off-Heap additional resources |
|
70% |
~751MB |
~42~72MB |
Unstable |
Increase Container Limit with G1GC (1.5Gi) |
|
60% |
~644MB |
~149~179MB |
Recommended |
Apply G1GC |
|
50% |
~536MB |
~257MB |
Safe but Heap may be insufficient |
Exclude from review |
-
The safest approach is to set it to 60% and monitor it, raising it if necessary.
-
Non-Heap or Off-Heap 200MB is not a fixed value and can cause OOMKilled under load situations such as additional file uploads.
Recommended JVM options (final)
# 현재
-XX:InitialRAMPercentage=25
-XX:MaxRAMPercentage=80
-XX:+UseParallelGC
# 권장
-XX:InitialRAMPercentage=25
-XX:MaxRAMPercentage=60 # Heap ~644MB로 제한
-XX:+UseG1GC # Container 친화적 GC
-XX:MaxMetaspaceSize=192m # Metaspace 상한 설정 (현재 122MB 사용 중)
-XX:MaxDirectMemorySize=140m # Direct Buffer 상한 설정 (현재 108MB 사용 중)-
Heap Max: ~644MB
-
Non-Heap: ~200MB
-
Total: ~844MB → Provide a buffer within the Container Limit of 1Gi
sby