Kubernetes OOMKilled Traps: How to Set Memory Limits That Never Crash Your Pods

You push your code to production on Friday afternoon. Twenty minutes later, alerts start firing. Your pod status changes to CrashLoopBackOff, and the restart reason says OOMKilled with exit code 137. Your container used more memory than Kubernetes allowed, and the Linux kernel stepped in to terminate it without warning.

Memory problems in Kubernetes are tricky because memory is a hard resource. If your app needs more CPU than its limit, Kubernetes throttles the CPU and your app runs slower. If your app needs more memory than its limit, the system kills it on the spot. In this guide, we will break down why containers get killed, how Kubernetes handles memory under the hood, and how to set resource limits so your pods stay online.

What Does OOMKilled Really Mean?

OOM stands for Out Of Memory. When you run a pod, Kubernetes creates Linux control groups (cgroups) on the worker node. These cgroups track every byte of memory your processes allocate.

There are two distinct scenarios that trigger an OOMKilled event:

• Container Limit Hit: Your container crossed the limits.memory value you defined in your deployment manifest. The cgroup kills the offending process immediately. The node itself still has plenty of free RAM, but your container is punished for breaking its boundary.
• Node Pressure Hit: The entire worker node ran out of physical memory. The Linux Out-Of-Memory Killer (OOM Killer) wakes up and starts killing pods to protect the host operating system from freezing. It picks pods based on their Quality of Service (QoS) tier and their oom_score_adj score.

You can quickly check which event happened by running this command in your terminal:

kubectl describe pod <your-pod-name> -n <your-namespace>

Look at the Last State section. If you see Exit Code: 137 and Reason: OOMKilled, memory was the culprit.

The Huge Trap: Working Set Memory vs. kubectl top

Most engineers make their first mistake by trusting kubectl top pod. They run the command, see that a container uses 300 Megabytes, and assume setting a limit of 500 Megabytes is safe. Three hours later, the pod crashes.

Here is why: Linux divides memory into multiple categories. The two most important ones are:

• Anonymous Memory (RSS): Memory allocated on the heap and stack for process variables, objects, and runtime data. Linux cannot release this memory unless the process frees it or writes it to swap.
• Page Cache: File contents cached in memory to speed up disk reads and writes. Linux can release this memory instantly if something else needs space.

Kubernetes calculates your usage using a metric called working_set_bytes. This formula is:

working_set_bytes = total_memory_used - inactive_file_cache

If your application reads files, handles large uploads, or writes logs to disk, your page cache spikes. If the kernel cannot reclaim that cache fast enough when your application requests more heap memory, your working set hits the limit, and the cgroup fires the kill signal.

How to Stop Language Runtimes From Blowing Up

Different programming languages handle memory allocation in very different ways. Putting a language inside a container without configuring its runtime is the fastest route to an OOM crash.

The Node.js Memory Trap

By default, Node.js limits its V8 heap to around 1.4 Gigabytes on 64-bit systems, regardless of whether your container limit is 512 Megabytes or 8 Gigabytes. If your container limit is 500 Megabytes, Node.js does not know that. It will gladly grow toward 1.4 Gigabytes until the cgroup kills it.

You must tell Node.js how much memory it can safely use with the --max-old-space-size flag:

# Leave roughly 25% of your container limit for native bindings and system overhead
node --max-old-space-size=384 server.js

The JVM Memory Trap

Modern versions of Java (Java 11 and newer) detect container limits through cgroups. However, older builds of Java 8 do not. Even with newer Java versions, the heap is only part of the story. The JVM also allocates memory for Metaspace, thread stacks, and garbage collection overhead.

Never set the Java heap to 100% of your container limit. Use container-aware flags:

java -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -jar app.jar

The Go Memory Trap

Go uses a runtime garbage collector that triggers based on a ratio of new memory to live memory (via GOGC). In older Go versions, this caused sudden spikes that breached container limits. If you use Go 1.19 or newer, set the GOMEMLIMIT environment variable:

env:
- name: GOMEMLIMIT
  value: "450MiB"

This tells the Go garbage collector to work harder when memory usage gets close to 450 Megabytes, preventing container kills.

Setting Requests and Limits: The QoS Secret

Kubernetes sorts every pod into one of three Quality of Service (QoS) classes. This class determines who gets killed first when a worker node runs low on RAM:

• BestEffort: You set no requests and no limits. These pods are the first to die when the node faces memory pressure.
• Burstable: You set requests lower than limits. The scheduler places the pod based on requests, but the pod can grow up to its limit. If the node runs out of RAM, Burstable pods get killed before Guaranteed pods.
• Guaranteed: You set requests equal to limits for both CPU and memory. These pods are the safest and get killed last.

For your core production microservices, make memory requests equal to memory limits. This gives your pod Guaranteed status for memory allocation and guarantees that the node actually has the RAM reserved for your container.

The Production-Ready Pod Template

Here is an example deployment manifest demonstrating safe container resource limits, runtime tuning, and safety margins:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      containers:
      - name: api
        image: payment-api:v1.2.0
        env:
        # Tune the runtime to stay safely below the 512Mi limit
        - name: GOMEMLIMIT
          value: "420MiB"
        resources:
          requests:
            # Scheduled based on realistic steady-state usage
            memory: "512Mi"
            cpu: "250m"
          limits:
            # Memory request equals limit to prevent eviction under node pressure
            memory: "512Mi"
            # We omit CPU limit or set it high to prevent CPU throttling
            cpu: "1000m"

How to Find Your Real Memory Target

Do not guess your resource numbers. Open your Prometheus or Grafana dashboard and inspect this specific query over a 7-day period:

container_memory_working_set_bytes{container="api"}

Take the 99th percentile of this value during peak traffic hours. Add a 25% safety margin on top of it. That combined number is your target for both requests.memory and limits.memory. If your app spikes beyond that number under high load, your application has a memory leak, and it is time to profile your code rather than blindly raising limits.

Summary Rules to Stop OOMKilled Loops

Follow these simple rules whenever you deploy to Kubernetes:

• Always base limits on container_memory_working_set_bytes, not plain RSS or simple averages.
• Set memory requests equal to memory limits for critical production services to prevent eviction during node pressure.
• Always configure your language runtimes (Node.js, Java, Go) to keep their heap sizes 20% to 25% lower than the container memory limit.
• Never leave memory limits undefined; an unconstrained container with a memory leak can knock an entire node offline.