Victor's room

Overview & Root Cause Summary: The error Docker container exited with code 137 indicates that the container was abruptly terminated by an operating system signal (128 + 9 = SIGKILL). In the vast majority of scenarios, this exit code is triggered by the Linux kernel’s Control Groups (cgroups) Out-Of-Memory (OOM) Killer when the containerized application’s memory consumption surpasses the hard limit allocated to the container (e.g. --memory or Compose mem_limit), or when the host machine completely runs out of physical RAM and swap space.

Understanding the Root Causes

• Container Memory Limit Breach (cgroups OOM): Docker enforces strict memory quotas defined via --memory or Compose resources. When resident memory (RSS) breaches this ceiling, the kernel cgroup killer terminates the primary process immediately.
• Unbounded Host Memory Starvation: The container has no configured limit, but the host machine’s total physical RAM and swap space are exhausted, forcing the host Linux OOM killer to sacrifice the container process.
• Runtime Heap Misconfiguration (Java / Node.js): Application runtimes unaware of container boundaries (e.g. legacy Java versions or unconfigured Node.js V8 engines) attempting to allocate heaps matching total host RAM rather than the container quota.
• Manual Process SIGKILL (docker kill / stop timeout): Executing docker kill or issuing docker stop when an application ignores SIGTERM and gets forcefully killed after the 10-second default timeout.

Step 1: Quick Fix (Diagnose OOMKilled State via docker inspect)

Inspect the terminated container’s state metadata to definitively confirm whether memory exhaustion triggered the termination.

# 1. Inspect the exit code and OOMKilled boolean flag:
docker inspect <container_name_or_id> --format='ExitCode: {{.State.ExitCode}} | OOMKilled: {{.State.OOMKilled}}'

# Expected output when killed by memory exhaustion:
# ExitCode: 137 | OOMKilled: true

# 2. Check kernel ring buffer logs on the host to confirm the cgroup kill event:
dmesg -T | grep -i -E "oom[-_]killer|killed process"

# 3. Inspect recent container logs prior to termination:
docker logs --tail 50 <container_name_or_id>

Step 2: Increase Memory Limits & Reservation in Docker Compose

Adjust memory ceilings and configure memory reservations to guarantee sufficient headroom for peak application spikes.

# Example configuration in docker-compose.yml:
services:
  web-app:
    image: my-node-app:latest
    # Compose v2 / v3 Resource Constraints:
    deploy:
      resources:
        limits:
          memory: 2G       # Hard upper ceiling before OOM trigger
        reservations:
          memory: 512M     # Guaranteed minimum RAM allocated on start

    # Legacy Compose v1 format:
    # mem_limit: 2048m
    # mem_reservation: 512m

# Apply the updated configuration in detached mode:
docker compose up -d --force-recreate

Step 3: Tune Application Runtime Heap Limits (Node.js, Java, Python)

Constrain application heap allocations to stay safely below the container’s hard memory boundary.

# --- Node.js / V8 Engine ---
# Set the max V8 heap size to ~75% of your container limit (e.g. 1.5GB for a 2GB container):
# In your Dockerfile or command:
CMD ["node", "--max-old-space-size=1536", "server.js"]

# --- Java / JVM (Java 11+) ---
# Ensure the JVM respects cgroup memory limits automatically:
CMD ["java", "-XX:+UseContainerSupport", "-XX:MaxRAMPercentage=75.0", "-jar", "app.jar"]

# --- Python / Celery / Gunicorn ---
# Prevent worker memory leaks by recycling workers after handling requests:
# In gunicorn command:
CMD ["gunicorn", "-w", "4", "--max-requests", "1000", "--max-requests-jitter", "100", "app:app"]

Verification & Testing Steps

Monitor real-time container memory consumption and verify that memory usage remains stable under traffic load.

# 1. Monitor live resource usage metrics across all active containers:
docker stats

# Observe the "MEM USAGE / LIMIT" and "MEM %" columns.
# Ensure memory utilization stays consistently below 80-85%.

# 2. Check container uptime and verify exit code 0 on graceful shutdown:
docker ps --filter "name=web-app"

# 3. Simulate traffic or workload to ensure no unexpected 137 terminations occur:
docker exec -it web-app free -m

Summary Comparison Table

Remediation Area	Applied Solution	Target Root Cause	Recommended Scope
Container Limits	limits.memory: 2G	Hard cgroup limit too restrictive	Docker Compose / K8s manifests
Memory Reservation	reservations.memory: 512M	Host RAM contention during startup	High-priority core services
Node.js Runtime	--max-old-space-size=1536	V8 heap exceeding container boundary	Node.js & frontend build tasks
Java JVM Runtime	-XX:MaxRAMPercentage=75.0	JVM ignoring cgroup constraints	Java Spring Boot & microservices