Victor's room

Overview & Root Cause Summary: The error Error response from daemon: unable to remove filesystem for <container_id>: remove /var/lib/docker/...: device or resource busy occurs when executing docker rm or docker compose down to delete a stopped container, but the Linux kernel refuses to unlink the container’s overlay2 filesystem or shared memory (shm) mount. This happens when a host process, zombie sub-process, or adjacent container holds an active mount reference or open file descriptor inside the container’s runtime directory.

Understanding the Root Causes

• Leaked Mount Namespace References: Another process on the host (or a sidecar container sharing namespaces) inherited the container’s mount table and continues to hold an active reference to /var/lib/docker/overlay2/ or /var/lib/docker/containers/<id>/mounts/shm.
• Host Security & Antivirus Scanners: Host-level endpoint protection agents, file indexers, or monitoring tools actively scanning files inside /var/lib/docker while Docker attempts to tear down the container layer.
• Stale Shared Memory (shm) or Overlay2 Mounts: The container exited abruptly before the container runtime (containerd-shim) could cleanly unmount the internal tmpfs or merged overlay directory.
• Hung Network Storage (NFS / CIFS) Bind Mounts: Unresponsive remote network shares mounted into the container freezing the Linux Virtual File System (VFS) during teardown.

Step 1: Quick Fix (Lazy Unmount the Stuck Container Mounts)

Manually detach the locked shm or overlay2 directory from the host filesystem namespace using a lazy unmount, then remove the container.

# 1. Inspect active mounts matching the stuck container ID (replace <short_id> with first 12 chars):
mount | grep <short_container_id>

# Example output:
# shm on /var/lib/docker/containers/a1b2c3d4.../mounts/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)

# 2. Perform a lazy unmount (-l) on the stuck path:
sudo umount -l /var/lib/docker/containers/<full_container_id>/mounts/shm

# (If an overlay2 merged directory is also listed in mount output):
# sudo umount -l /var/lib/docker/overlay2/<layer_id>/merged

# 3. Now delete the container cleanly:
docker rm -f <short_container_id>

Step 2: Locate and Terminate Host Processes Holding Mount Locks

Scan the Linux /proc filesystem to pinpoint external processes that have leaked into the container’s mount namespace.

# 1. Search all process mount tables for references to the container ID:
sudo grep -l <short_container_id> /proc/*/mountinfo 2>/dev/null | cut -d/ -f3 | sort -u

# 2. Inspect the offending Process IDs (PIDs) returned:
ps -fp <PID>

# 3. Terminate the zombie or leaked host process holding the mount:
sudo kill -9 <PID>

# 4. Retry removing the container:
docker rm -f <short_container_id>

Step 3: Restart the Docker Daemon & Configure Directory Exclusions

Reset the Docker daemon’s internal mount tracker and prevent host scanners from locking overlay storage in the future.

# 1. Restart the Docker service to release stale containerd-shim locks:
sudo systemctl restart docker

# 2. Remove the previously stuck container:
docker rm -f <short_container_id>

# 3. Best Practice: Exclude /var/lib/docker from host file indexers (updatedb / locate):
# Edit /etc/updatedb.conf and ensure PRUNEPATHS includes /var/lib/docker:
# PRUNEPATHS="/tmp /var/spool /media /var/lib/docker"

Verification & Testing Steps

Confirm that the container and its underlying filesystem directories have been completely removed from the host.

# 1. Verify the container no longer appears in the Docker registry list:
docker ps -a | grep <short_container_id>
# Expected output: (empty)

# 2. Confirm no residual mount points exist in the kernel mount table:
grep <short_container_id> /proc/mounts
# Expected output: (empty)

# 3. Verify the container directory is purged from disk:
sudo ls -la /var/lib/docker/containers/ | grep <short_container_id>

Summary Comparison Table

Remediation Technique	Primary Root Cause	Service Impact	Recommended Scenario
umount -l <mount_path>	Stale shm or overlay2 mount	Zero downtime for other containers	First-line immediate CLI fix
/proc/*/mountinfo Audit	Leaked mount namespace in host PID	Terminates only the offending PID	Sidecar / monitoring agent leaks
systemctl restart docker	Hung containerd-shim state	Briefly restarts active containers	Stubborn kernel-level lockups
Scanner Path Exclusion	Antivirus / updatedb locking layers	None (Improves disk I/O)	Long-term production prevention