So you ran make build-all on webwords and now your root filesystem is 100% full. BTRFS doesn't clean up Docker volumes the way you'd expect. Here's how to unpotato your system.
The companion webwords git repo lives here.
the problem
Docker's system prune doesn't fully clean up BTRFS subvolumes. Even after removing containers and images, the filesystem usage stays high because the underlying BTRFS subvolumes persist.
identify docker volumes
List all Docker volumes:
docker volume ls
Check which containers might be using volumes:
docker ps -a --filter volume=volume_name
delete docker volumes
Stop and remove any containers using the volumes:
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)
Remove all unused volumes:
docker volume prune -f
manual btrfs cleanup
If filesystem usage is still high, manually clean up BTRFS subvolumes.
Check Docker's storage driver:
docker info --format '{{.Driver}}'
If it shows btrfs, locate the subvolumes:
sudo ls /var/lib/docker/btrfs/subvolumes/
List all BTRFS subvolumes:
sudo btrfs subvolume list /var/lib/docker
Delete persistent subvolumes:
sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/HASH_HERE
Wait for async deletion to complete:
sudo btrfs subvolume sync /var/lib/docker
the nuclear option (that actually worked)
After running unpotato, Docker's metadata became corrupted with references to missing BTRFS subvolumes. Simple restarts and partial cleanups failed with errors like:
failed to register layer: stat /var/lib/docker/btrfs/subvolumes/53a19da3801c895b697625094ebc8f95668a212a4cbc923787ac0c86452d9f60: no such file or directory
The root cause: Docker's image metadata and containerd's content store still referenced layers that no longer existed in BTRFS.
Partial cleanup attempts that failed:
# These didn't fix the corruption:
sudo systemctl restart docker
sudo rm -rf /var/lib/docker/image/btrfs/layerdb/*
sudo rm -rf /var/lib/containerd/io.containerd.content.v1.content/*
The only solution that worked:
Complete reset of both Docker and containerd data stores:
sudo systemctl stop docker
sudo systemctl stop containerd
sudo rm -rf /var/lib/docker/*
sudo rm -rf /var/lib/containerd/*
sudo systemctl start containerd
sudo systemctl start docker
This deletes all Docker data including images, containers, volumes, and networks. But it's the only way to recover from corrupted layer metadata after aggressive BTRFS cleanup.
Automation script:
#!/bin/bash
# fix-docker-btrfs.sh - Nuclear reset for corrupted Docker BTRFS
set -e
echo "Stopping Docker daemon..."
systemctl stop docker
echo "Stopping containerd..."
systemctl stop containerd
echo "Removing all Docker btrfs data..."
rm -rf /var/lib/docker/*
echo "Removing all containerd data..."
rm -rf /var/lib/containerd/*
echo "Starting containerd..."
systemctl start containerd
echo "Starting Docker daemon..."
systemctl start docker
echo "Done! Docker has been completely reset."
verify cleanup
Check filesystem usage:
df -h /
The usage should drop significantly after proper cleanup.
prevention
Consider using a separate filesystem for Docker or configuring Docker to use a different storage driver if you frequently build large numbers of containers on BTRFS.