docker does not remove btrfs subvolumes when destroying container
phemmer opened this issue · comments
I receive the following error when deleting a container which created a btrfs subvolume (as happens when you run docker in docker).
# docker run --rm fedora:20 sh -c 'yum -y -q install btrfs-progs && btrfs subvolume create /test'
Public key for lzo-2.08-1.fc20.x86_64.rpm is not installed
Public key for e2fsprogs-libs-1.42.8-3.fc20.x86_64.rpm is not installed
Importing GPG key 0x246110C1:
Userid : "Fedora (20) <fedora@fedoraproject.org>"
Fingerprint: c7c9 a9c8 9153 f201 83ce 7cba 2eb1 61fa 2461 10c1
Package : fedora-release-20-3.noarch (@fedora-updates/$releasever)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-x86_64
Create subvolume '//test'
FATA[0033] Error response from daemon: Cannot destroy container c9badf5fc87bb9bfb50a3ee6e5e7c840476bd704e62404c9136aab4d27239d1e: Driver btrfs failed to remove root filesystem c9badf5fc87bb9bfb50a3ee6e5e7c840476bd704e62404c9136aab4d27239d1e: Failed to destroy btrfs snapshot: directory not empty
Info:
# docker info
docContainers: 22
Images: 47
Storage Driver: btrfs
Execution Driver: native-0.2
Kernel Version: 3.13.2-gentoo
Operating System: Gentoo/Linux
CPUs: 8
Total Memory: 15.64 GiB
Name: whistler
ID: RL3I:O6RS:UJRN:UU74:WAGE:4X5B:T2ZU:ZRSU:BN6Q:WN7L:QTPM:VCLN
Username: phemmer
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
# docker version
Client API version: 1.16
Go version (client): go1.3.3
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
@jfrazelle didn't you look into this? Did you get anywhere good? :)
still looking i really want to fix this, but it is a permissions thing, because we are making the same exact syscall as btrfs-tools
but this is a duplicate issue
duplicate of #7773, closing, lmk tho if you believe differently
actually I like your better, I'm going to close the other
This is something else, @jfrazelle. 😄
This is sub-subvolumes with BTRFS (ie, docker-in-docker artifacts, etc). This is why we had to add sort -r to our nuke script (so that it'd delete the inner subvolumes before the outer ones).
pretty sure we need this but in go lol https://lists.linuxcontainers.org/pipermail/lxc-devel/2014-July/009881.html
working on a port
but if someone else who is super good at C wants to do it go ahead :P
@tianon What are you using for a nuke script now?
Awesome, thank you!
Still hitting this. Are you planning to address this anytime soon?
I had problem with this today (BTRFS seemed out of space because of it) with version 1.9.1
Any progress?
@pkajaba I had to manually remove the nested subvolumes using $ btrfs directly.
@TomasTomecek me too
I'm having this problem too.
Does any one know of any workaround for it? the btrfs seems to be corrupted for me and somehow i need to make atleast the machine filesystem work.
To anyone else having to derp with this, I had to do similar to @TomasTomecek
┌[root@lovell]
└[/var/lib/docker]> btrfs subvolume delete btrfs/subvolumes/*
^that worked for me ;)
Anyone know a way to distinguish the orphan subvolumes? I don't want to blow away my running containers.
I encountered the issue in v1.13 in my CI environment. I eventually figured out that it occured when multiple Jenkins jobs were doing docker system prune simultaneously on a btrfs filesystem. Is there any doc or information regarding thread-safety of the docker daemon?
@thechane
I wrote a script that traverses the docker structures and finds what directories are orphaned and optionally deletes them. If it'd be useful to people I can put up on github and share here. I've only done basic testing on oraclelinux but it should apply to other OSes.
Fix embedded in "docker clean" would be nice
still hitting this bug. btrfs/subvolumes grows very fast.
@hipposareevil if you can post that script it would be super useful please :)
@johnharris85 Turns out that my script would negatively affect the system as it was deleting subvolumes that were tied to images. So if you tried to remove an image, it would be broken. It would work while you were running things, but then later when manipulating images you'd be hosed. :|
OK thanks @hipposareevil, anyway to filter out those volumes that were being used by images as well? Mind posting the script anyway (even as a gist) so I can hack on it?
@johnharris85 Shoot me an email at dockergist@mailinator.com
This is still an issue 😢
my subvolumes on btrfs grows to 222gb
I´ve deactivated btrfs for now and I´m using overlay2
with docker 1.13+ you have the command below that can help:
docker system prune -f
Since people here say this still happens, maybe it might be a good idea to reopen this ticket?
@huegelc Does the overlay2 driver work on a btrfs file system? The documentation says only ext4 and xfs are supported.
Using btrfs commands I could remove those sub-volumes e.g. :
btrfs subvolume delete eb669bae4f4aa17f3c432d956f481146e4ac77e3f1803fee15e1f2b17787510d-init
@devopxy yea that works, but it doesn't solve the underlying problem... what I usually do is uninstall docker, then wipe /var/lib/docker completely with all files and btrfs subvolumes inside, then reinstall.
Using btrfs commands I could remove those sub-volumes e.g. :
btrfs subvolume delete eb669bae4f4aa17f3c432d956f481146e4ac77e3f1803fee15e1f2b17787510d-init
Thx devopxy, that work for me to...
This command delete all sub-volumes present is the current directory :
btrfs subvolume delete *
Just ran into this today and had to clean it out. There doesn't appear to be any real solution...is there?
Anyways it seems that btrfs need some periodic cleaning operations.
On my side, I use this function in my .bashrc to do it:
unalias btrfsCleanup 2>/dev/null
btrfsCleanup() {
echo "btrfsCleanup"
sudo btrfs fi show
sudo btrfs fi df /
sudo btrfs fi usage /
sudo btrfs balance start -dusage=80 /
sudo btrfs scrub start -d /
sleep 120
sudo btrfs fi df /var
sudo btrfs fi usage /var
sudo btrfs balance start -dusage=80 /var
sudo btrfs scrub start -d /var
sleep 120
sudo btrfs fi df /var
sudo btrfs fi usage /var
echo "Done"
}
Last news on ticket #38207 trying to reopen this one, @thaJeztah needs someone that is capable of reproducing the error and provide all details asked by the issue template.
I think there are many more listeners on this thread so, if someone is still experiencing this issue, I think there might be a chance.
Experiencing the same issue.
# docker system prune -f
Total reclaimed space: 0B
# du -sh /var/lib/docker/btrfs/subvolumes
16G /var/lib/docker/btrfs/subvolumes
The following helps:
pushd /var/lib/docker/btrfs/subvolumes/
btrfs subvolume delete *
popd
The following helps:
pushd /var/lib/docker/btrfs/subvolumes/ btrfs subvolume delete * popd
This may break your docker build cache and other stuff
This may break your docker build cache and other stuff
What would you suggest?
ONLY WHEN ceph -s reports 'totally normal, everything you care about is running and all is perfect' -- do
docker system prune -a --volumes
If you do it under any other ceph operating condition -- not good.
@hcoin, This indeed helps:
$ docker system prune -a --volumes
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all volumes not used by at least one container
- all images without at least one container associated to them
- all build cache
Are you sure you want to continue? [y/N] y
Deleted Volumes:
...
Deleted Images:
...
Total reclaimed space: 5.033GB
docker system prune -a --volumes
does nothing for me and reports 0GB cleaned up. I have 22GB of subvolumes on a system with 16 running containers. It's a very low storage RPi system and I'm not sure what to do here without just nuking the /var/lib/docker folder and starting again.