[finished] Docker lvm maintenance: How to reapir thin pool

clupus · 2021-03-15 09:10:21

Hello,

I have two related problems and I do not really know how I can proceed here:

I have docker running here and I seem to have brought problems over me: I have a default docker install running (no special configuration) that uses the LVM loopback thin pool devices from the devicemapper storage service (the default of Archlinux at the time of installing).
Additionally, I use LVM for the partitioning of my hard drive.

I think my problem is that my system crashed recently and/or the memory on the partition /var/lib/docker has temporarily fallen to zero free bytes. I not sure which one triggered the problem ultimately.
Anyway, I have the problem that docker is no longer able to remove containers. When invoking docker rm <container> I get the message

container <hash>: driver "devicemapper" failed to remove root filesystem: failed to remove device <other_hash>: Error running DeleteDevice dm_task_run failed

Looking at the output of docker ps the container is either listed as shutting down or as Dead.

Another indication of the problem arose when I resized the partition where /var/lib/docker lives. I got a message about a thin pool I forgot to save for later reference. I can still reproduce a "similar" message when checking the list of LVs. Here is the beginning of such a command:

# lvs -a
  WARNING: /dev/mapper/docker-254:13-1572865-f133b1d955365586f6492bf8cb03c03fc1ffa0b133877e06cac902228ffb151f: Thin's thin-pool needs inspection.
  WARNING: /dev/mapper/docker-254:13-1572865-479ec6ae5283df942846520ace18f011b42f1f3e4b099b1c85993a8985ce7bcb: Thin's thin-pool needs inspection.
  WARNING: /dev/mapper/docker-254:13-1572865-e923a02bf74960b0ef249d4583ce2481130ef21e7fcc48d2691f61b505b8764e: Thin's thin-pool needs inspection.
  WARNING: /dev/mapper/docker-254:13-1572865-844a00a5c2133ba4e348fc02cfb9d56fce4ec249f709a1a90c0860a03599270e: Thin's thin-pool needs inspection.
  LV          VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  aurbuild    system -wi-ao----  20,00g                                                    
  data        system -wi-a----- 100,00g  
....

A bit digging into the docs and testing a few things out revealed the following:

My docker uses the lvm-loop storage driver. Verified by looing at docker info.
Docker uses thin pools for storing various files.
There are files in /var/lib/docker/devicemapper/devicemapper/ called data and metadata. These seem to be files containing the thin pool images.
When the thin pool is damaged and/or space runs out a repair might be required (reference: lvmthin).

While the docker daemon is running

In lvm -a does not list any thin pools. There are only my partitions as I created them during installation.
In losetup -a the two loop files are configured as /dev/loop?.

When looking at dmsetup table | grep docker | sort there are a lines indicating a thin pool is present:

docker: 0 125796352 crypt aes-cbc-plain64 :32:logon:cryptsetup:05f822d3-80b0-420a-96e7-802461371a75-d0 0 254:5 32768 1 allow_discards
docker-254:13-1572865-479ec6ae5283df942846520ace18f011b42f1f3e4b099b1c85993a8985ce7bcb: 0 20971520 thin 254:16 959
docker-254:13-1572865-844a00a5c2133ba4e348fc02cfb9d56fce4ec249f709a1a90c0860a03599270e: 0 20971520 thin 254:16 5360
docker-254:13-1572865-e923a02bf74960b0ef249d4583ce2481130ef21e7fcc48d2691f61b505b8764e: 0 20971520 thin 254:16 6980
docker-254:13-1572865-f133b1d955365586f6492bf8cb03c03fc1ffa0b133877e06cac902228ffb151f: 0 20971520 thin 254:16 5364
docker-254:13-1572865-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing

The devices 7:1 and 7.0 are /dev/loop{1,0}

I stopped the docker daemon to check if I can repair the thin pool. The result was as following:
1. The dmsetup table | grep docker does no longer output anything related to thin pools of docker
2. The loop devices are unbound. No loop devices are bound at all.
3. Manually binding loop devices to the loop files (like losetup /dev/loop0 /va/rlib/docker/devicemapper/devicemapper/data) works in general.
4. LVM does not detect the loop devices at all. So e.g. pvs does not list the loop devices (it should as far as I understand).
5. Explicitly showing the information using pvdisplay /dev/loop0 fails.
```
# pvdisplay /dev/loop0 /dev/loop1
  Failed to find physical volume "/dev/loop0".
  Failed to find physical volume "/dev/loop1".
```

The LVM tools seem to be unable to help me here. Before going into more invasive methods (I found this link) I wanted to ask what I should do best.

I have not restarted the machine and will try to delay this until the point where I know it is relatively safe. The last resort is to remove everything under /var/lib/docker and create all containers/images/volumes from scratch.

Do you have any good advice how to proceed here except for having a backup of all critical containers/data?

Last edited by clupus (2021-03-20 09:57:35)

frostschutz · 2021-03-15 10:58:37

I'm completely unfamiliar with docker.

Maybe this helps https://stackoverflow.com/questions/247 … pper-error

But do make backups before experimenting.

clupus · 2021-03-20 09:57:16

In the end, I finally saved my volumes and all and switched to a new storage driver in docker. That removed all my containers but my volumes were unaffected (nice as no backups would have been required).

I did not manage to repair my pool. Also, as the device-mapper storage driver (the one that created the thin pool) is deprecated I permanently switched to overlay2 storage driver. So, this will no longer be an issue for me.

Thanks for thinking with me.

Arch Linux

#1 2021-03-15 09:10:21

[finished] Docker lvm maintenance: How to reapir thin pool

#2 2021-03-15 10:58:37

Re: [finished] Docker lvm maintenance: How to reapir thin pool

#3 2021-03-20 09:57:16

Re: [finished] Docker lvm maintenance: How to reapir thin pool

Board footer