[SOLVED] Lots of ZFS snapshots appeared out of nowhere, what are they?

DPX-Infinity · 2016-08-11 17:34:22

I've successfully installed Arch with the root partition having ZFS on it. The partition table is GPT, and the partition itself is encrypted with LUKS, so after the boot it is decrypted first before ZFS mounts all datasets. ZFS is installed using zfs-dkms-git package from AUR, and I'm running the linux-ck kernel from the repo-ck. Everything works fine, however, in the zfs list output I started to see lots of small snapshots I didn't create:

NAME                                                                                                 USED  AVAIL  REFER  MOUNTPOINT
zroot/ROOT/default/0aacd26dc0815c8d8b1f905e0bb47e33879dd9f85639d1c3b49e2e4867ebb1ed@447533041           0      -  28,3M  -
zroot/ROOT/default/0cfec1c873cc8e24bb8bd84386df804c179bf18abac995b57dccaad8f3869344@663336658           0      -  73,9M  -
zroot/ROOT/default/0da39fca113865fdf89bb57fe767f07737e1d2b5eebcb5e3da0308597b67d06d@480262119           0      -  54,6M  -
zroot/ROOT/default/10aafc1c90fe9f11be79e0bc57958be987e728b8d648ebacaa5ba72d709ccc49@412138594           0      -  98,9M  -
zroot/ROOT/default/128b805e4f0fc33809fbf747677726bbeb0557727753e278704813db54286d19@765429610           0      -  72,7M  -
zroot/ROOT/default/1ab9c997ab2124f51d86e30473bc0f7a0c9adf2e978d7a36b3b3253900b14ee9@247451062           0      -  73,9M  -
zroot/ROOT/default/203ee45f1e49c026b0188279bbf2f84cf08f8510f43acdc1f7f8aa18fcc34728@337752853           0      -  54,6M  -
zroot/ROOT/default/31daa601d42596a76efb12be2d898337b0a58f5538736f8bebe034e977b34009@31327136            0      -   214M  -
zroot/ROOT/default/3570bbbc7f3cfffb7596abdd1da4dd09f9ecb6b8cb53d1432fc5899108f98767@599661713           0      -  54,6M  -
zroot/ROOT/default/4a92d02400d43df535a3464166f50f2261756df8c77accd46960b35e79358832@702041577           0      -  53,4M  -
zroot/ROOT/default/4f99d28b0825df10dec32b2ae63af0fbdf4c55280003870dc09167f20aaf5876@994340412           0      -  59,3M  -
zroot/ROOT/default/56271e08ee716441710b39ecc748e1f56757d31fb7cab734e48b6bfdc829d6ec@72188194            0      -  53,3M  -
zroot/ROOT/default/5a66721d489358a184f37bf8915db6c6dd9491fe01d3ede24dc3dd3d8ddf49b4@929945132           0      -   214M  -
zroot/ROOT/default/5cbd9312fc95663f14b410752675dacc108b0efe1560e53a0f33804e9c86ede8@580418089           0      -  98,9M  -
zroot/ROOT/default/5dc0a94b34a5021a05f06b60f44d26c949334a86e67eac46d5fe22c51459fd9c@554171249           0      -   100M  -
zroot/ROOT/default/629d78c21e3d1092fcdd4fc460754fa5c9a7ddc55470430c0a80ca9fc4e63a7b-init@293338908      0      -  57,2M  -
zroot/ROOT/default/6795aff444410402126a025167af37ef97d7681b81368ada47bca83fa214d5c2@957768778           0      -  72,7M  -
zroot/ROOT/default/6d2f0b1f18c56302b812e01919fbf8d45bc095d7ae5215e6a6d0a5865cae7373-init@116548383      0      -  3,09M  -
zroot/ROOT/default/7342af2aa581a249551362d402036702331a37f05c6e08b082fbd05a8a1898fe-init@161243531      0      -   214M  -
zroot/ROOT/default/7fde36135bce7e19540b69015be587db52798dea3d584d792db1f5548d193b29@147982840           0      -  73,9M  -
zroot/ROOT/default/82966b44f14ff405d22559c90cb3163561c036065fad470f813d8c1c5156277a@841217324           0      -   214M  -
zroot/ROOT/default/86001628b387a9ddbbc64ef4b604f02f5899346f925fe7475cbe8b9fb8ac25a3@505050772           0      -   214M  -
zroot/ROOT/default/86ec82febabb4f100f436bfd60b4b490bb272967798d5640e5ddfc27f4ee493e@809998495           0      -   100M  -
zroot/ROOT/default/8cfa4d50dc182eddf78bd79c3ff29ffc7a2053451449ef9f48f7d49b8f396cf2@293853948           0      -  56,0M  -
zroot/ROOT/default/8cfa4d50dc182eddf78bd79c3ff29ffc7a2053451449ef9f48f7d49b8f396cf2@576130457           0      -  56,0M  -
zroot/ROOT/default/8e56e876fb7f377d37fea16c248f1faea1c54fb704cacb7ec696d171c37a22f5@84325349            0      -  57,2M  -
zroot/ROOT/default/a2f9c41058aa692143e8997a38a76611949fb85d23d19969aaddbfe71c8a856e@258274297           0      -  98,9M  -
zroot/ROOT/default/ae77b34a7289b7d50922f90539602c74276ad5e95f65096b2d937fd2dd61651b@789612665           0      -   161M  -
zroot/ROOT/default/af0e66b4bbc08cc2754312a344ad7f6404071283c96f24b8ebb842afa24de7d0@79596698            0      -   161M  -
zroot/ROOT/default/b359bc5b18f41bbe15ef8627a3ac5c1cb7f3ddd3ef1ae642fc91bbb6c5e22022@357072439           0      -  3,09M  -
zroot/ROOT/default/b359bc5b18f41bbe15ef8627a3ac5c1cb7f3ddd3ef1ae642fc91bbb6c5e22022@176339030           0      -  3,09M  -
zroot/ROOT/default/b359bc5b18f41bbe15ef8627a3ac5c1cb7f3ddd3ef1ae642fc91bbb6c5e22022@37921931            0      -  3,09M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@284173196           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@673687997           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@889344191           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@950355313           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@848365034           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@892379162           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@570153387           0      -  59,3M  -
zroot/ROOT/default/b3a84d06aa3fed906356da7471cfae61b689906269195e87bed057af28ba635a@80977732            0      -  59,3M  -
zroot/ROOT/default/b474504b18832bcd4f901198f87430f3fe9927faa7b6511d50cec5d0c490ffdc@683770615           0      -  28,3M  -
zroot/ROOT/default/bbf1217e94c35958f1636155852c8fe3d6db27d6c162105ae48880ada10ab356@752085087           0      -  57,2M  -
zroot/ROOT/default/cf6cb5286e946c04a18d5eda8f90631aebcd153c69ca152cea162af920145bce@383432085           0      -  53,3M  -
zroot/ROOT/default/d38c244e1a77ac8bd2bcdf45393e7de29b0e93f961ae7e4a5281f371e99ffe34@301266832           0      -   214M  -
zroot/ROOT/default/d38c244e1a77ac8bd2bcdf45393e7de29b0e93f961ae7e4a5281f371e99ffe34@952214721           0      -   214M  -
zroot/ROOT/default/d38c244e1a77ac8bd2bcdf45393e7de29b0e93f961ae7e4a5281f371e99ffe34@551884202           0      -   214M  -
zroot/ROOT/default/db25853231f8606e67275e9fd99f2a1ce4b45e43a02ad91569e56e20a9ccd084@343566800           0      -  3,09M  -
zroot/ROOT/default/ddbf70e2c0358773714cf97c175cd45f65903d580988db836c5d62557e4b8f95@110516206           0      -   100M  -
zroot/ROOT/default/e32c9971bcd67b197e117bc25fd015ffe34799e10a08b71c232c6803017aa506@528758659           0      -  57,2M  -
zroot/ROOT/default/e60505378e26115a1532d1230650e6b0a00e3580b571ebf7db507813a4871933@752918094           0      -  25,9M  -
zroot/ROOT/default/e78514e7a61e2068ba75d69bc3605cf17dab27fdcd06f631cb716c9cf2f3c4c5@885628689           0      -   100M  -
zroot/ROOT/default/e83d65d66ceb2ee45300e60874c70a617189d43acfa3f4471c348355b0866359@556658568         11K      -  3,09M  -
zroot/ROOT/default/e95730ce5fa4ec4afdd85e17d6ae48ccee5be9505056315fc0097b73749d76f8@595100655           0      -   100M  -
zroot/ROOT/default/eac686ca82e1246b3248674685f2dd1b1767c3976074ce16807e2684ade4b507-init@61864658       0      -   214M  -
zroot/ROOT/default/f3e6a5fa8be2d2480f12c9bb90229e33bd2e2fdac287fad5a0c37e42735c50c2@452150896           0      -  98,9M  -
zroot/ROOT/default/ff49b3689a57085ef7e4c9c58a9fa2fb75d21cbfa16a1752820cc69fe2677272@861972694           0      -  72,7M  -
zroot/ROOT/default/ff7b78c97d0e39e958444551e6d40930424a272cb8043e34c5a6f14dc60a26f8@896079102           0      -   214M  -

Apparently these snapshots are only created under the root dataset, i.e. that which is mounted on /. The dataset which is mounted on /home does not have such snapshots. Their names look like hashes of something. I don't have any cron installed and only four systemd timers are enabled (logrotate, man-db, shadow and systemd-tmpfiles-clean), so it isn't some kind of recurring job. Moreover, the following command, which displays the time and the day these snapshots were created, shows this:

% zfs list -t snapshot | awk '{print $1}' | grep '^zroot' | while read n; do zfs get creation $n; done | grep -v PROPERTY | grep -o '[0-9]\?[0-9] [0-9][0-9]:[0-9][0-9]' | sort | uniq
10 12:34
10 12:35
10 14:36
10 19:38
10 19:48
10 19:51
10 20:01
10 20:03
8 16:55
8 21:39
8 21:40
8 22:31
8 22:32
8 22:33
8 22:34

Here 10 is August, 10th and 8 is August, 8th. I did not do anything special at these times, and they look very strange.

Does anyone know what these snapshots are and whether it is safe to delete them? And how to prevent their appearance in the future? I couldn't find even the slightest hint on what's going on in google, and I've also read the entire ZFS page on the wiki several times but there is nothing there too.

Last edited by DPX-Infinity (2016-08-13 10:30:36)

alphaniner · 2016-08-11 18:15:22

Have you checked the journal? Even if it's not immediately obvious as the cause, maybe there's a similar pattern of events that occured on each of 22:{31,32,33,34} on the 8th.

null · 2016-08-11 18:20:56

Why shouldn't they be save to delete? They are only snapshots after all Also they are all zero sized except one so it's probably save even in terms of data preservation.

But I also have no idea where they could be from. The program that creates them could really create snapshots with better names!

Do you use the zfs git packages? Or do you have any other zfs related packages installed?

Last edited by null (2016-08-11 18:21:21)

DPX-Infinity · 2016-08-13 10:20:38

alphaniner wrote:

Have you checked the journal? Even if it's not immediately obvious as the cause, maybe there's a similar pattern of events that occured on each of 22:{31,32,33,34} on the 8th.

No, I didn't, but I did it right now and apparently it happened every time I worked with Docker! I use docker-compose for development, and apparently the system log contains messages about creation and deletion of Docker networks and containers at these times. Actually, these hashes in snapshot names are awfully similar to hashes that Docker uses to identify things, although none of these hashes seem to match anything inside Docker now. I'll try to watch and check whether the Docker activity is related to the snapshots creation, although a quick googling didn't show anything about it.

null wrote:

[Do you use the zfs git packages? Or do you have any other zfs related packages installed?

I have the following packages related to ZFS installed:

% pacman -Q | grep zfs
zfs-dkms-git 0.6.5_r384_gcf2731e-1
zfs-utils-dkms-git 0.6.5_r384_gcf2731e-1

and nothing else.

Last edited by DPX-Infinity (2016-08-13 10:22:05)

DPX-Infinity · 2016-08-13 10:30:15

Oh well, apparently it is indeed Docker. Its documentation implies that the ZFS storage driver is enabled automatically if /var/lib/docker is on ZFS filesystem, and indeed, docker info displays that it has ZFS storage driver enabled:

% docker info
...
Storage Driver: zfs
 Zpool: zroot
 Zpool Health: ONLINE
 Parent Dataset: zroot/ROOT/default
 Space Used By Parent: 5522984448
 Space Available: 148640021504
 Parent Quota: no
 Compression: lz4
...

I think this answers my question.

Arch Linux

#1 2016-08-11 17:34:22

[SOLVED] Lots of ZFS snapshots appeared out of nowhere, what are they?

#2 2016-08-11 18:15:22

Re: [SOLVED] Lots of ZFS snapshots appeared out of nowhere, what are they?

#3 2016-08-11 18:20:56

Re: [SOLVED] Lots of ZFS snapshots appeared out of nowhere, what are they?

#4 2016-08-13 10:20:38

Re: [SOLVED] Lots of ZFS snapshots appeared out of nowhere, what are they?

#5 2016-08-13 10:30:15

Re: [SOLVED] Lots of ZFS snapshots appeared out of nowhere, what are they?

Board footer