[Partially Solved] Bind Mount of ZFS Dataset: Different Content

brando56894 · 2016-05-03 04:37:50

The mods may have an issue with the following file names I'm posting, but I feel it is essential to discussing my problem, as I would really like to understand what's going on here....

I have my pools mounted in /mnt and I have NFSv4 exports setup, as the wiki suggests I made bind mounts for my datasets, using the tweaked options for ZFS as suggested.

 [root@nas ~]# zfs list storage/downloads
NAME                USED  AVAIL  REFER  MOUNTPOINT
storage/downloads   462G  2.10T  6.19G  /mnt/storage/downloads

/etc/fstab

/mnt/storage/downloads                          /srv/nfs/downloads              none            bind,defaults,nofail,x-systemd.requires=zfs-mount.service       0 0

Here's where it get's really odd.....completely different file sets exist in /mnt/storage/downloads/torrents/seeding and /srv/nfs/downloads/torrents/seeding!

 [root@nas ~]# ls -lh /mnt/storage/downloads/torrents/seeding/
total 444
drwxrwxrwx 3 1000 987   3 Mar 18 18:02 Dirty.Jobs
drwxrwxrwx 9 1000 987   9 Mar 20 14:24 Family.Guy
drwxrwxrwx 4 1000 987   5 Mar 20 11:19 First48
drwxrwxrwx 5 1000 987   5 Mar 18 18:12 Law.and.Order.SVU
drwxrwxrwx 3 1000 987   3 Mar 18 18:34 MythBusters
drwxrwxrwx 3 1000 987   3 Feb 26 20:10 Planet.Earth
drwxrwxrwx 3 1000 987   3 Mar 18 19:32 Ren.and.Stimpy
drwxrwxrwx 6 1000 987   6 Mar 20 10:18 Sealab2021
drwxrwxrwx 5 1000 987   5 Mar 17 04:35 sickrage
drwxrwxrwx 5 1000 987   6 Mar 20 15:47 Tripping.The.Rift
drwxrwxrwx 2 1000 987   2 Mar 20 11:17 '"Ugly Americans"'
drwxrwxrwx 3 1000 987  17 Mar 20 11:17 'Ugly Americans'
drwxrwxrwx 3 1000 987   3 Mar 17 01:00 Unwrapped

 [root@nas ~]# ls -lh /srv/nfs/downloads/torrents/seeding/
total 2.5G
drwxr-xr-x 2 65534 65534    2 May  2 20:16 sickrage
-rw-rw-rw- 1 bran  users 932M May  2 20:07 The.Blacklist.S03E18.mkv
-rw-rw-rw- 1 bran  users 756M May  2 20:07 The.Blacklist.S03E19.mkv
-rw-rw-rw- 1 bran  users 776M May  2 22:30 Tripping.The.Rift.The.Movie.mkv

I created an empty file in /srv/nfs/downloads/torrents/seeding to see if it would show up in the other mountpoint and it didn't! Just to make sure I wasn't going crazy and forgot to create the bind mount, I unmounted /srv/nfs/downloads and the files inside downloads disappeared! I mounted downloads and the files reappeared! I restarted the nfs-server service to see if that would fix anything and it didn't. Here's proof that the bind mount works.

[root@nas ~]# mount|grep srv
storage/downloads on /srv/nfs/downloads type zfs (rw,noatime,xattr,noacl)

How can a bind mount cause two completely different sets of files in the same location?!

Last edited by brando56894 (2016-05-03 19:05:02)

ewaller · 2016-05-03 13:54:19

This is not really a direct violation violation of our policy, you are not asking how to do something that violates copyright; although that may be going on tangentially. Be advised, these are public forums and everyone can see them.

I must say, I don't think we've the same taste in entertainment

Not a zfs user, but my first inclination is to ask: What is the full output of mount?

brando56894 · 2016-05-03 14:22:22

Thanks for the clarification! I remember when I was posting something to this extent I was notified that it wasn't a good idea to do so, now that I think of it it was over at the FreeNAS forums haha

 [root@nas ~]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=15388348k,nr_inodes=3847087,mode=755)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
/dev/sdi2 on / type f2fs (rw,relatime,background_gc=on,user_xattr,acl,inline_data,extent_cache,active_logs=6)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sdi1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3078244k,mode=700)
safekeeping on /mnt/safekeeping type zfs (rw,noatime,xattr,posixacl)
safekeeping/backups on /mnt/safekeeping/backups type zfs (rw,noatime,xattr,posixacl)
safekeeping/misc on /mnt/safekeeping/misc type zfs (rw,noatime,xattr,posixacl)
storage on /mnt/storage type zfs (rw,noatime,xattr,noacl)
storage/downloads on /mnt/storage/downloads type zfs (rw,noatime,xattr,noacl)
storage/downloads/torrents on /mnt/storage/downloads/torrents type zfs (rw,noatime,xattr,noacl)
storage/multimedia on /mnt/storage/multimedia type zfs (rw,noatime,xattr,noacl)
storage/multimedia/movies on /mnt/storage/multimedia/movies type zfs (rw,noatime,xattr,noacl)
storage/multimedia/tv on /mnt/storage/multimedia/tv type zfs (rw,noatime,xattr,noacl)
storage/vm-storage on /mnt/storage/vm-storage type zfs (rw,noatime,xattr,noacl)
/dev/sdi3 on /var/lib/libvirt/images type xfs (rw,relatime,attr2,inode64,noquota)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
storage/multimedia/tv on /srv/nfs/tv type zfs (rw,noatime,xattr,noacl)
storage/multimedia/movies on /srv/nfs/movies type zfs (rw,noatime,xattr,noacl)
storage/vm-storage on /srv/nfs/vm type zfs (rw,noatime,xattr,noacl)
safekeeping/backups on /srv/nfs/backups type zfs (rw,noatime,xattr,posixacl)
safekeeping/misc on /srv/nfs/misc type zfs (rw,noatime,xattr,posixacl)
storage/downloads on /srv/nfs/downloads type zfs (rw,noatime,xattr,noacl)

ewaller · 2016-05-03 14:39:13

storage/downloads/torrents on /mnt/storage/downloads/torrents type zfs (rw,noatime,xattr,noacl)
storage/multimedia on /mnt/storage/multimedia type zfs (rw,noatime,xattr,noacl)

Those strike me as odd. The pair below those do too.

storage/downloads has an empty directory called multimedia, and you are mounting storage/multimedia on /mnt/storage/multimedia (AKA /storage/download/multimedia). I've not tried using mountpoints in NFS volumes. I could be way off base, but it seems dicey.

alphaniner · 2016-05-03 15:10:24

What about /mnt/storage/downloads/torrents/ and /srv/nfs/downloads/torrents/? If you create a file in one does it appear in the other?

You bound /mnt/storage/downloads, and if I understand correctly /mnt/storage/downloads/torrents is a separate zfs fs. I don't think bind mounts are "recursive" to other filesystems, so /srv/nfs/downloads/torrents/ would have initially been an empty directory.

brando56894 · 2016-05-03 15:57:59

ewaller wrote:

storage/downloads/torrents on /mnt/storage/downloads/torrents type zfs (rw,noatime,xattr,noacl)
storage/multimedia on /mnt/storage/multimedia type zfs (rw,noatime,xattr,noacl)
Those strike me as odd. The pair below those do too.
storage/downloads has an empty directory called multimedia, and you are mounting storage/multimedia on /mnt/storage/multimedia (AKA /storage/download/multimedia). I've not tried using mountpoints in NFS volumes. I could be way off base, but it seems dicey.

Nothing about this is odd (to me at least), it may look odd to you since you don't use ZFS (I'll admit, it took me a good few months to wrap my head around ZFS when coming from mdadm and RAID5 lol). I don't understand what you mean, there is no multimedia directory in my storage/downloads dataset, multimedia is a dataset within my storage pool, did you confuse directories when typing your response?

 [root@nas ~]# zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
storage                     8.49T  2.05T    96K  /mnt/storage
storage/downloads            498G  2.05T  42.4G  /mnt/storage/downloads
storage/downloads/torrents   431G  2.05T   431G  /mnt/storage/downloads/torrents
storage/downloads/usenet    24.8G  2.05T  24.8G  /mnt/storage/downloads/usenet
storage/multimedia          7.99T  2.05T   100K  /mnt/storage/multimedia
storage/multimedia/movies    760G  2.05T   760G  /mnt/storage/multimedia/movies
storage/multimedia/tv       7.25T  2.05T  7.25T  /mnt/storage/multimedia/tv

 [root@nas ~]# ls /mnt/storage/downloads/
torrents  usenet

alphaniner wrote:

What about /mnt/storage/downloads/torrents/ and /srv/nfs/downloads/torrents/? If you create a file in one does it appear in the other?
You bound /mnt/storage/downloads, and if I understand correctly /mnt/storage/downloads/torrents is a separate zfs fs. I don't think bind mounts are "recursive" to other filesystems, so /srv/nfs/downloads/torrents/ would have initially been an empty directory.

Nope! Doesn't show up either!

 [root@nas ~]# cd /srv/nfs/downloads/torrents/

 [root@nas torrents]# ls
incomplete  seeding

 [root@nas torrents]# touch testfile

 [root@nas torrents]# ls /srv/nfs/downloads/torrents/
incomplete  seeding  testfile

 [root@nas torrents]# ls /mnt/storage/downloads/torrents/
incomplete  seeding

Just to test your theory, I created another bind mount (couldn't modify the ones above since they're in use) and the testfile exists that I made in the previous bindmount, but still doesn't exist at the original mount point!

  [root@nas torrents]# cd /

 [root@nas /]# mkdir test

 [root@nas /]# mount --bind /mnt/storage/downloads/ /test

 [root@nas /]# ls /test/torrents/
incomplete  seeding  testfile

[root@nas /]# ls /mnt/storage/downloads/torrents/
incomplete  seeding

After closer inspection to this, I realized that I mounted the dataset downloads and not downloads/torrents, which are in fact two separate filesystems, even though ZFS mounts them recursively. So it looks like you have the solution to this brain teaser, when I did the bindmount it didn't recursively mount the torrents dataset, but the programs created those directories, making it look like they were in fact mounted, which led to massive confusion.

brando56894 · 2016-05-03 17:11:25

I can't seem to delete the directory for some reason. I killed my KVM which was using the directories and I killed the NFS daemon, so nothing should be using it.

 [root@nas /]# rm -rf /srv/nfs/downloads/torrents
rm: cannot remove '/srv/nfs/downloads/torrents': Device or resource busy
 [root@nas /]# lsof /srv/nfs/downloads/torrents
 [root@nas /]# fuser -m /srv/nfs/downloads/torrents

The directory itself is empty, I had no problems deleting the usenet folder.

Last edited by brando56894 (2016-05-03 17:15:47)

alphaniner · 2016-05-03 17:37:56

I'm still thinking the torrents directory was carried over as an empty directory when you bound storage/downloads, in which case it would be in use by the storage/downloads/torrents zfs fs. Try unmounting the bindmount.

brando56894 · 2016-05-03 19:03:27

I'm about 99.9999% sure it was created by either transmission or manually by myself inside the VM. No matter what I do I can't delete the directory, I've tried unmounting the bind mount, remounting it and then deleting the directory and that didn't work, nor did rebooting the server, binding /mnt/storage/downloads to /test and then trying to remove the directory, but that didn't work either. The usenet directory also re-appeared!

It doesn't really matter because in a few days I'll be moving the downloads dataset to it's own pool once my two 2.5" WD Raptors come in (they're 150 GB and since they were refurbished they were a whopping $15 a piece on NewEgg!!) in a few days. Still not sure if I want to use ZFS on it or MDADM since it's just going to be temporary storage space and it needs to be quick (I'm striping them together, may get a third drive if it's worth it).

fuzzy2 · 2016-05-04 18:14:27

The problem here is that when you bind-mount ZFS, you don’t. As you can see, it simply mounts the filesystem once again. This time, however, and that’s the issue here, child filesystems are not mounted. This issue can be reproduced easily:

[root@server bla]# mount | grep hd2
hd2 on /hd2 type zfs (rw,relatime,xattr,posixacl)
hd2/Banana on /hd2/Banana type zfs (rw,relatime,xattr,posixacl)
hd2 on /mnt/bla type zfs (rw,relatime,xattr,posixacl)

[root@server bla]# ls -al /mnt/bla
total 5
drwxr-xr-x 8 root root 8 May  4 20:00 .
drwxr-xr-x 5 root root 5 Jan  4 22:28 ..
drwxr-x--- 2 root root 3 May  4 20:03 Banana

[root@server bla]# ls -l /hd2/Banana
total 1
-rw-r----- 1 root root 0 May  4 20:06 'Can'\''t see me'

[root@server bla]# ls -l /mnt/bla/Banana
total 1
-rw-r----- 1 root root 0 May  4 20:03 'Hallo!'

[root@server bla]# zfs umount hd2/Banana

[root@server bla]# ls -l /hd2/Banana
total 1
-rw-r----- 1 root root 0 May  4 20:03 'Hallo!'

However, after that, hd2/Banana will refuse to mount: cannot mount '/hd2/Banana': directory is not empty

So basically this is all about child filesystems not being mounted. The folder where a child filesystem is to be mounted (/hd2/Banana in my example) is created automatically, that’s why it existed in the first place.

Arch Linux

#1 2016-05-03 04:37:50

[Partially Solved] Bind Mount of ZFS Dataset: Different Content

#2 2016-05-03 13:54:19

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#3 2016-05-03 14:22:22

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#4 2016-05-03 14:39:13

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#5 2016-05-03 15:10:24

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#6 2016-05-03 15:57:59

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#7 2016-05-03 17:11:25

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#8 2016-05-03 17:37:56

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#9 2016-05-03 19:03:27

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

#10 2016-05-04 18:14:27

Re: [Partially Solved] Bind Mount of ZFS Dataset: Different Content

Board footer