You are not logged in.

Pages: 1

#1 2021-04-01 09:12:48

risto3
Member
Registered: 2016-09-27
Posts: 36

problem deleting rw files on nfs home

we're experiencing apparently problems on all our client machines...

Environment: 

server (Linux smicro 5.10.25-1-lts #1 SMP Sat, 20 Mar 2021 19:37:44 +0000 x86_64 GNU/Linux)

exportfs -sv says, for example, my $HOME is thus:
    /export/home/richard  192.168.0.0/23(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,secure,root_squash,no_all_squash)

and that $HOME is a bind mount:
    dpool/export/home/richard on /export/home/richard type zfs (rw,relatime,xattr,posixacl)
    systemd-1 on /home/richard type autofs (rw,relatime,fd=64,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20653,user,x-systemd.automount)
    systemd-1 on /export/home/richard type autofs (rw,relatime,fd=64,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20653)
    dpool/export/home/richard on /home/richard type zfs (rw,nosuid,nodev,noexec,relatime,xattr,posixacl)
    dpool/export/home/richard on /export/home/richard type zfs (rw,relatime,xattr,posixacl)

and an example problematic file:
$ LANG=C getfacl .local/share/.org.chromium.Chromium.ZFOK1U 
  # file: .local/share/.org.chromium.Chromium.ZFOK1U
  # owner: richard
  # group: users
  user::rw-
  group::---
  other::---

$ LANG=C stat .local/share/.org.chromium.Chromium.ZFOK1U 
  File: .local/share/.org.chromium.Chromium.ZFOK1U
  Size: 4074      	Blocks: 17         IO Block: 4096   regular file
  Device: 4bh/75d	Inode: 697700      Links: 1
  Access: (0600/-rw-------)  Uid: ( 1001/ richard)   Gid: (  100/   users)
  Access: 2021-04-01 09:41:27.563315345 +0200
  Modify: 2021-04-01 09:41:27.563315345 +0200
  Change: 2021-04-01 09:41:27.563315345 +0200
  Birth: -

on my client machine (Linux odroid-001e06336dd6 5.11.4-1-ARCH #1 SMP Sun Mar 7 23:46:10 UTC 2021 aarch64 GNU/Linux):
mount says:
    smicro:/home/richard on /home/richard type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.52,local_lock=none,addr=192.168.0.1)

$ LANG=C getfacl .local/share/.org.chromium.Chromium.ZFOK1U 
  # file: .local/share/.org.chromium.Chromium.ZFOK1U
  # owner: richard
  # group: users
  user::rw-
  group::---
  other::---

$ LANG=C nfs4_getfacl .local/share/.org.chromium.Chromium.ZFOK1U 
  A::OWNER@:rwatTcCy
  A::GROUP@:tcy
  A::EVERYONE@:tcy

$ LANG=C stat .local/share/.org.chromium.Chromium.ZFOK1U 
  File: .local/share/.org.chromium.Chromium.ZFOK1U
  Size: 4074      	Blocks: 17         IO Block: 1048576 regular file
  Device: 41h/65d	Inode: 697700      Links: 1
  Access: (0600/-rw-------)  Uid: ( 1001/ richard)   Gid: (  100/   users)
  Access: 2021-04-01 09:41:27.563315345 +0200
  Modify: 2021-04-01 09:41:27.563315345 +0200
  Change: 2021-04-01 09:41:27.563315345 +0200
  Birth: -

mount stats for $HOME:
Stats for smicro:/home/richard mounted on /home/richard:

  NFS mount options: rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.52,local_lock=none
  NFS mount age: 1 day, 1:42:59
  NFS server capabilities: caps=0x3ffb7fff,wtmult=512,dtsize=1048576,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x68803,acl=0x3,sessions,pnfs=notconfigured,lease_time=90,lease_expired=0
  NFS security flavor: 1  pseudoflavor: 0

NFS byte counts:
  applications read 11280629278 bytes via read(2)
  applications wrote 1398628039 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 5129298126 bytes via NFS READ
  client wrote 1251828425 bytes via NFS WRITE
RPC statistics:
  514881 RPC requests sent, 514878 RPC replies received (3 XIDs not found)
  average backlog queue length: 0

READ:
	165498 ops (32%) 
	avg bytes sent per op: 224	avg bytes received per op: 31097
	backlog wait: 0.121609 	RTT: 8.965915 	total execute time: 9.144503 (milliseconds)
LOOKUP:
	94467 ops (18%) 	21894 errors (23%)
	avg bytes sent per op: 254	avg bytes received per op: 265
	backlog wait: 0.075910 	RTT: 3.379508 	total execute time: 3.563625 (milliseconds)
GETATTR:
	53964 ops (10%) 	1 errors (0%)
	avg bytes sent per op: 207	avg bytes received per op: 263
	backlog wait: 0.075624 	RTT: 2.774980 	total execute time: 2.997054 (milliseconds)
CLOSE:
	31219 ops (6%) 	5 errors (0%)
	avg bytes sent per op: 226	avg bytes received per op: 232
	backlog wait: 0.114450 	RTT: 2.873154 	total execute time: 3.054550 (milliseconds)
STATFS:
	30363 ops (5%) 
	avg bytes sent per op: 207	avg bytes received per op: 160
	backlog wait: 0.111386 	RTT: 2.910747 	total execute time: 3.162072 (milliseconds)
WRITE:
	28849 ops (5%) 
	avg bytes sent per op: 43637	avg bytes received per op: 288
	backlog wait: 39.886478 	RTT: 22.871815 	total execute time: 62.861347 (milliseconds)
SETATTR:
	22981 ops (4%) 	1 errors (0%)
	avg bytes sent per op: 248	avg bytes received per op: 287
	backlog wait: 0.105914 	RTT: 6.198468 	total execute time: 6.435186 (milliseconds)
OPEN_NOATTR:
	20651 ops (4%) 	30 errors (0%)
	avg bytes sent per op: 276	avg bytes received per op: 354
	backlog wait: 0.139073 	RTT: 2.981115 	total execute time: 3.183962 (milliseconds)
OPEN:
	13305 ops (2%) 	2376 errors (17%)
	avg bytes sent per op: 347	avg bytes received per op: 340
	backlog wait: 0.138820 	RTT: 7.573694 	total execute time: 7.780458 (milliseconds)
ACCESS:
	12305 ops (2%) 
	avg bytes sent per op: 215	avg bytes received per op: 275
	backlog wait: 0.065664 	RTT: 3.010890 	total execute time: 3.243722 (milliseconds)
DELEGRETURN:
	9659 ops (1%) 	2 errors (0%)
	avg bytes sent per op: 228	avg bytes received per op: 271
	backlog wait: 27.208821 	RTT: 3.867792 	total execute time: 31.196811 (milliseconds)
REMOVE:
	6472 ops (1%) 	1735 errors (26%)
	avg bytes sent per op: 223	avg bytes received per op: 110
	backlog wait: 0.072775 	RTT: 5.458900 	total execute time: 5.623609 (milliseconds)
RENAME:
	5204 ops (1%) 	109 errors (2%)
	avg bytes sent per op: 294	avg bytes received per op: 151
	backlog wait: 0.056879 	RTT: 6.654304 	total execute time: 6.761337 (milliseconds)
LOCK:
	3854 ops (0%) 
	avg bytes sent per op: 264	avg bytes received per op: 112
	backlog wait: 0.258433 	RTT: 3.135963 	total execute time: 3.439024 (milliseconds)
LOCKU:
	3742 ops (0%) 
	avg bytes sent per op: 236	avg bytes received per op: 112
	backlog wait: 0.084981 	RTT: 2.691342 	total execute time: 2.816141 (milliseconds)
FREE_STATEID:
	2224 ops (0%) 
	avg bytes sent per op: 172	avg bytes received per op: 88
	backlog wait: 0.089029 	RTT: 2.871403 	total execute time: 3.005845 (milliseconds)
READDIR:
	1790 ops (0%) 
	avg bytes sent per op: 231	avg bytes received per op: 26525
	backlog wait: 0.038547 	RTT: 54.616760 	total execute time: 54.745810 (milliseconds)
COMMIT:
	1047 ops (0%) 
	avg bytes sent per op: 208	avg bytes received per op: 104
	backlog wait: 2.789876 	RTT: 16.681948 	total execute time: 19.520535 (milliseconds)
CREATE:
	499 ops (0%) 	193 errors (38%)
	avg bytes sent per op: 260	avg bytes received per op: 252
	backlog wait: 0.040080 	RTT: 8.773547 	total execute time: 8.937876 (milliseconds)
ALLOCATE:
	153 ops (0%) 
	avg bytes sent per op: 248	avg bytes received per op: 272
	backlog wait: 0.019608 	RTT: 2.764706 	total execute time: 2.875817 (milliseconds)
OPEN_DOWNGRADE:
	43 ops (0%) 
	avg bytes sent per op: 224	avg bytes received per op: 112
	backlog wait: 0.046512 	RTT: 2.534884 	total execute time: 2.627907 (milliseconds)
LINK:
	20 ops (0%) 
	avg bytes sent per op: 276	avg bytes received per op: 316
	backlog wait: 0.000000 	RTT: 33.000000 	total execute time: 33.300000 (milliseconds)
SYMLINK:
	18 ops (0%) 	5 errors (27%)
	avg bytes sent per op: 280	avg bytes received per op: 280
	backlog wait: 0.000000 	RTT: 31.722222 	total execute time: 31.833333 (milliseconds)
LOCKT:
	13 ops (0%) 
	avg bytes sent per op: 248	avg bytes received per op: 96
	backlog wait: 0.000000 	RTT: 2.692308 	total execute time: 2.769231 (milliseconds)
READLINK:
	6 ops (0%) 
	avg bytes sent per op: 196	avg bytes received per op: 118
	backlog wait: 0.000000 	RTT: 2.000000 	total execute time: 2.166667 (milliseconds)
SEQUENCE:
	3 ops (0%) 
	avg bytes sent per op: 132	avg bytes received per op: 80
	backlog wait: 0.000000 	RTT: 2869.000000 	total execute time: 2869.000000 (milliseconds)
SERVER_CAPS:
	2 ops (0%) 
	avg bytes sent per op: 184	avg bytes received per op: 164
	backlog wait: 0.000000 	RTT: 2.500000 	total execute time: 2.500000 (milliseconds)
GETACL:
	2 ops (0%) 
	avg bytes sent per op: 204	avg bytes received per op: 188
	backlog wait: 0.000000 	RTT: 1.500000 	total execute time: 1.500000 (milliseconds)
EXCHANGE_ID:
	2 ops (0%) 
	avg bytes sent per op: 256	avg bytes received per op: 100
	backlog wait: 0.000000 	RTT: 1.500000 	total execute time: 1.500000 (milliseconds)
LISTXATTRS:
	2 ops (0%) 
	avg bytes sent per op: 208	avg bytes received per op: 112
	backlog wait: 0.000000 	RTT: 2.500000 	total execute time: 3.000000 (milliseconds)
NULL:
	1 ops (0%) 
	avg bytes sent per op: 44	avg bytes received per op: 24
	backlog wait: 3.000000 	RTT: 2.000000 	total execute time: 5.000000 (milliseconds)
FSINFO:
	1 ops (0%) 
	avg bytes sent per op: 184	avg bytes received per op: 168
	backlog wait: 0.000000 	RTT: 1.000000 	total execute time: 1.000000 (milliseconds)
PATHCONF:
	1 ops (0%) 
	avg bytes sent per op: 176	avg bytes received per op: 116
	backlog wait: 0.000000 	RTT: 1.000000 	total execute time: 1.000000 (milliseconds)
CREATE_SESSION:
	1 ops (0%) 
	avg bytes sent per op: 224	avg bytes received per op: 124
	backlog wait: 0.000000 	RTT: 3.000000 	total execute time: 4.000000 (milliseconds)
RECLAIM_COMPLETE:
	1 ops (0%) 
	avg bytes sent per op: 140	avg bytes received per op: 88
	backlog wait: 0.000000 	RTT: 9.000000 	total execute time: 9.000000 (milliseconds)


finally, if I try to delete the file on the client:
$ LANG=C rm .local/share/.org.chromium.Chromium.ZFOK1U 
rm: cannot remove '.local/share/.org.chromium.Chromium.ZFOK1U': Operation not permitted
$ LANG=C rm -f .local/share/.org.chromium.Chromium.ZFOK1U 
rm: cannot remove '.local/share/.org.chromium.Chromium.ZFOK1U': Operation not permitted

but on the server, no problem:
$ rm .local/share/.org.chromium.Chromium.ZFOK1U 
$ LANG=C stat .local/share/.org.chromium.Chromium.ZFOK1U 
stat: cannot statx '.local/share/.org.chromium.Chromium.ZFOK1U': No such file or directory

any hints on how to resolve this, there are gobs of files that are normally autodeleted which are failing for all our users.

cheers, and thanks in advance

Last edited by risto3 (2021-04-01 12:21:34)

Offline

#2 2021-04-01 12:16:55

2ManyDogs
Forum Fellow
Registered: 2012-01-15
Posts: 4,646

Re: problem deleting rw files on nfs home

Please edit your post and use [ code ] tags (not quote tags) when posting output. This makes the output easier to read and provides a scroll box for long content.

https://gitlab.archlinux.org/archlinux/ … s-and-code
https://bbs.archlinux.org/help.php#bbcode

Offline

#3 2021-04-01 13:53:43

seth
Member
Registered: 2012-09-03
Posts: 60,948

Re: problem deleting rw files on nfs home

$ LANG=C stat .local/share/.org.chromium.Chromium.ZFOK1U 
  File: .local/share/.org.chromium.Chromium.ZFOK1U
$ LANG=C rm .local/share/.org.chromium.Chromium.ZFOK1U 
rm: cannot remove '.local/share/.org.chromium.Chromium.ZFOK1U': Operation not permitted
% cd /tmp
% mkdir test
% touch test/file
% sudo chown root:root test
% stat test test/file
% rm test/file

Which permissions are relevant to delete the file? What are you actually writing? The file?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#4 2021-04-01 14:43:57

risto3
Member
Registered: 2016-09-27
Posts: 36

Re: problem deleting rw files on nfs home

Well, since these are user files (and root is blocked on the home directories, I believe by the 'root_squash' option)
So I'll modify that a bit.... but to show that it's strange that deleting files in .local/tmp works but not in .local/share

richard@odroid-001e06336dd6:/home/richard/.local$ LANG=C stat tmp share
  File: tmp
  Size: 15              Blocks: 153        IO Block: 1048576 directory
Device: 41h/65d Inode: 19440       Links: 3
Access: (0755/drwxr-xr-x)  Uid: ( 1001/ richard)   Gid: (   10/   wheel)
Access: 2021-04-01 16:31:10.161332825 +0200
Modify: 2021-04-01 16:31:18.250577725 +0200
Change: 2021-04-01 16:31:18.250577725 +0200
 Birth: -
  File: share
  Size: 67              Blocks: 1009       IO Block: 1048576 directory
Device: 41h/65d Inode: 61          Links: 42
Access: (0755/drwxr-xr-x)  Uid: ( 1001/ richard)   Gid: (   10/   wheel)
Access: 2021-04-01 16:30:34.794634172 +0200
Modify: 2021-04-01 16:30:32.884812444 +0200
Change: 2021-04-01 16:30:32.884812444 +0200
 Birth: -
richard@odroid-001e06336dd6:/home/richard/.local$ nfs4_getfacl tmp
A::OWNER@:rwaDxtTcCy
A::GROUP@:rxtcy
A::EVERYONE@:rxtcy
richard@odroid-001e06336dd6:/home/richard/.local$ nfs4_getfacl share
A::OWNER@:rwaDxtTcCy
A::GROUP@:rxtcy
A::EVERYONE@:rxtcy
richard@odroid-001e06336dd6:/home/richard/.local$ touch tmp/file share/file
richard@odroid-001e06336dd6:/home/richard/.local$ LANG=C stat tmp/file share/file 
  File: tmp/file
  Size: 0               Blocks: 1          IO Block: 1048576 regular empty file
Device: 41h/65d Inode: 701625      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/ richard)   Gid: (  100/   users)
Access: 2021-04-01 16:33:28.708399886 +0200
Modify: 2021-04-01 16:33:28.708399886 +0200
Change: 2021-04-01 16:33:28.708399886 +0200
 Birth: -
  File: share/file
  Size: 0               Blocks: 1          IO Block: 1048576 regular empty file
Device: 41h/65d Inode: 701628      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/ richard)   Gid: (  100/   users)
Access: 2021-04-01 16:33:28.738397084 +0200
Modify: 2021-04-01 16:33:28.738397084 +0200
Change: 2021-04-01 16:33:28.738397084 +0200
 Birth: -
richard@odroid-001e06336dd6:/home/richard/.local$ LANG=C rm -f tmp/file share/file
rm: cannot remove 'share/file': Operation not permitted

and then on the server

richard@odroid-001e06336dd6:/home/richard/.local$ ssh smicro
Last login: Thu Apr  1 16:30:19 2021 from 192.168.0.52
richard@smicro:/home/richard$ cd .local/
richard@smicro:/home/richard/.local$ LANG=C stat tmp/file share/file
stat: cannot statx 'tmp/file': No such file or directory
  File: share/file
  Size: 0               Blocks: 1          IO Block: 131072 regular empty file
Device: 4bh/75d Inode: 701628      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/ richard)   Gid: (  100/   users)
Access: 2021-04-01 16:33:28.738397084 +0200
Modify: 2021-04-01 16:33:28.738397084 +0200
Change: 2021-04-01 16:33:28.738397084 +0200
 Birth: -
richard@smicro:/home/richard/.local$ getfacl tmp share
# file: tmp
# owner: richard
# group: wheel
user::rwx
group::r-x
other::r-x

# file: share
# owner: richard
# group: wheel
user::rwx
group::r-x
other::r-x

richard@smicro:/home/richard/.local$ LANG=C rm share/file

Offline

#5 2021-04-01 14:54:11

seth
Member
Registered: 2012-09-03
Posts: 60,948

Re: problem deleting rw files on nfs home

Directory stat on the server?
Never underestimate zfs to do weird stuff, https://github.com/openzfs/zfs/issues/6889

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#6 2021-04-01 15:10:08

risto3
Member
Registered: 2016-09-27
Posts: 36

Re: problem deleting rw files on nfs home

Well, though I've come across [open]zfs issues before, I'll like not to jump just yet to conclusions...especially given the directory 'stats' are
identical, on the server and on the client, for the directories which work as well as for those that don't seem to. (nothing 'sticky', either:-)

I do find it odd that '.local/tmp/' files seem okay to delete but not '.local/share/' files...

Is there perhaps a watchdog process I'm not aware of on either the client or the server?

If this is not reproducible on a non-zfs system, then perhaps I'll at least cross post an openzfs issue.

Offline

#7 2021-04-01 15:12:06

progandy
Member
Registered: 2012-05-17
Posts: 5,279

Re: problem deleting rw files on nfs home

You need to have write and execute permissions on the directory itself to delete files contained in it, but that seems to be correct.

Last edited by progandy (2021-04-01 15:13:26)

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#8 2021-04-01 15:33:25

seth
Member
Registered: 2012-09-03
Posts: 60,948

Re: problem deleting rw files on nfs home

What if you (on the server) bind-mount /home/richard/.local away from zfs (in doubt into a tmpfs)?
Can you delete files there at will?

What if you then export that bind-mount (ie. /tmp/richard) and mount it via nfs?
Can you delete files there at will?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Online

#9 2021-04-01 16:14:55

risto3
Member
Registered: 2016-09-27
Posts: 36

Re: problem deleting rw files on nfs home

what I did try is chgrp users and chmod 700 on .local/share  (instead of wheel and 755)

then I could delete 'file' just fine... curiously, setting it back as it was does not reproduce the problem.
(though I can reproduce it one level up, that is directly in .local/ instead of .local/share.

Is there some deeper file analysis I could do on the zfs snapshot?
That is, something that could be in the header to determine if it is zfs at fault or not?

Offline

Pages: 1

Board footer

Powered by FluxBB