df says disk is full, but du does not

leonixyz · 2018-02-17 18:11:33

I have an external usb disk attached, the disk has an udf filesystem and is used as storage for a torrent client.

Suddenly, the torrent client stopped working, and I noticed that the external disk (/dev/sdb1) appears to be full

$ df -h
Filesystem      Size  Used Avail Use% Mounted on

[...] # my mounts

/dev/sdb1       2.8T  2.8T     0 100% /media/fatboy

It's weird, because I didn't used up to 2.8T...

$ cd /media/fatboy
$ du -h

[...] # lots of directories

165G	.

But event weirder is that the filesystem seems to have no inodes at all (this was my first try with udf)

$ df -i
Filesystem     Inodes IUsed  IFree IUse% Mounted on

[...] # my mounts

/dev/sdb1           0     0      0     - /media/fatboy

fsck doesn't help

$ sudo umount /dev/sdb1 && sudo fsck /dev/sdb1

I have no idea what to do to free up space.

Trilby · 2018-02-17 18:24:08

The lack of inodes is almost certainly the problem. I suspect that's how `df` is gettings it's numbers (0 available -> it must be full) and it'd definitely prevent any writing to the device.

As to why there are no inodes at all, I'm clueless. You've been using this device under arch before?

frostschutz · 2018-02-17 19:02:17

UDF is not fit for use in Linux as anything other than read-only.

I just tried it (again after years) and... it's just as horrible as I remember, within 5 minutes it dumped no less than 500'000 lines of kernel panics in my logs.

[35089.286995] ------------[ cut here ]------------
[35089.286999] kernel BUG at fs/udf/truncate.c:276!
[35089.287007] invalid opcode: 0000 [#1] SMP
[35089.287017] CPU: 0 PID: 15005 Comm: rm Tainted: G        W        4.15.2 #3
[35089.287020] Hardware name: Gigabyte Technology Co., Ltd. B85M-D2V/B85M-D2V, BIOS F2 08/06/2015
[35089.287033] RIP: 0010:udf_truncate_extents+0x2b9/0x2c0
[35089.287036] RSP: 0018:ffffc90001fdfda0 EFLAGS: 00010246
[35089.287041] RAX: 0000000000000000 RBX: ffff8803db085fe0 RCX: 0000000000000000
[35089.287044] RDX: 00000000ffffffff RSI: 00000000000000d8 RDI: 00000000000000e8
[35089.287048] RBP: ffff88027d608800 R08: 00000000000000d8 R09: ffffc90001fdfdc0
[35089.287051] R10: 0000000000000000 R11: 00000000000001e9 R12: 0000000000000001
[35089.287054] R13: 0000000000000010 R14: 0000000000000000 R15: ffff8803db085fe0
[35089.287058] FS:  00007f07c1e5b500(0000) GS:ffff88041f200000(0000) knlGS:0000000000000000
[35089.287062] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[35089.287065] CR2: 0000559397c65328 CR3: 00000002e08d4003 CR4: 00000000001606f0
[35089.287067] Call Trace:
[35089.287076]  udf_setsize+0x236/0x380
[35089.287085]  ? fsnotify_grab_connector+0x37/0x60
[35089.287091]  ? __inode_wait_for_writeback+0x6a/0xc0
[35089.287096]  udf_evict_inode+0x75/0xd0
[35089.287103]  evict+0xb7/0x180
[35089.287110]  do_unlinkat+0x17f/0x2e0
[35089.287116]  do_syscall_64+0x5e/0x110
[35089.287127]  entry_SYSCALL_64_after_hwframe+0x21/0x86
[35089.287131] RIP: 0033:0x7f07c19aafd7
[35089.287134] RSP: 002b:00007ffcef1c6398 EFLAGS: 00000202 ORIG_RAX: 0000000000000107
[35089.287139] RAX: ffffffffffffffda RBX: 0000559397c65450 RCX: 00007f07c19aafd7
[35089.287142] RDX: 0000000000000000 RSI: 0000559397c64220 RDI: 00000000ffffff9c
[35089.287144] RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000
[35089.287147] R10: 0000000000000164 R11: 0000000000000202 R12: 0000559397c64190
[35089.287150] R13: 00007ffcef1c64d0 R14: 0000559397c65450 R15: 0000000000000000
[35089.287153] Code: ba 28 00 00 00 f6 c1 10 75 15 8b 53 88 8d b2 d8 00 00 00 81 c2 b0 00 00 00 83 e1 08 0f 45 d6 29 d0 41 89 c6 e9 1e fe ff ff 0f 0b <0f> 0b 0f 0b 90 90 90 55 53 48 89 f3 48 83 ec 08 48 8b 47 08 48 
[35089.287220] RIP: udf_truncate_extents+0x2b9/0x2c0 RSP: ffffc90001fdfda0
[35089.287225] ---[ end trace dde705525da0a96a ]---

Get rid of it.

What I tested was what a torrent client might do... create a file with fixed size (truncate -s 1G somefile) then write into it (shred somefile). I did this on a ramdisk loop device and the write speed was an amazing 8MB per second. (printing kernel panics is slow business).

Filesystem was "full" after writing 144Megs. Tried to delete the file, rm segfaulted. Trying to umount, it's busy even though it's not. I have to reboot the machine now or run out of kernel panic logspace. (Edit: rebooted now. remind me to boot up a VM next time)

As to why there are no inodes at all, I'm clueless.

Not every filesystem uses inodes (or has limits in that regard) but it's probably a matter of corrupt filesystem in this case.

Last edited by frostschutz (2018-02-17 19:10:57)

Trilby · 2018-02-17 19:38:24

frostschutz wrote:

Not every filesystem uses inodes...

Hmm.

Maniaxx · 2018-02-17 19:55:21

UDF is probably not intended for massive random writes/fragments. The price is massive overhead data. Rather download the files completely to hdd first and then write the file in one go to the disc.

Arch Linux

#1 2018-02-17 18:11:33

df says disk is full, but du does not

#2 2018-02-17 18:24:08

Re: df says disk is full, but du does not

#3 2018-02-17 19:02:17

Re: df says disk is full, but du does not

#4 2018-02-17 19:38:24

Re: df says disk is full, but du does not

#5 2018-02-17 19:55:21

Re: df says disk is full, but du does not

Board footer