Arch Linux

Dornith

Member

Registered: 2013-02-19

Posts: 78

NILFS nilfs_cleanerd Hangs forever - error -28 cleaning segments

I installed NILFS quite a while ago for my backup partition, and it worked quite dandy for a while, but now I'm getting some weird issues.

Primarily, after my backup script mounted the device, but before the script could run, nilfs_cleanerd started (I could not find a parent pid, so I presume it's an orphan) and it's been running ever since.  I checked the logs and it's filled with this:

Jun 02 22:51:01 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:06 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:11 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:17 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:22 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:27 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:32 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:37 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:42 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:47 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:52 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:51:57 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:03 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:08 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:13 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:18 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:23 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:28 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:33 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:38 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:44 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:49 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:54 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:52:59 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:04 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:09 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:14 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:19 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:25 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:30 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:35 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:40 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:45 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:50 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:53:55 imperator kernel: NILFS (sdb1): error -28 cleaning segments
Jun 02 22:54:00 imperator kernel: NILFS (sdb1): error -28 cleaning segments

I was able to kill the backup process and unmount the drive, as I thought that might either stop nilfs_cleanerd or maybe fix the problem, but it seems that hasn't worked since I'm still getting the same message and can't remount the drive.  Does anyone know what might be causing this and how to fix it?

Dornith

Member

Registered: 2013-02-19

Posts: 78

Re: NILFS nilfs_cleanerd Hangs forever - error -28 cleaning segments

I've rebooted, and now that I'm able to mount it, I've gotten some more details.  It seems that despite only having 20 GiB of data, I have 140 GiB/140 Gib used up (not too unexpected with a logging O.S.):

$ du -h --max-depth=1 /mnt/backUp/ 2> /dev/null
14G     /mnt/backUp/daily.0
59M     /mnt/backUp/daily.1
71M     /mnt/backUp/daily.2
71M     /mnt/backUp/daily.3
71M     /mnt/backUp/daily.4
71M     /mnt/backUp/daily.5
1.7M    /mnt/backUp/daily.6
79M     /mnt/backUp/weekly.0
73M     /mnt/backUp/weekly.1
4.7G    /mnt/backUp/weekly.2
4.0K    /mnt/backUp/weekly.3
372M    /mnt/backUp/monthly.0
81M     /mnt/backUp/monthly.1
85M     /mnt/backUp/monthly.2
20G     /mnt/backUp/
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdb1       140G  140G     0 100% /mnt/backUp

After looking around a bit I realized that the error code is a no space error.  The problem is I can't make space without the garbage collector, and the garbage collector can't run because I don't have the space.

It seems I have some circular dependencies.