undeletable ghostly file

lloeki · 2008-10-23 22:05:16

okay, I'm at a loss here so I'm calling in for help.

I have a bunch of files coming from an rsync. they land on my (ext3) filesystem as an automated incremental rotating backup. now I want to delete some of the older backups but some files just won't go.

here's the fun:

$ ls -loai
ls: cannot access liste témoins.htm: No such file or directory
ls: cannot access acide acétique glacial.pdf: No such file or directory
ls: cannot access tamis moléculaire.pdf: No such file or directory
total 20
1974598 drwxr-xr-x 2 lloeki 16384 2008-10-15 11:58 .
1974596 drwxr-xr-x 3 lloeki  4096 2008-10-15 11:58 ..
1974960 -????????? ? ?          ?                ? acide acétique glacial.pdf
1975007 -????????? ? ?          ?                ? liste témoins.htm
1975022 -????????? ? ?          ?                ? tamis moléculaire.pdf

now you know, this is a nice tux ghost I have here.

of course rm -f/unlink/whatever fails, since the file does not "exists".

so I guess it's a broken inode in the directory listing. corrupt fs? e2fsck -f doesn't give a damn and reports pristine fs.

now I just can't believe there's no f*cking manual way to unconditionally remove an entry from a directory i.e without checking that the inode points to anything.

this corruption seems tied to that rsync scripty stuff I made up as I don't see it coming up anywhere else, but regularly coming up with only common point being rsync, so I bet it's safe for me to just abuse the fs and remove the entry form the directory table.

any ext2 guru care to give a hand?

EDIT: thinking of it, it might just be because I'm using --link-dest and --delete-during together. then there's some race condition or something that makes a directory entry end up without content.
does no fix my stuff but hey, that makes sense.

Last edited by lloeki (2008-10-23 22:21:02)

Xyne · 2008-10-23 22:24:01

Is the corruption limited to file names with French characters and spaces in them? Does it happen to all of them?

Sorry, I have no suggestions of how to actually purge them (other than possibly humorous ones which would involve French stereotypes).

bludra84 · 2008-10-23 23:34:21

This happened at my work the other day. Try deleting it under your root account.

Daenyth · 2008-10-24 02:56:46

Are you able to "stat" the file to find inode? Then you can use find or something to delete it that way.

lloeki · 2008-10-24 07:52:52

of course I tried root already

stat(2) doesn't work ("cannot stat"), but ls -i gives inodes (see above)
using find -inum $inode gives "no such file or directory".

it just seems like there's a (filename, inode) tuple in the directory listing, but no file at the matching inode. this could theoretically happen if one creates a (second) hardlink to a file and removes the first one (i.e. its original name) at the same time, thus the reference count drops to zero while the second hardlink is being created, thus data gets dropped yet second hardlink links to it. it could happen if either operation is not atomic. if this is the case, it's a serious bug.

Last edited by lloeki (2008-10-24 07:53:31)

briest · 2008-10-24 19:50:19

lloeki wrote:

now I just can't believe there's no f*cking manual way to unconditionally remove an entry from a directory i.e without checking that the inode points to anything.

Try debugfs.

schivmeister · 2008-10-24 20:55:26

This could be an encoding issue, yes. Happened to me just once..Luckily it was a new partition so I just reformatted it. Could you create a similar file?

lloeki · 2008-10-25 23:18:30

we're on the right track.

debugfs:
- is able to stat the matching inodes. whoohoo.
- displays unicode chars as two-byte raw hex codes. how do I give that as an argument to its rm (or unlink) function? that's all I miss to get rid of it.

as for encoding, I never had an issue with that, utf8 all the way seems no prob till now. also it doesn't really seem an encoding issue, as the rights in ls would certainly show up properly, and they don't, while the filename shows up correctly.

larmeh · 2008-10-26 06:07:49

I just wanted to say that we can rule out encoding problems here. I found this thread googling for the same problem over which I stumbled yesterday when I saw a bunch of files inside my home directory which were created by Azureus.

$ ls -la
total 28
drwxr-xr-x 2 blah blah 24576 2008-10-25 20:14 .
drwxr-xr-x 9 blah blah 4096 2008-10-25 20:13 ..
?--------- ? ? ? ? ? 156BEF2D68A2DAD4AD296452C6FCFF2FAA1D583E.dat
?--------- ? ? ? ? ? 16B59E3EA2763132F8D23B575D3EEB8C96CE2DF4.dat
?--------- ? ? ? ? ? 188A63CB3B973290CAE416A357B00446197A794F.dat
...

The volume is ext3 formatted and all "ghostly" files have a similar name, i.e. there are no umlauts or other special characters involved. Just wanted to say that so it might make tracking down the problem somewhat easier.

Last edited by larmeh (2008-10-26 06:12:32)

lloeki · 2008-10-26 11:11:09

ideed a step in the same direction.

btw I previously moved away the broken folder (thus creating a brand new bakcup tree), and some time later (backup script is invoked in a cron job) the problem came up again. also the problem never came up in the last rotation folder, but only in previous ones. now I removed --delete-during from the rsync line and did some backup/rotations running my script by hand, and the problem did not come up once in five rotations.

to easily find the broken files I use this command:

find somedir -print0 | xargs -0 stat > /dev/null

for each broken file, it turns up 'cannot stat $filename' on stderr

Last edited by lloeki (2008-10-26 12:43:00)

larmeh · 2008-10-26 13:14:52

A simple manual fsck did the job for me. In my case, those 'ghostly' files were old entries in the fs-tree which had no corresponding inodes. Hence an ls -i returned 0 for each file. However, after the fsck all inodes were pointing to a tree entry again. Hell, I was even able to read/write these files and delete them by hand using 'rm'.
@lloeki, please try the same on your system, I'm dying to know if it fixes your problem as well.

Have a good one,
Lars

lloeki · 2008-10-26 20:28:21

as I said in my original post fsck was among the first things I tried, and did nothing.

eventually I managed to remove the culprit entries from each directory with debugfs's 'rm' command. this failed (or rather was leaving me with a scratching head) earlier because I was remotely logging in as root, which was falling back to C locale... hence printing dual-hex code chars for non-ascii ones and couldn't give accented chars as arguments as filename to rm... an export LC_ALL="en_US.UTF-8" and some commands later and I was done with those pesky entries. then I 'rm -R' the dirs and voilà.

now I'm doing another fsck just to make sure the fs is clean (after all, we're using fairly dangerous stuff here).

bah. linux just never ends up empowering you, this is almost insane.

many thanks to everyone

Last edited by lloeki (2008-10-26 20:28:44)

Arch Linux

#1 2008-10-23 22:05:16

undeletable ghostly file

#2 2008-10-23 22:24:01

Re: undeletable ghostly file

#3 2008-10-23 23:34:21

Re: undeletable ghostly file

#4 2008-10-24 02:56:46

Re: undeletable ghostly file

#5 2008-10-24 07:52:52

Re: undeletable ghostly file

#6 2008-10-24 19:50:19

Re: undeletable ghostly file

#7 2008-10-24 20:55:26

Re: undeletable ghostly file

#8 2008-10-25 23:18:30

Re: undeletable ghostly file

#9 2008-10-26 06:07:49

Re: undeletable ghostly file

#10 2008-10-26 11:11:09

Re: undeletable ghostly file

#11 2008-10-26 13:14:52

Re: undeletable ghostly file

#12 2008-10-26 20:28:21

Re: undeletable ghostly file

Board footer