Corrupted/orphaned files on ntfs

l0tek · 2022-05-20 10:09:02

Once a week I use rsync to backup an external (usb) hdd to a directory on another hdd (both ntfs, for use on windows 8.1.)

However I often notice, if I run chkdsk from windows on the destination hdd (right after running rsync) it fixes several "orphaned files", as if the filesystem becomes corrupted for some reason. I was wondering if others have ever noticed something like that?

smartmontools doesn't shows any bad values for that hdd. Also I have the ntfs_defaults specified at https://wiki.archlinux.org/title/NTFS#udisks_support if that matters.

Any ideas?

Last edited by l0tek (2022-05-20 10:09:32)

seth · 2022-05-20 12:28:18

Unclean removal (you forget to umount) or writing to an open filesystem (see 3rd link below for a common cause)

l0tek · 2022-05-20 13:15:33

Actually I'm very paranoid about losing my data and always use the sync command before umount (which I always do). The destination filesystem is only used for the rsync operation and then unmounted. No other operation done in either Linux or Windows as the disk is only used for backups - until I run chkdsk on it.

seth · 2022-05-20 13:22:14

So is this a dual boot system?
"hdd" is "hdd", not "ssd that I called hdd for tradition"?

Maniaxx · 2022-05-20 14:09:03

When i transitioned from ntfs-3g to ntfs3 i had one incident after some time. Around 50% of the files/folders were missing. They all reappeared after chkdsk/win10. I've also seen 0 bytes files (that had been deleted before).

l0tek · 2022-05-20 15:18:54

> So is this a dual boot system?
> "hdd" is "hdd", not "ssd that I called hdd for tradition"?

It's a laptop. Source and destination are 2 different usb rotational hdds, which I manually connect to the laptop. I run rsync, then sync, umount, and disconnect.
I know I could do the same using the robocopy command on windows, but I don't really like it.

In the rare occasion when I need a previous version of a file (on windows), I connect the backup hdd and get the file I need. That's how I noticed files I was sure I had copied were missing, and used chkdsk. It happened several times, but only after rsync. Copying a single file from source hdd to backup hdd (as a test) never resulted in a corrupted filesystem.

l0tek · 2022-05-20 15:19:53

n/t

Last edited by l0tek (2022-09-25 14:07:10)

halogen · 2023-01-06 10:54:06

l0tek wrote:

if I run chkdsk from windows on the destination hdd (right after running rsync) it fixes several "orphaned files"

I have a similar issue.

I'm using an usb 3 external disk (ntfs3) to share files I work on at home with Windows machines at work. There are lots of git subdirectories on this disk, and once in a while I run a bash script that does a git pull (and sometimes clones) recursively.

When I do this, and then use this disk again on any windows computer, chkdsk reports 100% of the time a few invalid filenames/orphaned files, fixes to unindexed files and to the master file table bitmap. No unclean removal, no dmesg errors, SMART values are ok, already tried replacing the usb cable.

I'm about to replace the bash script with a windows batch file but really don't understand why updating lots of files from linux is causing this?

Last edited by halogen (2023-01-06 10:55:20)

V1del · 2023-01-06 11:03:19

ntfs3 is quite new and has somewhat of a tendency to be rough around the edges. If you need a more well tested ntfs implementation and can live with the fuse perf hit use ntfs-3g

Last edited by V1del (2023-01-06 11:04:14)

halogen · 2023-01-06 11:23:29

Ok thanks, I'll try ntfs-3g. Better than switching to (ugh) batch scripts.

Edit: now I'm confused, dmesg says "ntfs3", but ls -altr /sbin/mount.ntfs returns /usr/bin/ntfs-3g (which is already installed), how do I switch from ntfs3 to ntfs-3g without having to manually mount disks as such?

Last edited by halogen (2023-01-06 11:31:28)

V1del · 2023-01-06 12:06:10

How are you mounting it currently? If via fstab you just adjust the FS column to read ntfs-3g explicitly, if via udisks it should use ntfs-3g via that default fallback. What's the output of

findmnt /$mountpointofntfsdisk

Last edited by V1del (2023-01-06 12:06:36)

halogen · 2023-01-06 12:19:40

It's automounted via gnome nautilus, findmnt says "fuseblk" so it's already ntfs-3g?

V1del · 2023-01-06 12:27:12

it is. Are you noticing any actual problems despite the "theorethical" brokenness identified by chkdsk? From my experience while there can be a variety of reasons why chkdsk mucks up, I've never had any real corruption with ntfs-3g while I've had quite a few instances of actual brokenness with ntfs3 (though they did get less in recent kernels, and there have been a bunch of fixes in the just released 6.1.3)

halogen · 2023-01-06 12:45:49

I can't really say if there are actual problems as I mostly work on a few of those files. I guess orphaned files are missing in the directories. And then there's chkdsk itself which keeps on deleting the "damaged" files, so later a git pull fails (I detect this in the bash script and re-clone)

Last edited by halogen (2023-01-06 12:56:33)

halogen · 2023-01-24 12:37:35

BTW interestingly chkdsk always marks as invalid a file called "fontawesome-webfont.woff2?v=4.6.3" (which I then have to redownload via git), and always says it has to correct indexes for ".HFS+ Private Directory Data" (... why hfs if this is ntfs?)

seth · 2023-01-24 15:24:58

Why do you have to re-download it, why do you download it itfp?
https://en.wikipedia.org/wiki/Web_Open_Font_Format

correct indexes for ".HFS+ Private Directory Data" (... why hfs if this is ntfs?)

Have you used the partition from MacOS?

halogen · 2023-01-27 06:37:32

seth wrote:

Why do you have to re-download it, why do you download it itfp?

I just run a script that takes care of that, as I have to re-clone several directories anyway because of the FS damages...

Have you used the partition from MacOS?

Never, I just mount the ntfs partition on linux as described above. Then when I run chkdsk on windows it reports those weird errors about HFS+

d.ALT · 2023-01-27 07:49:52

halogen wrote:

I'm using an usb 3 external disk

halogen wrote:

Then when I run chkdsk on windows it reports those weird errors about HFS+

What kind of external USB HDD are you using? Is this a WesternDigital by any chance?

halogen · 2023-01-27 16:22:43

d.ALT wrote:

Is this a WesternDigital by any chance?

Toshiba, 2 TB

seth · 2023-01-27 16:43:35

Never, I just mount the ntfs partition on linux as described above. Then when I run chkdsk on windows

iOS (iPad, iPhone, other iDings)?

I have to re-clone several directories anyway

One of those directories being a webcache?
How did "fontawesome-webfont.woff2?v=4.6.3" end up in your git repo? Did you try whether maybe the copy of that directory is already corrupted?
Do you maybe also clone the ".HFS+ Private Directory Data" from there?

halogen · 2023-01-27 18:28:20

seth wrote:

iOS (iPad, iPhone, other iDings)?

Just linux and windows 10.

One of those directories being a webcache? How did "fontawesome-webfont.woff2?v=4.6.3" end up in your git repo? Did you try whether maybe the copy of that directory is already corrupted?

I searched for *webcache* and *hfs* recursively but there's no directory on that disk containing those strings.
Anyway since chkdsk doesn't report the name of the problematic hfs (?) directory I don't know where to look, too many directories...

fontawesome comes from this cloned repo: https://github.com/FrenchYeti/dexcalibur (I didn't do any local changes to it)

Last edited by halogen (2023-01-27 18:32:20)

seth · 2023-01-27 20:56:00

fontawesome comes from this cloned repo

Path?

Anyway since chkdsk doesn't report the name of the problematic hfs (?) directory I don't know where to look, too many directories...

".HFS+ Private Directory Data" is a directory name.
It is/was btw. apparently part of the hfsplus kernel module, https://apple.stackexchange.com/questio … ctory-data

halogen · 2023-01-28 09:23:02

seth wrote:

Path?

./src/webserver/public/vendor/font-awesome/fontawesome-webfont.woff2?v=4.6.3

".HFS+ Private Directory Data" is a directory name

Ok found it. Since chkdsk deleted it, I've rerun the update script, then found the directory at:

./tests/integration/archive/dmg/__output__/fruits_osx.dmg_extract/fruits/.HFS+ Private Directory Data'$'\r

from https://github.com/onekey-sec/unblob

So it's not a FS issue, just chkdsk not handling it correctly?

halogen · 2023-01-28 09:36:37

After running the script I deleted the problematic git directories, unmounted, mounted in windows and now chkdsk is reporting several "free space marked as allocated in the bitmap". I really don't understand if this is a real issue or if chkdsk is just dumb...

Last edited by halogen (2023-01-28 09:37:55)

seth · 2023-01-28 09:42:47

D'ohh…

"?" is not a legal filename character in the win32 API.
You can store it on an ntfs FS, but windows programs won't like it.
Likewise the "\" in that directory path that includes a CR for some reason.

Since you're running chkdsk (a windows program) it will not like those names - that's all.
And the HFS dir is there from some integration test for somethingsomething DMG, which is a macOS specific container/image format.
So nothing to see here either.

Arch Linux

#1 2022-05-20 10:09:02

Corrupted/orphaned files on ntfs

#2 2022-05-20 12:28:18

Re: Corrupted/orphaned files on ntfs

#3 2022-05-20 13:15:33

Re: Corrupted/orphaned files on ntfs

#4 2022-05-20 13:22:14

Re: Corrupted/orphaned files on ntfs

#5 2022-05-20 14:09:03

Re: Corrupted/orphaned files on ntfs

#6 2022-05-20 15:18:54

Re: Corrupted/orphaned files on ntfs

#7 2022-05-20 15:19:53

Re: Corrupted/orphaned files on ntfs

#8 2023-01-06 10:54:06

Re: Corrupted/orphaned files on ntfs

#9 2023-01-06 11:03:19

Re: Corrupted/orphaned files on ntfs

#10 2023-01-06 11:23:29

Re: Corrupted/orphaned files on ntfs

#11 2023-01-06 12:06:10

Re: Corrupted/orphaned files on ntfs

#12 2023-01-06 12:19:40

Re: Corrupted/orphaned files on ntfs

#13 2023-01-06 12:27:12

Re: Corrupted/orphaned files on ntfs

#14 2023-01-06 12:45:49

Re: Corrupted/orphaned files on ntfs

#15 2023-01-24 12:37:35

Re: Corrupted/orphaned files on ntfs

#16 2023-01-24 15:24:58

Re: Corrupted/orphaned files on ntfs

#17 2023-01-27 06:37:32

Re: Corrupted/orphaned files on ntfs

#18 2023-01-27 07:49:52

Re: Corrupted/orphaned files on ntfs

#19 2023-01-27 16:22:43

Re: Corrupted/orphaned files on ntfs

#20 2023-01-27 16:43:35

Re: Corrupted/orphaned files on ntfs

#21 2023-01-27 18:28:20

Re: Corrupted/orphaned files on ntfs

#22 2023-01-27 20:56:00

Re: Corrupted/orphaned files on ntfs

#23 2023-01-28 09:23:02

Re: Corrupted/orphaned files on ntfs

#24 2023-01-28 09:36:37

Re: Corrupted/orphaned files on ntfs

#25 2023-01-28 09:42:47

Re: Corrupted/orphaned files on ntfs

Board footer