folder (really!) invisible, but accessible (not hidden by ".")

sbt1 · 2024-05-11 09:17:11

Hi,

I've got a strange problem:

A folder (with a "normal" name, not "." in the beginning) from my external HDD suddenly went missing.

Indication:

1. When looking in PCManFM or Dolphin it just isn't there (not hidden CTRL+h doesn't change a thing)
2. with "ls -la" it isn't shown
3. when creating a new folder ("mkdir $FOLDER" or by file manager) with the same name it says that it already exists
4. when accessing in terminal by "cd $FOLDER" I'm in there and can access the contents, "pwd" then shows the correct path
5. when accessing per navigation bar (top of pcmanfm for example) by adding "/$FOLDER" to the path, I'm in there and can use it as normal
6. when checking from windows everything looks fine, folder is there
7. syncthing uses it as usual (as the path to this folder is directly provided, I don't know if I could add these paths again though or how it would behave if I added the parent directory)
8. when backing up the whole drive (means: parent directory of the disappeared folder) by rsync this folder is treated as not-existent (-> --delete removed it at backup target, I lost ~2 TB data at backup location, which was a huge pain to get there in the first place with 1,7MB/s upload speed...)

Probable trigger for this behavior:

I once started syncthing without the external drive attached. It recognized the folder was missing, later everything went as normal.
This may be the reason, because another folder on that drive that is synced by syncthing too (only theses 2 are synced) shows the same behavior. All other folders (untouched by syncthing) behave as expected.

All packages are up to date, drive is formatted NTFS, Windows file system check didn't show any problem.

Has someone ever heard of something like this or has an idea in which direction I need to research? I'm totally out of ideas.

Probable idea: move contents out of directory, try rm $FOLDER and mkdir $FOLDER, move contents back. Or: try to move it around/rename it with "mv" and see what happens. But I'm somewhat hesitant to do anything, as this is such a weird situation I have never heard of and can't classify.

Thanks fo reading & have a nice weekend!

Last edited by sbt1 (2024-05-11 15:35:00)

Lone_Wolf · 2024-05-11 10:18:31

Does the name of the folder end with a $ ?

sbt1 · 2024-05-11 11:10:39

nope, the are called
"3D Printing"
".y"
if this is important. Is strongly suspect syncthing, as these are the folders shared by it.

I'd really like to know whats going on here...

sbt1 · 2024-05-11 12:23:17

okay, I was brave to just

mv .y abc
mv abc .y

and everything looks fine. Perhaps someone has still an idea what might go on here, I'll leave the second folder for further testing, in case someone has an idea in the future.

Still marked as solved though as a simple mv seems to solve the issue.

@Lone_Wolf: Thank you very much for your time!

EDIT: Not solved, sorry for that. When renaming the folder it is shortly visible but then disappears again, so this is no solution. Like there is something on a rampage letting these folders disappear. I killed syncthing beforehand, so this should not be the culprit.

So, when I rename a folder by "mv .y abc" the folder "abc" instantly disappears and shows the same behavior.

The slightest hint how to investigate further would be highly appreciated :-)

Last edited by sbt1 (2024-05-11 15:38:51)

SeagullFish · 2024-05-27 21:33:35

sbt1 wrote:

The slightest hint how to investigate further would be highly appreciated :-)

Hello. Thank you for your post.

I am responding to this thread, because I am experiencing the same (or a very similar) issue.

I don't know exactly what causes this. However, I suspect this is not at all related to Syncthing, PCmanFM, Dolphin or any other file managing applications. My suspicion is that this issue could be related to the NTFS file system itself, based on information from various other threads online. A few examples:

I have an external HDD (Western Digital My Passport, similar to this one). I have formatted it with one single partition of type NTFS. Similar to your case, I am also using mine for backup purposes, although I don't use any type of synchronization application the way you do. Instead, I back up files from the internal storage of my computer by manually copying files using a regular file manager. (I am running Arch Linux with the Cinnamon Desktop Environment. Hence, I am using the included file manager Nemo.)

Recently, the same issue that you are describing also happened to me. I copied files to the external HDD, and subsequently used the "turn off" function provided within the gnome-disk-utility application. Moreover, I physically disconnected the external HDD by pulling the cable. The next time I connected the external HDD again, the symptoms you are describing applied to me:

When browsing the external HDD using Nemo, many files and folders are suddenly "missing". (Turning on the "Show hidden files" feature in Nemo does not change this fact.)
The files and folders are also "missing" when running "ls -al" in a terminal emulator.
If I try to copy one of the "missing" files or folders from my internal storage to the external HDD again, I get a dialogue box telling me that a file or folder with the same name already exists in the target directory, and I am asked if I wish to duplicate, merge or overwrite the file/folder, or cancel the action. (Despite this message, the file/folder is still not visible in Nemo.) I have not tried to actually overwrite anything, by fear of messing things up even more.
I may access a "missing" folder by typing the path name manually without problems, either using the navigation bar in Nemo, or by using cd in a terminal emulator. The contents within the "missing" folder is now visible.
I have not tried connecting the external HDD to a computer running MS Windows after the issue occurred. I really do wonder what the outcome of that would be, but I am worried that such an action might mess things up even more.

From various forum posts, I notice that a recurring point is made that NTFS is a proprietary file system designed to work with MS Windows, and it is not guaranteed to work well with Linux. For that reason, a recurring recommendation is also to stop using NTFS-formatted partitions with Linux, and rather use the Ext4 file format instead.

Nevertheless, I fail to see how such a solution may come without a significant disadvantage. If I wipe my entire external HDD, and create a new Ext4 formatted partition onto it, then how may I be able to read from (or write to) the external HDD if I connect it to a computer running a different operating system, like MS Windows or MacOS? I don't suppose the Ext4 file system is natively supported by Windows or MacOS?

Any suggestions are welcome.

skunktrader · 2024-05-27 21:52:28

If you run chkdsk while the disk is connected to "another" operating system, does it report (and/or fix) any errors?

sbt1 · 2024-05-28 18:05:02

SeagullFish wrote:

sbt1 wrote:
The slightest hint how to investigate further would be highly appreciated :-)
Hello. Thank you for your post.
I am responding to this thread, because I am experiencing the same (or a very similar) issue.
[...]
Any suggestions are welcome.

Thank you very much for your post, as this error is so very specific I thought I was the only one having this problem.

I believe you might be right regarding the whole closed-source-thing - perhaps we both ran into some corner case which would have not occured if we were running Windows:

- when I connected the drive to Windows everything showed up, as if nothing had happened
- I ran the windows filesystem-check (this time directly with repair option though) again, and suddenly this error was gone, everything looks fine from the linux point of view

Do you have a Windows machine to run fschk? (I went the GUI-way -> right click on drive, properties, in the corresponding tab the check/repair function)

I will leave this thread open for you, and mark it [solved] when your problem is also gone :-)

sbt1 · 2024-05-28 18:05:56

skunktrader wrote:

If you run chkdsk while the disk is connected to "another" operating system, does it report (and/or fix) any errors?

That actually did the trick, only "check" didn't find anything, but "check&repair" (in Windows) was the solution.
Thank you very much!

Last edited by sbt1 (2024-05-28 18:06:26)

Lone_Wolf · 2024-05-28 19:19:39

Please prepend [Solved] to the thread title (edit first post)

kokoko3k · 2024-05-28 19:31:39

Just for reference, is it mounted via ntfs-3g (fuse) or ntfs3 (kernel)?

SeagullFish · 2024-05-28 21:07:13

sbt1 wrote:

I will leave this thread open for you, and mark it [solved] when your problem is also gone :-)

Ok. I'll see what I can do. Right now I don't have access to a computer running MS Windows. So it may take a few days before I am able to follow up on this. Hopefully, the issue will be solved for my part as well.

kokoko3k wrote:

Just for reference, is it mounted via ntfs-3g (fuse) or ntfs3 (kernel)?

The partition is auto-mounted when I connect the cable from the external HDD to the computer, so I honestly don't know what software is actually performing the mounting. How can I find out?

V1del · 2024-05-28 22:39:12

check the output of findmnt or mount. If it's automounted likely by udisks and that recently switched to ntfs3 by default, and ntfs3 is somewhat notorious for having a variety of potential corruption issues (things seem to have gotten better recently though) but if you're using it you better have some form of Windows with chkdsk handy.

kokoko3k · 2024-05-29 04:26:10

How can I find out?

So I guess is ntfs3, is ntfs-3g package even installed?

Btw, just type 'mount' and see the line that matches your device.

Edit:
Ops, missed the very same reply from V1del.

Last edited by kokoko3k (2024-05-29 04:27:32)

SeagullFish · 2024-05-29 19:49:55

kokoko3k wrote:

So I guess is ntfs3, is ntfs-3g package even installed?

Apparently, yes:

 [foo@bar ~]$ sudo pacman -Qi ntfs-3g
Name                   : ntfs-3g
Version                : 2022.10.3-1
Description            : NTFS filesystem driver and utilities
[...]
Offering               : ntfsprogs
Depends on             : util-linux  fuse2
Required by            : mintstick
Otional for            : gparted  libblockdev-fs
Collides with          : ntfsprogs
Replaces               : ntfsprogs
[...]
Installation reason    : Installed as a dependency for another package

V1del wrote:

check the output of findmnt or mount. If it's automounted likely by udisks and that recently switched to ntfs3 by default [...]

kokoko3k wrote:

Btw, just type 'mount' and see the line that matches your device.

This is the output from findmnt (truncated for privacy/brevity):

[foo@bar ~]$ sudo findmnt
TARGET                         SOURCE          FSTYPE          OPTIONS
[...]
├─/run                         run             tmpfs           rw,nosuid,nodev,relatime,mode=755,inode64
│ ├─/run/media/foo/Ext_HDD     /dev/sda1       ntfs3           rw,nosuid,nodev,relatime,uid=1000,gid=1000,iocharset=utf8
[...]

This is the output from mount (also truncated):

[foo@bar ~]$ sudo mount
[...]
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
[...]
/dev/sda1 on /run/media/foo/Ext_HDD type ntfs3 (rw,nosuid,nodev,relatime,uid=1000,gid=1000,iocharset=utf8,uhelper=udisks2)

cryptearth · 2024-05-30 20:01:27

SeagullFish wrote:

Nevertheless, I fail to see how such a solution may come without a significant disadvantage. If I wipe my entire external HDD, and create a new Ext4 formatted partition onto it, then how may I be able to read from (or write to) the external HDD if I connect it to a computer running a different operating system, like MS Windows or MacOS? I don't suppose the Ext4 file system is natively supported by Windows or MacOS?
Any suggestions are welcome.

Use exFAT - M$ made it pubic back in 2019 so proper drivers could be developed.

sbt1 wrote:

skunktrader wrote:
If you run chkdsk while the disk is connected to "another" operating system, does it report (and/or fix) any errors?
That actually did the trick, only "check" didn't find anything, but "check&repair" (in Windows) was the solution.
Thank you very much!

Although long time since - when I used chkdsk to get systems of my friends back up I did it this way:
booting a live WinPE system - so the OS doesn't lock me out
use CHKDSK with these options:
/F /R /X /B /V
yes - depending on the drive and interface it can take forever - but that's pretty much the full brute force to get a NTFS back in order with windows' own tools

Last edited by cryptearth (2024-05-30 20:05:39)

SeagullFish · 2024-06-15 13:18:12

sbt1 wrote:

Do you have a Windows machine to run fschk? (I went the GUI-way -> right click on drive, properties, in the corresponding tab the check/repair function)
I will leave this thread open for you, and mark it [solved] when your problem is also gone :-)

Sorry for my late response. It took me a while to get arranged with a computer running MS Windows, as I have been busy with other things.

skunktrader wrote:

If you run chkdsk while the disk is connected to "another" operating system, does it report (and/or fix) any errors?

Thank you for your suggestion. This solved the issue also in my case.

Contrary to sbt1, I executed MS chkdsk from MS cmd (Windows command prompt). Hence, chkdsk provided some output. I don't know if this information may contribute any value to the discussion, but in case it does, I am pasting it here (censored for privacy reasons):

C:\>chkdsk D: /F
The type of the file system is NTFS.
Volume label is Ext_HDD.

Stage 1: Examining basic file system structure ...
  38144 file records processed.
File verification completed.
 Phase duration (File record verification): 1.63 seconds.
  9 large file records processed.
 Phase duration (Orphan file record recovery): 0.00 milliseconds.
  0 bad file records processed.
 Phase duration (Bad file record checking): 0.43 milliseconds.

Stage 2: Examining file name linkage ...
Deleted invalid filename blob?download (1821) in directory 1802.
File 1821 has been orphaned since all its filenames were invalid
Windows will recover the file in the orphan recovery phase.
Correcting minor file name errors in file 1821.
  90 reparse records processed.
Correcting error in index $I30 for file 43.
CHKDSK discovered free space marked as allocated in the bitmap for index $I30 for file 43.
Sorting index $I30 in file 43.
Deleting index entry blob?download in index $I30 of file 1802.
  44822 index entries processed.
Index verification completed.
 Phase duration (Index verification): 11.16 seconds.
CHKDSK is scanning unindexed files for reconnect to their original directory.
Recovering orphaned file [censored] (44) into directory file 43.
Recovering orphaned file [censored] (4B) into directory file 43.
Recovering orphaned file [censored] (4F) into directory file 43.
Recovering orphaned file [censored] (1108) into directory file 43.
Recovering orphaned file [censored] (1108) into directory file 43.
Recovering orphaned file [censored] (110B) into directory file 43.
Recovering orphaned file [censored] (110D) into directory file 43.
Recovering orphaned file [censored] (110D) into directory file 43.
Recovering orphaned file [censored] (110F) into directory file 43.
Recovering orphaned file [censored] (1110) into directory file 43.
Skipping further messages about recovering orphans.
  1194 unindexed files scanned.
  38 unindexed files recovered to original directory.
 Phase duration (Orphan reconnection): 0.00 milliseconds.
CHKDSK is recovering remaining unindexed files.
  1156 unindexed files recovered to lost and found.
    Lost and found is located at \found.000

 Phase duration (Orphan recovery to lost and found): 0.00 milliseconds.
  90 reparse records processed.
 Phase duration (Reparse point and Object ID verification): 2.46 milliseconds.

Stage 3: Examining security descriptors ...
Security descriptor verification completed.
 Phase duration (Security descriptor verification): 78.36 milliseconds.
  3340 data files processed.
 Phase duration (Data attribute verification): 1.48 milliseconds.
CHKDSK is verifying Usn Journal...
Usn Journal verification completed.
Correcting errors in the master file table's (MFT) BITMAP attribute.

Windows has made corrections to the file system.
No further action is required.

[censored] KB total disk space.
[censored] KB in 33570 files.
[censored] KB in 3343 indexes.
         0 KB in bad sectors.
[censored] KB in use by the system.
[censored] KB occupied by the log file.
[censored] KB available on disk.

[censored] bytes in each allocation unit.
[censored] total allocation units on disk.
[censored] allocation units available on disk.
Total duration: 38.76 seconds (38766 ms).

The new directory named "found.000" contains restored files that I consider to be garbage. (That is, files that I have previously deleted from the HDD, etc.).

Lone_Wolf wrote:

Please prepend [Solved] to the thread title (edit first post)

@sbt1: You may change the status of this thread to [SOLVED] now.

cryptearth · 2024-06-15 18:48:21

although thanks for the reply I would still recommend:
- run CHKDSK again but with the whole suite of: /F /R /X /B /V - as this is an external drive there's no need for winPE, that's only for a system drive
- if you plan to further use it to exchange files between windows and linux: backup all data from it and reformat it as exfat - this will prevent such issues in the future

Also: You can mark your topic as solved yourself by editing the first post.

Last edited by cryptearth (2024-06-15 18:48:46)

Lone_Wolf · 2024-06-16 10:25:51

cryptearth wrote:

Also: You can mark your topic as solved yourself by editing the first post.

Only the thread starter can do that and that's sbt1 .

cryptearth · 2024-06-16 13:37:25

Lone_Wolf wrote:

cryptearth wrote:
Also: You can mark your topic as solved yourself by editing the first post.
Only the thread starter can do that and that's sbt1 .

Failed to notice - maybe splitting into two topics then?

Arch Linux

#1 2024-05-11 09:17:11

folder (really!) invisible, but accessible (not hidden by ".")

#2 2024-05-11 10:18:31

Re: folder (really!) invisible, but accessible (not hidden by ".")

#3 2024-05-11 11:10:39

Re: folder (really!) invisible, but accessible (not hidden by ".")

#4 2024-05-11 12:23:17

Re: folder (really!) invisible, but accessible (not hidden by ".")

#5 2024-05-27 21:33:35

Re: folder (really!) invisible, but accessible (not hidden by ".")

#6 2024-05-27 21:52:28

Re: folder (really!) invisible, but accessible (not hidden by ".")

#7 2024-05-28 18:05:02

Re: folder (really!) invisible, but accessible (not hidden by ".")

#8 2024-05-28 18:05:56

Re: folder (really!) invisible, but accessible (not hidden by ".")

#9 2024-05-28 19:19:39

Re: folder (really!) invisible, but accessible (not hidden by ".")

#10 2024-05-28 19:31:39

Re: folder (really!) invisible, but accessible (not hidden by ".")

#11 2024-05-28 21:07:13

Re: folder (really!) invisible, but accessible (not hidden by ".")

#12 2024-05-28 22:39:12

Re: folder (really!) invisible, but accessible (not hidden by ".")

#13 2024-05-29 04:26:10

Re: folder (really!) invisible, but accessible (not hidden by ".")

#14 2024-05-29 19:49:55

Re: folder (really!) invisible, but accessible (not hidden by ".")

#15 2024-05-30 20:01:27

Re: folder (really!) invisible, but accessible (not hidden by ".")

#16 2024-06-15 13:18:12

Re: folder (really!) invisible, but accessible (not hidden by ".")

#17 2024-06-15 18:48:21

Re: folder (really!) invisible, but accessible (not hidden by ".")

#18 2024-06-16 10:25:51

Re: folder (really!) invisible, but accessible (not hidden by ".")

#19 2024-06-16 13:37:25

Re: folder (really!) invisible, but accessible (not hidden by ".")

Board footer