After a fsck on a 933GB etx4 file system I now have 33GB in lost + found. I'm blaming Ubuntu for this problem. Ubuntu would never shut down and I always had to hold the power button down. I've run smart tools on the disk and it report fine.
So a couple of questions.
1. Is the filesystem now safe to use? (smarttools reports the disk is fine)
2. Are the files in lost+found that are actual files document etc safe to just copy out of the directory?
3. Is there anyway to try recover all the #628362 type files?
4. Should I just delete all the files in lost and found?
First of all, I enable the sysreq magic key (in /etc/sysctl.conf). In case of problem, Press Alt-Prtscreen-S (the keyboard is assumed to be US-qwerty) to sync the filesystem, wait there is no activitu on the disk and force the sutdown (you can also Alt-Prtscreen-b to forec reboot). If the Sysreq magic key is not enabled, wait a couple of minutes before forcing the shutdown. Now your specific problem: boot the system with forcefsck or fsck.mode=force as a boot parameter. After that, the filesystrem itself would be fine. However the contents of the files can be problematic and it is up to you to decide if your files are corrupted enough to need a reinstall (although the journal guarantees some consistencies, but I am not sure what). The files that are likely to be corrupted are the files that were written at the moment of the shutdown. I do not think there is another way to recover the files than to copy manually these files out of /lost-found manually.
There is extundelete, for example, that handles lost+found. The system does not really access those files, fsck just throws everything it finds in there. If you are sure nothing is missing, throw it away.