everything's gone after a restart

qu@rk · 2021-09-30 08:04:10

Hello all,
I have a Thinkpad t440p with Arch installed. Had a dm-crypt setup for some time now, and everything worked fine. I just updated now, made a network mount in dolphin and restarted the system. At reboot it failed to boot. Used the Arch install stick and I can see the ssd with fdisk -l, but no partitions. cfdisk shows the ssd as being completely free space. I have no idea what to do next, or what happened. It's like my whole setup just vanished. I think the closest analog would be if I'd made a ssd wipe with hdparm, just that I didn't do it.
I have backups so data loss is not an issue, I'm more concerned about this thing happening when I really need to use the laptop on the field. How/why could this happen, and what can I do now to understand what happened?

V1del · 2021-09-30 08:12:27

Hardware failure? Check the disk health with smartctl: https://wiki.archlinux.org/title/S.M.A.R.T. any messages in dmesg?

qu@rk · 2021-09-30 08:42:26

SMART says everything is fine, no issues. There's some stuff in dmesg:

[    5.948163] GPT:first_usable_lbas don't match.
[    5.948166] GPT:2048 != 34
[    5.948167] GPT:disk_guids don't match.
[    5.948168] GPT:partition_entry_array_crc32 values don't match: 0xab54d286 != 0x28c8ce72
[    5.948170] GPT: Use GNU Parted to correct GPT errors.
[   11.062176] systemd-gpt-auto-generator[242]: EFI loader partition unknown, exiting.
[   11.062181] systemd-gpt-auto-generator[242]: (The boot loader did not set EFI variable LoaderDevicePartUUID.)
[  120.074526] GPT:first_usable_lbas don't match.
[  120.074530] GPT:2048 != 34
[  120.074534] GPT:disk_guids don't match.
[  120.074535] GPT:partition_entry_array_crc32 values don't match: 0xab54d286 != 0x28c8ce72
[  120.074538] GPT: Use GNU Parted to correct GPT errors.

Couldn't find anything else.

qu@rk · 2021-09-30 09:05:21

Using gdisk I see this:

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: OK
Main partition table: OK
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Command (? for help): v

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Problem: main GPT header's first usable LBA pointer (2048) doesn't
match the backup GPT header's first usable LBA pointer (34)

Problem: main header's disk GUID (FE16638E-E230-2544-BC19-FB0013E87247) doesn't
match the backup GPT header's disk GUID (585A4737-CBBE-4956-8F6E-9122C5FD8351)
You should use the 'b' or 'd' option on the recovery & transformation menu to
select one or the other header.

Warning: There is a gap between the main partition table (ending sector 33)
and the first usable sector (2048). This is helpful in some exotic configurations,
but is unusual. The util-linux fdisk program often creates disks like this.
Using 'j' on the experts' menu can adjust this gap.

Identified 3 problems!

So there's some issues, but there's no partitions being shown in gdisk. Any chance of recovery if I repair the backup partition table?

edit: found some others online that had issues with network shared storage corrupting GPT tables or the filesystem.

Last edited by qu@rk (2021-09-30 09:06:54)

V1del · 2021-09-30 10:51:40

What exactly did you network mount? Dolphin configures user space network mounts certainly nothing that has even a chance of corrupting the local partition table.

qu@rk · 2021-09-30 10:55:48

a sftp share through ssh non-standard port, using an auth key. I think I chose sftp and not fish.
I tried "e" and "d" repair commands in gdisk, no more issues reported by gdisk but didn't solve the issue, lost everything. I mean I have to reinstall everything.

edit: I tried the same thing on my PC but it's not working. I had issues with errors, not connecting, worked once, then didn't again. OS setup is almost identical to the laptop. Now I'm not touching it anymore on the PC. Out of curiosity I'll try it again on the laptop to see if I get the same issues. But I'm not risking my main PC with this.
On Ubuntu/Gnome worked fine with no issues from my PC.

Last edited by qu@rk (2021-09-30 11:45:22)

seth · 2021-09-30 13:20:43

You lost the partition table, I doubt that's related to dolphin or mounting some network filesystem.
Either you botched some fdisk/parted/gdisk/… interaction of the sector got corrupted - possibly due to SSD trimming or possibly when installing/updating the bootmanager - likely long before the reboot.

qu@rk · 2021-09-30 14:14:41

that's a bit chilling. does this sort of thing happen often?
I can't remember what got updated, been like 5 days since I last updated it. but I basically turned it on, updated, and 20 minutes later I mounted the network share, then rebooted, nothing else.
what would be a mitigation against this sort of thing? raid?

seth · 2021-09-30 14:20:52

It especially happesns if you enabled trimming on a device that doesn't support it.
https://wiki.archlinux.org/title/Solid_state_drive#TRIM

=> Did you, how and on what device?

I basically turned it on, updated, and…

This is likely where stuff broke. Do you recall anything extraordinary during the update?

If you can recover/reconstruct the previous partition table and didn't mess w/ the device otherwise, your data might actually still be there.
Maybe testdisk can reconstruct it, https://wiki.archlinux.org/title/File_r … _utilities

qu@rk · 2021-09-30 14:31:59

I use a SanDisk SD7SN6S ssd, should support it. I am using one of those crappy sata/m2 adapters, so that might also be the culprit. Didn't move the laptop, was on my desk.
Update went smooth, but did spend some time compiling SNORT.
I don't really need the data on the disk, I might try to recover it out of curiosity. But I had the efi partition, /boot on a LUKS1 partition and / on a LUKS2 partition. Might be a bit hard putting it all back together, tho I would really need to recover / only to see what happened.

Arch Linux

#1 2021-09-30 08:04:10

everything's gone after a restart

#2 2021-09-30 08:12:27

Re: everything's gone after a restart

#3 2021-09-30 08:42:26

Re: everything's gone after a restart

#4 2021-09-30 09:05:21

Re: everything's gone after a restart

#5 2021-09-30 10:51:40

Re: everything's gone after a restart

#6 2021-09-30 10:55:48

Re: everything's gone after a restart

#7 2021-09-30 13:20:43

Re: everything's gone after a restart

#8 2021-09-30 14:14:41

Re: everything's gone after a restart

#9 2021-09-30 14:20:52

Re: everything's gone after a restart

#10 2021-09-30 14:31:59

Re: everything's gone after a restart

Board footer