Arch Linux

Rec100

Member

Registered: 2022-03-12

Posts: 29

[SOLVED] Can not login after update - I/O Errors

Not sure if this is the correct forum for my problem, but at least, I assume it's caused by an ungracefully finished update.

I ran pacman -Syu on April 8th after reading about the adjustment of the vm.max_map_count paramater. I did not reboot but kept on working.

On April 9th I ran a fresh backup of my system, copying ~1TB of data to a microSD card. Of course this takes a while. When I came back to the machine a couple of hours later, the laptop was partly unresponsive and in some parts of the Gnome GUI, font characters were replaced by blocks (█). So, after a while I decided to just power off.

When switching the laptop on, I can access BIOS, and get to the luksdev password prompt. Afterwards, I face three different, random scenarios:

Scenario 1: I see I/O errors, from fsck - ERROR: Bailing out. Run 'fsck /dev/mapper/luksdev' manually - when I do this, even several times (for about 20 minutes), nothing is fixed. However, I have a prompt.

Scenario 2: I see the boot process with many errors, not getting anywhere, not having a prompt.

Scenario 3: The boot process looks fairly good (all green, not reds) and I end up with a GNOME log in screen. Which does not allow me to login. It seems just animations work (for example, the power off button does not work). When I click on the date, I see the bricked fonts again █. I got wifi, bluetooth, etc. But I can not log in. TTY1-7 just show a blinking underscore, but do not accept input. So, I'd say, no prompt here. This is exactly the situation I was in, after coming back to the machine, to check on the status of the backup.

In BIOS (Lenovo) I ran quick hardware diagnostics:

- Storage:
  - Device read test: Failed
  - NVMe  read test: Failed

I am currently continuing the tests, but this takes time.

The machine is a Lenovo ThinkPad X1 Carbon Gen 9 by the way, 2TB SSD.

I wonder what happened and how I can fix the situation.

Any ideas?

Last edited by Rec100 (2024-04-12 08:53:51)

xerxes_

Member

Registered: 2018-04-29

Posts: 681

Re: [SOLVED] Can not login after update - I/O Errors

Did system update finished successfully and you started doing backup or while you were updating the system you start backup, but you didn't seen if update finished without error(s)? How big that update was - amount of packages to update? Can you arch-chroot from archiso to your system?

Rec100

Member

Registered: 2022-03-12

Posts: 29

Re: [SOLVED] Can not login after update - I/O Errors

The update finished successfully, but I did not reboot. I did the backup after the update finished. When I updated I just saw a few packages, about 15-20. I had updated with reboot two days earlier.

Regarding archiso, do you mean a live USB? I did not have the time to try that, yet. Will create a medium in about an hour or so. The laptop still does BIOS diagnostics. This may take about 5 hours to complete. I will not interrupt.

seth

Member

Registered: 2012-09-03

Posts: 51,796

Re: [SOLVED] Can not login after update - I/O Errors

So, after a while I decided to just power off.

Try to always avoid that, check whether you can switch VT, ssh in or shutdown/reboot w/ (frenetic use of) ctrl+alt+del or a short push on the power button.

In BIOS (Lenovo) I ran quick hardware diagnostics:
- Storage:
  - Device read test: Failed
  - NVMe  read test: Failed

That's bad.
=> grml.org (you want to do this from a live distro)
https://wiki.archlinux.org/title/S.M.A.R.T.
https://wiki.archlinux.org/title/Badblocks # BE CAREFUL WITH THAT! One bad mishap on an encrypted device can effectively close the device.

Do you have backups of your data?
If you don't, try to open and mount the device from grml and backup all/as much private data as you can (the valuable stuff is anything you cannot just download from the internet again)

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Rec100

Member

Registered: 2022-03-12

Posts: 29

Re: [SOLVED] Can not login after update - I/O Errors

Headaches.

I have two backups of everything. I was just in the process of renewing the backup sd cards. Just running a rsync for a couple of hours syncing my files to the first of the two new sd cards. So, the backup I did when the problem occurred is not important.

Should I refrain from a fresh install on the same laptop disk? I'm wondering if there's a hardware failure, or if this is something I can "fix" with a fresh install. I have the install medium ready for use.

--- edit

Currently performing a non-destructive Read-write test with badblocks from the USB flash installer. I see many errors. 1200+ errors at 5% into the test (more than 950 bad blocks).
As stated here - https://wiki.archlinux.org/title/Badblo … e_fidelity - "Although there is no firm rule, it is common thinking that a new drive should have zero bad sectors. Over time, bad sectors will develop and although they are able to be defined to the file system so that they are avoided, continual use of the drive will usually result in additional bad sectors forming and it is usually the harbinger of its eventual death. Replacing the device is recommended."

I guess it's time for a new disk.

Last edited by Rec100 (2024-04-11 23:20:17)