Laptop incapable of running commands, now can't boot Arch ISO

1ptb3b · 2021-08-23 21:30:26

For the past few months, my Dell XPS 13 9380 has been getting in a weird state where I can't run anything or shutdown Gnome. I would just be browsing online and all of a sudden I would notice I can't open any more apps. If I happened to have gnome-terminal running, then whenever I tried running _any_ command I would get something like this.

$ ls
/bin/ls: Input/output error

When the laptop would get like this, I would usually have to press and hold the power button to reboot everything. That would usually fix it.

Until last night.

Now whenever I try to boot, I see GDM, NetworkManager, CUPS, and a lot of other services fail to start up. I tried switching to another TTY to log in, but strangely I can only enter my username because immediately after that I get an error that says:

login: me
Incorrect login

I never get a chance to enter my password.

I've also tried rebooting and pressing F2 to run the Dell BIOS Diagnostics. I've run it a few times and it shows there are no errors.

Finally, I decided to try to boot to an Arch ISO. I made sure the ISO works by booting into the shell on my X1 Carbon. Everything seems fine on the X1. But when I tried booting the ISO on the XPS, I get this error (along with many other systemd failed to start X errors) and I never get to the shell.

tmp.mount: Failed to execute /usr/bin/mount: Input/output error
tmp.mount: Failed at step EXEC spawning /usr/bin/mount: Input/output error
Failed to mount Temporary Directory /tmp
SQUASHFS error: Unable to read fragment cache entry
SQUASHFS error: Unable to read page, block 2566a02, size 48868

Is there anything else I can try to debug this issue?

Last edited by 1ptb3b (2021-08-23 21:33:43)

Trilby · 2021-08-23 21:38:24

Looks like bad ram. If you're lucky, it just could be a ram chip jostled out of place, though I don't know if that'd really give these symptoms.

Run a memtest. I think this *might* be included with syslinux on the iso, but it's been a while since I needed it. There are ample tiny memtest isos around out there that you could throw on any flash drive to boot to.

progandy · 2021-08-23 21:38:36

That might be a problem with your RAM. Maybe try to boot and run memtest.
The RAM is soldered in that model, so you'll have to ask dell for help or find someone who can do that repair. If you still have warranty use that.

Last edited by progandy (2021-08-23 21:40:37)

1ptb3b · 2021-08-23 21:56:16

How do I run a memtest?

I found this in the wiki: https://wiki.archlinux.org/title/Syslinux#Using_memtest , but I don't quite understand how I'm supposed to use that file. Am I supposed to edit the Arch ISO I'm booting from? Am I supposed to edit something through the EFI Shell? I can't get to the ZSH shell that the ISO normally gives you.

I also found this: http://www.memtest.org/#downiso , but I'm not sure if I need the bootable ISO or the bootable binary. What's the difference between bootable ISO and binary?

Ooh, and there is also this: https://www.memtest86.com/ (proprietary) and https://www.memtest86.com/tech_creating-linux-mac.html

Last edited by 1ptb3b (2021-08-23 22:12:21)

1ptb3b · 2021-08-24 01:59:46

Ok, I ended up doing a memtest with https://www.memtest86.com/ and I used the instructions here https://www.memtest86.com/tech_creating-linux-mac.html to make a bootable USB.

I ran the test and got a big green PASS. Here is the result summary.

MemTest86 V9.2 Free (Build: 2000) Result summary
PassMark Software
www.passmark.com

Test Start Time			: 2021-08-23 22:04:27
Elapsed Time			: 3:19:37
CPUs Active			: 4
CPU Temperature Min/Max/Ave	: 59C/83C/72C
RAM Temperature Min/Max/Ave	: -/-/-
# Tests Passed			: 48/48 (100%)

Lowest Error Address		: N/A
Highest Error Address		: N/A
Bits in Error Mask		: 00000000000000
Bits in Error			: Total: 0 Min: 0 Max: 0 Avg: 0
Max Contiguous Errors		: 0

<Press any key to continue>

seth · 2021-08-24 06:26:44

You have to run maaaany cycles for significant results ("over night" is short…)

But if it's not RAM, pci/usb may be dropping out, you get an I/O error because the disk isn't there anymore.
Can you boot the install iso w/

acpi=off iommu=soft pcie_aspm=off nomodeset pci=noacpi

(it's a broad collection of parameters - if it works, we'll need to filter this down)

1ptb3b · 2021-08-24 06:49:15

After some more Googling, it seems like maybe the issue is faulty storage.

* https://linoxide.com/inputoutput-error- … tart-linux
* https://superuser.com/questions/225788/ … tput-error
* https://unix.stackexchange.com/question … y-commands
* https://wiki.archlinux.org/title/badblocks

I'm investigating this clue currently.

Also, I've discovered that if I try to boot the ISO enough times, every now and then I can get to a shell and can connect to the internet. I should probably try to get my data out...

seth · 2021-08-24 06:51:46

Disk errors are the common IO error explanation, but they don't cover the iso failing as well.
The error had to be higher up, which is the bus.

1ptb3b · 2021-08-24 17:31:20

I rebooted the ISO and pressed e to edit the first boot entry. I then added

acpi=off iommu=soft pcie_aspm=off nomodeset pci=noacpi

I was able to boot to the shell previously, but now I just get a black screen.

After a minute or so I got this.

NMI watchdog: Watchdog detected hard LOCKUP on cpu 0

Last edited by 1ptb3b (2021-08-24 17:32:04)

1ptb3b · 2021-08-24 17:41:21

I now have a suspicion it's maybe battery related. I saw previously in the BIOS that my battery was at 77%. That was around the same time I was having problems booting the ISO. After leaving the laptop plugged in for a few hours, I'm guessing the battery is now full. Miraculously, I can now more consistently boot the ISO. The OS was stable enough to allow me to export my data (with tar and nc).

I'm currently running a read-write test with badblocks. https://wiki.archlinux.org/title/badblo … structive)

# badblocks -nsv /dev/device

After this, I want to test if I can boot into the laptop's OS while plugged into power. And I also want to test if things start failing again once I start running on the battery.

Last edited by 1ptb3b (2021-08-24 20:59:51)

Trilby · 2021-08-24 20:40:43

Your symptoms rule out a block storage device issue: booting to the iso does not even require an internal drive to be present.

As noted above bad RAM is still a prime suspect unless you have done a full test on it, alternatively it could be a data-bus issue ... but not a internal drive problem, so running badblocks doesn't seem relevant.

(edit: strictly speaking your symptoms don't actually rule out a drive issue ... it's possible the drive and memory or bus are bad - but quite unparsimonious if you haven't recently played smash-the-laptop. In any case, a bad drive is not relevant to your symptoms and so there is no reason to suspect it).

Last edited by Trilby (2021-08-24 20:42:46)

seth · 2021-08-24 20:48:34

I want to test if I can boot into the laptop's OS while plugged into power. And I also want to test if things start failing again once I start running on the battery.

Did you run memtest on external power or the battery?

If the former, my money is here. If the battery cannot sufficiently supply the system, all sorts of weird things can happen as parts of the HW randomly drop out.
If you suspect the battery to be the cause, try to bypass it (ie. operate the notebook on the AC and w/ the battery removed - if the BIOS lets you)

1ptb3b · 2021-08-24 20:52:37

Hm. The badblocks test came back without any errors.

As for the hypothesis that having the battery at 100% fixed things, here's the results.
* Rebooted to laptop's OS with battery at 100% and plugged in -> continues to fail to boot
* Rebooted to ISO with battery at 100% and plugged in -> still flaky, getting "ERROR: /dev/disk/by-label/ARCH_202108... device did not show up after 30 seconds..." off and on

Also, I tried adding these parameters to my laptop's default boot entry.

acpi=off iommu=soft pcie_aspm=off nomodeset pci=noacpi

I'm getting this output.

pcieport 0000:00:1d.0: pciehp: Slot #12 AttnBtn- PwrCtrl- MRL- AttnInd- HotPlug+ Surprise+ Interlock- NoCompl+ IbPresDis- LLActRep+
pcieport 0000:00:1d.4: can't find IRQ for PCI INT A: please tryusing pci=biosirq
Serial: 2850/16550 driver, 32 ports, IRQ sharing enabled
nvme 0000:6e:00.0: can't find IRQ for PCI INT A; please try using pci=biosirq
NMI watchdog: Watchdog detected hard LOCKUP on cpu 0

seth · 2021-08-24 21:01:20

The kernel wrote:

please try using pci=biosirq

Also

nvme_core.default_ps_max_latency_us=0

Also check

cat /proc/acpi/battery/BAT*/state # this is not going to work if you disabled acpi w/ a kernel parameter

progandy · 2021-08-24 21:43:48

Maybe some of the capacitors (or some other parts) in the power supply circuits on the board broke. That can cause those issues as well.

Here is a thread about a 9350 rendered unbootable, dell claimed the circuitry was damaged by having it constantly connected to the charger: https://www.dell.com/community/XPS/XPS- … -p/6080979

Last edited by progandy (2021-08-24 21:44:17)

1ptb3b · 2021-08-24 22:48:59

Aaaahaaa!!! I managed to boot back to the laptop OS!!!

I think I might have hit one of these weird failure states while I was running pacman a few days ago. So looks like the reason I couldn't boot normally was because my packages were all messed up. I tried reinstalling all my packages again.

I used these links:
* https://wiki.archlinux.org/title/Chroot#Using_chroot
* https://wiki.archlinux.org/title/pacman … dependency
* https://wiki.archlinux.org/title/pacman … )%22_error

And ended up running this command a few times (each time seemed to build a little more):

pacman --sysroot /mnt -Syu $(pacman --sysroot /mnt -Qnq) --overwrite="*"

Still not exactly sure what's causing these failure states though.

seth · 2021-08-24 22:50:49

So looks like the reason I couldn't boot normally was because my packages were all messed up.

This cannot explain the failure to boot the installation iso.

1ptb3b · 2021-08-24 22:56:45

Here's what upower says, running on bad laptop.

[baddie ~]$ upower -i /org/freedesktop/UPower/devices/line_power_AC
  native-path:          AC
  power supply:         yes
  updated:              Tue 24 Aug 2021 03:37:13 PM PDT (895 seconds ago)
  has history:          no
  has statistics:       no
  line-power
    warning-level:       none
    online:              yes
    icon-name:          'ac-adapter-symbolic'

[baddie ~]$ upower -i /org/freedesktop/UPower/devices/battery_BAT0
  native-path:          BAT0
  vendor:               LGC-LGC6.73
  model:                DELL H754V8C
  serial:               462
  power supply:         yes
  updated:              Tue 24 Aug 2021 03:53:17 PM PDT (10 seconds ago)
  has history:          yes
  has statistics:       yes
  battery
    present:             yes
    rechargeable:        yes
    state:               fully-charged
    warning-level:       none
    energy:              40.7436 Wh
    energy-empty:        0 Wh
    energy-full:         40.7436 Wh
    energy-full-design:  51.9992 Wh
    energy-rate:         0 W
    voltage:             8.74 V
    percentage:          100%
    capacity:            78.3543%
    technology:          lithium-ion
    icon-name:          'battery-full-charged-symbolic'

Is that voltage ok? I checked on my X1 and it's up to 15V, granted it's running on battery and not plugged in. I'm a little scared to unplug the XPS now. XD

Ah, ok. According to this, I think that voltage may be fine. (reports 7.60 VDC)
* https://www.dell.com/support/manuals/en … lang=en-us

Last edited by 1ptb3b (2021-08-24 23:05:48)

1ptb3b · 2021-08-24 23:14:28

Tried booting the ISO again, pressed e to edit params and added these.

acpi=off iommu=soft pcie_aspm=off nomodeset pci=biosirq nvme_core.default_ps_max_latency_us=0

Looks like it's hanging again. There's a single line so far.

PCI: Unknown option `biosirq'

seth · 2021-08-25 19:48:55

Checked, biosirq is x86-32 only

The voltage is for the fully charged battery - has the device so far been reliable under those circumstances?
Does the voltage drop (fast) if you unplug it and the battery drains?

1ptb3b · 2021-08-25 22:48:05

Hmmm... Weird state again. I've been running the laptop plugged in 100% of the time for the past 2 days and now I've hit the weird state again.

For some reason, I can run some commands, but not others. Since I can't use upower anymore, I used cat to print out some battery info.

$ cat /sys/class/power_supply/BAT0/voltage_now # microvolts
8696000
$ cat /sys/class/power_supply/BAT0/current_now # microamps
1000
$ cat /sys/class/power_supply/BAT0/status
Full
$ cat /sys/class/power_supply/BAT0/charge_now # coulombs?
5361000

# These commands work.
* cd
* ls
* which
* sed
* clear
* nautilus
* echo
* vim (except for no writing)
* journalctl

# These commands DONT work. (Input/output erorr)
* upower
* go
* find
* touch
* watch
* firefox
* sort
* df
* du
* stat
* ld
* nc
* less
* more
* lsblk
* man
* sudo

The filesystem has gone into read-only mode.

$ echo "hi" > test.txt
bash: test.txt: Read-only file system

Here are the journalctl logs of the boot with the weird state.
https://drive.google.com/file/d/1CNDTqr … sp=sharing

The last entry in the logs is Aug 25 14:53:05, but I kept the laptop running at least until 15:30 when I decided to press and hold the power button. I'm guessing shortly after 14:53 the file system went into read-only mode.

Does the voltage drop (fast) if you unplug it and the battery drains?

The voltage seems to stay stable after 5 minutes of running on battery.

Last edited by 1ptb3b (2021-08-25 22:59:13)

Arch Linux

#1 2021-08-23 21:30:26

Laptop incapable of running commands, now can't boot Arch ISO

#2 2021-08-23 21:38:24

Re: Laptop incapable of running commands, now can't boot Arch ISO

#3 2021-08-23 21:38:36

Re: Laptop incapable of running commands, now can't boot Arch ISO

#4 2021-08-23 21:56:16

Re: Laptop incapable of running commands, now can't boot Arch ISO

#5 2021-08-24 01:59:46

Re: Laptop incapable of running commands, now can't boot Arch ISO

#6 2021-08-24 06:26:44

Re: Laptop incapable of running commands, now can't boot Arch ISO

#7 2021-08-24 06:49:15

Re: Laptop incapable of running commands, now can't boot Arch ISO

#8 2021-08-24 06:51:46

Re: Laptop incapable of running commands, now can't boot Arch ISO

#9 2021-08-24 17:31:20

Re: Laptop incapable of running commands, now can't boot Arch ISO

#10 2021-08-24 17:41:21

Re: Laptop incapable of running commands, now can't boot Arch ISO

#11 2021-08-24 20:40:43

Re: Laptop incapable of running commands, now can't boot Arch ISO

#12 2021-08-24 20:48:34

Re: Laptop incapable of running commands, now can't boot Arch ISO

#13 2021-08-24 20:52:37

Re: Laptop incapable of running commands, now can't boot Arch ISO

#14 2021-08-24 21:01:20

Re: Laptop incapable of running commands, now can't boot Arch ISO

#15 2021-08-24 21:43:48

Re: Laptop incapable of running commands, now can't boot Arch ISO

#16 2021-08-24 22:48:59

Re: Laptop incapable of running commands, now can't boot Arch ISO

#17 2021-08-24 22:50:49

Re: Laptop incapable of running commands, now can't boot Arch ISO

#18 2021-08-24 22:56:45

Re: Laptop incapable of running commands, now can't boot Arch ISO

#19 2021-08-24 23:14:28

Re: Laptop incapable of running commands, now can't boot Arch ISO

#20 2021-08-25 19:48:55

Re: Laptop incapable of running commands, now can't boot Arch ISO

#21 2021-08-25 22:48:05

Re: Laptop incapable of running commands, now can't boot Arch ISO

Board footer