Archiso hangs when SD card present at boot in RTS5260 Reader

nix-a-churl · 2022-02-06 17:04:10

Hey folks,

This is my first post here on the forum, and I'm also new to Arch, so apologies in advance for mistakes.

Summary

I've followed the wiki to install Arch on a Dell XPS 13 2-in-1 (9310 2n1), booting to LVM (root, swap, home) on LUKS from a unified kernel image on an SD card.

The system hangs when booting with an SD card in the integrated reader. UEFI firmware reads the card and starts the kernel without any issue, but the kernel then hangs while trying to access the LUKS keyfile from the card. Provided no card is present at boot, the integrated SD card reader seems fine with Arch generally, and mounts/reads/writes without issue. The SD card reader seems fine with Windows 10 irrespective of whether or not a card is present at boot.

After some investigation I still couldn't find the cause for this behaviour. Testing with Archiso gives the same problem, however, suggesting the conflict is not specific to my installation. The difference is that my installation hangs with no access to console as the root account is locked, whereas Archiso does manage to finally boot, albeit with great delay compared to booting with an empty card reader.

Having noted this, I'll post output and discuss the problem as it relates to Archiso in the first instance, which should also help reproduceability. Hopefully, solving the problem with Archiso may also solve the problem with my installation.

I'd really appreciate any help or advice anyone can offer, even if just suggestions as to where to look for more ideas, as I've been struggling with this for a while. Please don't hesitate to let me know if more information would help!

Details

Here are the details of the SD card reader.

# lspci -k | grep -A 3 Card
0000:71:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5260 PCI Express Card Reader (rev 01)
    Subsystem: Dell Device 09ff
    Kernel driver in use: rtsx_pci
    Kernel modules: rtsx_pci

While booting, the following console messages drew my attention. These start jobs seem to hang for quite a while.

[      ] (2 of 3) A start job is running for Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling (8s / no limit)
[      ] (3 of 3) A start job is running for Wait for udev To Complete Device Initialization (8s / 3min)
[FAILED] Failed to start Wait for udev To Complete Device Initialization.
See 'systemctl status systemd-udev-settle.service' for details.
INFO: task blkid:318 blocked for more than 122 seconds.
      Tainted: P           OE     5.16.4-arch1-1 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
INFO: task lvm:328 blocked for more than 122 seconds.
      Tainted: P           OE     5.16.4-arch1-1 #1
A start job is running for Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling (3min 25s / no limit)
I/O error, dev mmcblk0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
A start job is running for Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling (5min 16s / no limit)
A start job is running for Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling (5min 26s / no limit)
Finished Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.

When Archiso reaches a prompt, invoking `systemctl status systemd-udev-settle.service` yields the following.

× systemd-udev-settle.service - Wait for udev To Complete Device Initialization
     Loaded: loaded (/usr/lib/systemd/system/systemd-udev-settle.service; static)
     Active: failed (Result: exit-code) since Sun 2022-02-06 14:46:01 UTC; 3min 45s ago
       Docs: man:systemd-udev-settle.service(8)
    Process: 439 ExecStart=udevadm settle (code=exited, status=1/FAILURE)
   Main PID: 439 (code=exited, status=1/FAILURE)
        CPU: 3.291s

Feb 06 14:44:01 archiso systemd[1]: Starting Wait for udev To Complete Device Initialization...
Feb 06 14:44:01 archiso udevadm[439]: systemd-udev-settle.service is deprecated. Please fix livecd-alsa-unmuter.service not to pull it in.
Feb 06 14:46:01 archiso systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE
Feb 06 14:46:01 archiso systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'.
Feb 06 14:46:01 archiso systemd[1]: Failed to start Wait for udev To Complete Device Initialization.
Feb 06 14:46:01 archiso systemd[1]: systemd-udev-settle.service: Consumed 3.291s CPU time.

The following are relevant lines from journalctl -b

kernel: mmc0: cannot verify signal voltage switch
kernel: mmc0: new ultra high speed SDR104 SDHC card at address 0001
kernel: mmcblk0: mmc0:0001 00000 29.3 GiB 
kernel: I/O error, dev mmcblk0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
kernel: Buffer I/O error on dev mmcblk0, logical block 0, async page read
systemd-udevd[346]: mmc0:0001: Worker [375] processing SEQNUM=3825 is taking a long time
systemd-udevd[346]: mmc0:0001: Worker [375] processing SEQNUM=3825 killed
systemd-udevd[346]: mmc0:0001: Worker [375] failed
kernel:  mmcblk0: unable to read partition table
systemd-udevd[346]: mmcblk0: Worker [446] processing SEQNUM=5090 is taking a long time
kernel: I/O error, dev mmcblk0, sector 61405056 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
kernel: I/O error, dev mmcblk0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
kernel: I/O error, dev mmcblk0, sector 61405056 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
kernel: Buffer I/O error on dev mmcblk0, logical block 7675632, async page read
systemd-udevd[346]: mmcblk0: Worker [446] processing SEQNUM=5090 killed
systemd-udevd[346]: mmcblk0: Worker [446] failed
kernel: mmc0: card 0001 removed
systemd-udevd[346]: mmcblk0: Worker [1178] processing SEQNUM=5133 is taking a long time

Having booted with the card in the reader, removing and reinserting the card gives the following.

root@archiso ~ # udevadm monitor
monitor will print the received events for:
UDEV - the euent which udev sends out after rule processing
KERNEL - the kernel uevent
KERNEL[516.531230] remove   /devices/virtual/bdi/179:0 (bdi)
KERNEL[516.531357] remove   /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001/block/mmcblk0 (block)
UDEV  [516.531487] add      /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001/block/mmcblk0 (block)
UDEV  [516.531829] remove   /devices/virtual/bdi/179:0 (bdi)
UDEV  [516.532499] bind     /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
UDEV  [516.533041] remove   /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001/block/mmcblk0 (block)
KERNEL[516.561253] unbind   /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
KERNEL[516.561384] remove   /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
UDEV  [516.562115] unbind   /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
UDEV  [516.562444] remove   /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
KERNEL[517.032043] add      /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
KERNEL[517.032995] add      /devices/virtual/bdi/179:0 (bdi)
UDEV  [517.033539] add      /devices/pci0000:00/0000:00:1c.0/0000:71:00.0/rtsx_pci_sdmmc.0/mmc_host/mmc0/mmc0:0001 (mmc)
UDEV  [517.033949] add      /devices/virtual/bdi/179:0 (bdi)

The following console messages are output during shutdown of Archiso.

[FAILED] Failed unmounting /run/archiso/bootmnt.
[  OK  ] Unmounted /run/archiso/cowspace.
[  OK  ] Unmounted /run/credentials/systemd-sysusers.service.
[  OK  ] Unmounted Temporary Directory /tmp.
[  OK  ] Stopped target Preparation for Local File Systems.
[  OK  ] Stopped target Swaps.
[  OK  ] Reached target Unmount All Filesystems.
         Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[  OK  ] Stopped Create Static Device Modes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[      ] A stop job is running for Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling (1min 1s / 1min 31s)
[  OK  ] Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
[  OK  ] Finished System Reboot.
[  OK  ] Reached target System Reboot.
[ 1124.991197] watchdog: watchdog0: watchdog did not stop!
[ 1125.091460] watchdog: watchdog0: watchdog did not stop!
[ 1125.103342] sd-umoun[1454]: Failed to unmount /oldroot: Device or resource busy
[ 1125.104588] sd-umoun[1455]: Failed to unmount /oldroot/run/archiso/bootmnt: Device or resource busy
[ 1125.106059] sd-umoun[1456]: Failed to unmount /oldroot/run: Device or resource busy
[ 1125.107437] shutdown[1]: Could not detach loopback /dev/loop0: Device or resource busy
[ 1125.109069] shutdown[1]: Failed to finalize file systems, loop devices, ignoring.

nix-a-churl · 2022-02-07 10:28:54

So, I'm inclined to think this might be a driver bug, or I'm being stupid and missing something. Any advice either on what to do about a driver bug or how to go about investigating that, or pointing out something I've missed, would be very much appreciated

Cheers!

qinohe · 2022-02-07 16:42:40

Looking ar:

kernel: I/O error, dev mmcblk0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
kernel: Buffer I/O error on dev mmcblk0, logical block 0, async page read
systemd-udevd[346]: mmc0:0001: Worker [375] processing SEQNUM=3825 is taking a long time
systemd-udevd[346]: mmc0:0001: Worker [375] processing SEQNUM=3825 killed
systemd-udevd[346]: mmc0:0001: Worker [375] failed
kernel:  mmcblk0: unable to read partition table

I'd say there is something wrong with that SD-card.
Have you tried to re-partition the card? Have you tried another one?
Have you tried a USB memory stick to see if you can successfully finish your installation?
This is something I never done before myself, so I could be wrong!

nix-a-churl · 2022-02-07 20:38:06

Thanks for this!

qinohe wrote:

I'd say there is something wrong with that SD-card.
Have you tried to re-partition the card? Have you tried another one?

I've tried three different SD cards and the result is the same. Interestingly, these SD cards work fine when plugged in only after boot, or when plugged in booting Windows (I need windows for some work things ).

qinohe wrote:

Have you tried a USB memory stick to see if you can successfully finish your installation?

Copying from the SD card to a USB memory stick allows me to boot my installation nicely! I did that earlier today as a compromise In the long run I still want to be able to use the SD card slot to boot as it's easier to carry a card in my wallet and always have it with me, but until it's fixed the USB is a good compromise.

qinohe wrote:

This is something I never done before myself, so I could be wrong!

It's still helpful! The way the problem occurs with multiple cards and also irrespective of using my installation or Archiso suggests to me a driver bug or something like that, but I haven't the faintest idea how to begin investigating that or what to do about anything I might find!

progandy · 2022-02-07 21:21:52

nix-a-churl wrote:

Copying from the SD card to a USB memory stick allows me to boot my installation nicely! I did that earlier today as a compromise In the long run I still want to be able to use the SD card slot to boot as it's easier to carry a card in my wallet and always have it with me, but until it's fixed the USB is a good compromise.

This is not really a solution for the sdcard problem, but some years ago I was able to find a 16GB usb2 drive that is basically only a single circuit board thick. My model is no longer sold, but maybe the clevx foldit is still available or with keywords like "usb udp flash".

Last edited by progandy (2022-02-07 21:24:05)

nix-a-churl · 2022-02-08 14:27:52

progandy wrote:

This is not really a solution for the sdcard problem, but some years ago I was able to find a 16GB usb2 drive that is basically only a single circuit board thick. My model is no longer sold, but maybe the clevx foldit is still available or with keywords like "usb udp flash".

Oh yeah I used to have one like that! They're pretty cool, but mine didn't seem to last very long. As for this machine, it only has usb c ports

Arch Linux

#1 2022-02-06 17:04:10