Arch Linux

TieMajor

Member

Registered: 2022-02-04

Posts: 28

USB blocking at boot

Hello, It seems that one of my USB port on my motherboard died. I get errors related to the same bus and port at boot and it slows down the boot quite a lot of time (1m30s) trying to deal with it before giving up and continuing.
I have tried every hardware troubleshooting steps including disconnecting all USB devices leaving only the GPU connected on the PCI ports, physically disconnected the USB connection to the motherboard from the case but the issue persist.

dmesg excerpt of the errors:

[    7.237851] usb 1-9: device descriptor read/64, error -110
[   23.108851] usb 1-9: device descriptor read/64, error -110
[   23.594788] usb 1-9: new high-speed USB device number 5 using xhci_hcd
[   29.253847] usb 1-9: device descriptor read/64, error -110
[   45.125825] usb 1-9: device descriptor read/64, error -110
[   45.234166] usb usb1-port9: attempt power cycle
[   45.674795] usb 1-9: new high-speed USB device number 6 using xhci_hcd
[   50.475387] usb 1-9: Device not responding to setup address.
[   55.480372] usb 1-9: Device not responding to setup address.
[   55.687810] usb 1-9: device not accepting address 6, error -71
[   56.066824] usb 1-9: new high-speed USB device number 7 using xhci_hcd
[   60.868364] usb 1-9: Device not responding to setup address.
[   65.880404] usb 1-9: Device not responding to setup address.
[   66.087895] usb 1-9: device not accepting address 7, error -71
[   66.090890] usb usb1-port9: unable to enumerate USB device

I have come to accept the fact that somehow one USB port on my 350 USD motherboard is dead however I would really like to just disable the port at boot especially to not have the boot times affected as much.
While searching I have not found any way to easily disable a specific USB port at boot on linux. I was expecting a kernel parameter or something that could be added as a boot parameter.

I have tried with udev rules but it seems those kick in later in the boot process.

ACTION=="add", SUBSYSTEMS=="usb", ATTRS{busnum}=="1", ATTRS{devpath}=="9", RUN+="/bin/sh -c 'echo 0 > /sys$DEVPATH/authorized'

I have tried using usbblock but it also seems this kicks in later in the boot process since the daemon wants udev.

block via-port "1-9"

Is there a way to do it or some other way that I didn't think of to reduce the impact on boot times ?

EDIT:
Booting with usbcore.nousb "solves" the issue. Boot times are "normal" again but of course no other USB devices are working.
Booting with usbcore.old_scheme_first=1 has no effect.

Last edited by TieMajor (2025-03-02 09:04:59)

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Which kernel are you using and which version is this on (post the output of "uname -r")?

Also is this an issue that recently started or was this always like this?

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Sorry about that, I am on 6.13.5-arch1-1. This issue has been going on for a while but has been quite annoying recently since I needed to switch between my dual boot more often hence why I am only acting on it now.

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Is the issue still present when you switch to the "linux-lts" package?

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

I just tested this and yes the issue is still present when booting with linux-lts.

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Is it still present on the old linux lts that was based on the 6.6 series?

sudo pacman -U https://archive.archlinux.org/packages/l/linux-lts/linux-lts-6.6.72-1-x86_64.pkg.tar.zst

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Just tested with your package and yes the issue still occurs.
However it seems that with this version systemd seems to get ahead a bit more and leads to a wait job on journald while the USB delays the boot.
Otherwise looking at the kernel debug logs the issues with the USB port are still there (always the same hub and port), the boot is still delayed.

Again this issue has been here for a while, unfortunately I cannot give an exact date but more than 3 months at least.

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Let's just check the 6.1 series aswell, if it's not fixed in there I think we declare it as "always has been this way" x)

sudo pacman -U https://archive.archlinux.org/packages/l/linux-lts/linux-lts-6.1.71-1-x86_64.pkg.tar.zst

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Okay, some progress here. This kernel seems to not have the blocking issue. The USB port (always same hub and port) still fails to initialize correctly however it's not blocking the boot process anymore. Huge speedup in the boot process.

Checkout the full dmesg logs https://pastebin.com/ChJQiLCg. It might not be obvious but I am getting into the greeter at around the 9 seconds mark now.

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

I am not sure what the next steps are here. Do I try to bisect this and report it as a kernel bug or could it be because of arch for some reason ?

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Do you consider 6.1 'good'? It sounds a bit like "still broken, but a little less".

Generally, are you confident to do the bisection on your own or do you need some help?
If you want we could also provide you with pre-built kernel images for you to test (which greatly speeds up the test time)  

Good info to get you started is:
- https://docs.kernel.org/admin-guide/rep … sions.html
- https://wiki.archlinux.org/title/Kernel … egressions

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Yes I would consider 6.1 good. I am just not sure if this means the issue is purely related to the kernel or something added in the arch flavor of the kernel.
I should be able to do a bisection but waiting 1m30s every time is going to be brutal .
I would greatly appreciate pre-built images but do you have them for all commits ? I would be running a bisect on the repo getting the hash and installing the pre-built kernel with the same hash ?

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Please try the following kernel image:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.3.r5640.gb68ee1c-1-x86_64.pkg.tar.zst

Also please test the latest mainline so that we know that the bug has not already been fixed in the upstream kernel:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.14rc5-1-x86_64.pkg.tar.zst

(note that this installs the kernel as linux-mainline, so you need to configure your bootloader to boot it (for example via grub-mkconfig -o ... or by writing the systemd-boot loader entry))

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Thanks for your help.

I tried both kernel the current mainline (6.14rc5-1) still has the issue.
The 6.3.r5640 straight up kernel panics very early in the boot process, the USB stack doesn't even seem initialized since even after being dropped into a console the keyboard is not responsive. I can only share a picture from my phone of the error if you want and it's relevant.
I tried booting to it twice, same result.

Last edited by TieMajor (2025-03-03 15:27:47)

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

So I went a ahead and tested https://pkgbuild.com//\~gromit/linux-bi … kg.tar.zst which does not present the issue at boot (and does not panic ).

One interesting thing I noticed was that I rebooted quite fast and the bad USB was blocking the reboot. However I would still say that's still good as the boot time is not impacted. Shutdown after only being booted for a few second is not a use case I am interested in x).

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Okey, so let's skip that as this error is most likely not related to our problem, please try the following instead:

sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.2.rc7.r157.g1fd09e2-1-x86_64.pkg.tar.zst

So I went a ahead and tested https://pkgbuild.com//\~gromit/linux-bi … kg.tar.zst which does not present the issue at boot (and does not panic)

Did you just choose that at random?

Last edited by gromit (2025-03-03 15:52:33)

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Did you just choose that at random?

I wouldn't say random, but more of me trying to remember when the issue actually started occurring based off the dates displayed.

I was just trying https://pkgbuild.com//\~gromit/linux-bi … kg.tar.zst which starts showing issues in the form of systemd wait job on journald.

I think there is no need to test 6.2 since it might be in between 6.7 and 6.10 ? I have maybe a false memory coming back to me that this started around a round number (6.10 maybe ?). Do you think giving a try to 6.9 next ?

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

We will be in the right range quickly, as bisection halves the range on each test. Is the revision from my previous comment good or bad? I don't quite understand your comment as you say that 6.2 should be not affected but it kinda sounds like it is?

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Ah sorry if I expressed myself badly,

I tested https://pkgbuild.com///~gromit/linux-bi … kg.tar.zst (v6.7) and it is good.
Then I tested https://pkgbuild.com///~gromit/linux-bi … kg.tar.zst (v6.10) which is bad.

So my understanding is that the culprit would be after 6.7 or before or included v6.10, right ?

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

Did you also test "https://pkgbuild.com/\~gromit/linux-bis … kg.tar.zst"?

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

I did not test https://pkgbuild.com//~gromit/linux-bis … kg.tar.zst I will now.

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

https://pkgbuild.com//~gromit/linux-bis … kg.tar.zst does not boot at all. I get a message "Waiting 10 seconds for device /dev/disk/by-uuid ..." and the get dropped in a console. However the USB stack does not initialize, USB keyboard is not responsive.

I think for some reason this is just a bad kernel ? It seems weird that v6.2 and v6.3 do not boot but v6.7 works fine ?

Last edited by TieMajor (2025-03-03 16:31:51)

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

I continued trying kernels and I land on https://pkgbuild.com///\~gromit/linux-b … kg.tar.zst being good.
Since I tested https://pkgbuild.com///~gromit/linux-bi … kg.tar.zst and it was bad I think we can say that this kernel is the culprit.

If I am not mistaken those are following each other (last available 6.9 first available 6.10).

I think we can safely discard 6.2 et 6.3 crashing, I don't know what causes them to react in such a way (crashing very early during the boot process) but I don't think it's related to the USB issue here.

What would be the next step here ? I assume bisect on a commit by commit basis until finding the actual culprit ? I assume just knowing it was introduced early in 6.10 is not enough for a bug report ?

Last edited by TieMajor (2025-03-03 17:18:54)

TieMajor

Member

Registered: 2022-02-04

Posts: 28

Re: USB blocking at boot

Okay so I switched to bisecting the kernel commit by commit and compiling every time and I managed to narrow down the bad commit.

The problem is that the commit seems unrelated to the actual problem of the USB stack. The first bad commit is 69a429905ceccad547e4a532b08f9d32c7f3422a (drm/panthor: Relax the constraints on the tiler chunk size).
It seems extremely weird to me that this would be the actual source of the problem. I even tested the commit directly behind it e3193f0fbd6d83510ff6879ac248f42a7c0fefe7 (drm/panthor: Make sure the tiler initial/max chunks are consistent) and it does not have the issue of delayed boot times.

I am puzzled by this and I just want to be sure I did not mess something up majorly here and if I should report this as is or not. Just as a final sanity check I am trying to revert this bad commit on the latest stable kernel to see if it fixes the issue or not.

EDIT:
Turns out the issue i still present after the revert of this commit. I also noticed the systemd messages where different waiting on mounting a drive instead of the wait job for journald I saw. I will continue to bisect ...

Last edited by TieMajor (2025-03-04 15:11:06)

gromit

Package Maintainer (PM)

From: Germany

Registered: 2024-02-10

Posts: 1,131

Website

Re: USB blocking at boot

How did you do the bisection?