Arch Linux

dragomang87

Member

Registered: 2014-09-19

Posts: 8

[SOLVED] possible missing firmware message and how to make sure

So I read the wiki page and
https://bbs.archlinux.org/viewtopic.php?id=279289

I understand what goes on, but just to start on the same page, let me recap.
`mkinitcpio` fallback includes everything, but there are two levels, including the modules/drivers and for each driver including its (possibly multiple) firmware.
`autodetect` is only module autodetect, it filters and keeps only the needed modules for the machine but still tries to include all firmware for that module (correct?).
The usual question is "Why is the warning still there and why is it ok to ignore?" and essentially the answer is "it depends, fallback or autodetect?":
- fallback: it is expected, and unless you are willing to include all firmware ever available the warning means "if you want to use this fallback on a machine that is not yours, in rare cases it might miss functionality"
- autodetect: this is where it gets annoying, the answer is essentially "well, you can still ignore it, if you were missing firmware you would know already because your system would be broken, also, if your machine is mainstream the firmware will be there", so the answer to "but I still want to suppress a warning" is "install the firmware the you might not need"

However, I don't agree that you would always notice if you were actually missing firmware. You might just have reduced functionality that you were not using or things running slower or hotter than they need to be.
Or you might have ignored/not seen some news or announcement that suddenly made your machine need explicitly installed firmware and suddenly the same warning that was to be ignored before now becomes a true warning.
So, while the missing firmware message in autodetect is not a problem, it does leave me with the question "am I sure that I didn't miss some firmware?".

I am missing a tool or guide to answer this question, or is the only answer
- "search the information for every entry that lspci and lsusb give you"?
Because the kernel/system already does the job of looking for the needed firmware, so somewhere it is done already.
- Does an actually missing firmware show as error in journalctl/dmesg?
- Does an optional actually missing firmware show anywhere?
- Is there a tool that can tell you if a firmware matches any of your hardware?
If you search firmware on the wiki you get directly the page for fwupd, which is not the kind of firmware we are talking about.
Also,
- Do missing modules/firmware in the initramfs become completely irrelevant the moment the kernel manages to mount a root where they are available? even if root is mounted manually in rescue shell ?(maybe exiting or manually resuming the boot process is needed?)

PS: thanks to anyone taking the time to read this
PPS: love arch and I have already done my minimum share of rescuing my system from different disks or roots and updating the initramfs using arch-chroot (honestly, one of the best arrows in the quiver)

Last edited by dragomang87 (Today 18:22:52)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,209

Re: [SOLVED] possible missing firmware message and how to make sure

If you really get missing firmware in autodetect then that could indeed be a problem. Instead of dealing in theory can you provide a real world example of your own system showing which firmware it might lack? (on my system, there's nothing missing on the autodetect)

1. basically that, cross reference the vid:pids with the firmwares that modinfo $module gives you. which is basically the only common thing you can somewhat rely on

2. Depends most drivers will list something when they fail to load a firmware, some might not

3. You can check how mkinitcpio tries to do it https://github.com/archlinux/mkinitcpio … tions#L670 -- basically check whether any of the firmwares matches the globs you extract via modinfo, if some are missing then it prints the warning

4. Depends some drivers will lookup certain firmware versions and fall back to different versions depending on whether they've been found -- e.g. iwlwifi is a good example of this, it often gets firmware support in the kernel module before said firmware is publicly released and leaves a trace of which firmwares it failed to load before eventually setting on something that's available.

5. Not if the module in question is already explictly loaded and part of the initramfs as it will just use things from there, e.g. if you "follow" the general reccomendation of having early KMS enabled in order to get a more seamless GPU transition, then those drivers won't reload the firmware from the root filesystem since you already loaded them as part of the initramfs

The only globally correct answer to all of your questions would be to read the kernel code for every driver that's relevant to your system, since the relevant module codes that actually load the firmware are the things that make the decision. modprobe/modinfo just provides matches based on vids:pids so the kernel knows when it should load a certain module, how said module then handles the firmware is up to the module.

Generally speaking, with a default autodetect setup, only the most necessary baseline will get included (e.g. block device drivers so you can actually mount a root fs) the vast majority of stuff will get loaded later anyway.

That firmware gets removed from the common linux-firmware repo is very unlikely, the only case I've ever knowingly seen that was that Renesas USB stopped working without an explicit firmware included and that was indeed a PITA for affected people, needing to ensure to grab an AUR package to get USB working. That was eventually worked around by still relying on on HW firmware in the kernel.

Last edited by V1del (2024-11-14 15:47:58)

dragomang87

Member

Registered: 2014-09-19

Posts: 8

Re: [SOLVED] possible missing firmware message and how to make sure

ok, yes, you are right, better start from the beginning, right now I am coming from trying to understand my USB speed

One of my backup external drives failed, so it was time to buy new ones and I decided to buy some future proof speed
- fanxiang 1TB External SSD https://www.amazon.de/dp/B0D2KW5GJ5 with nominal speed 3700MB/s and actually reported 3788MB/s
- Kingstons XS2000 https://www.amazon.de/dp/B09F6279PY with nominal speed up to 2000MB/s and reported ~1000MB/s
- old ssd still working: ADATA SE730H with nominal 500MB/s and a 256GB M.2 disk from my old machine that I made into an external drive (the one that died was my final leftover hdd)
everything external is now exFAT

This new speed is way above luxury for me already and of course you can never fully expect nominal speed,
but I have a T480s, so they should easily max out the three 5Gb/s~600MB/s usb 3.1 gen 1 ports and the 10Gb/s 3.1 Gen 2/ Thunderbolt 3 port,
at least when I backup gallery stuff and especially backing up large multiple GB videos.

The fanxiang gave me 300MB/s, so I am satisfied enough, but the Kingstons dropped to essentially usb 2.0 40MB/s after the buffer was filled even though the speed was negotiated correctly:
[32225.402908] usb 2-1: new SuperSpeed USB device number 7 using xhci_hcd
[32225.416471] usb 2-1: New USB device found, idVendor=0951, idProduct=176b, bcdDevice= 1.00
[32225.416490] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[32225.416498] usb 2-1: Product: XS2000
[32225.416503] usb 2-1: Manufacturer: Kingston
[32225.416509] usb 2-1: SerialNumber: 50026B72831CCFBA

The kingstons feels cheap as ****, but since it has a decent reported speed, before complaining I wanted to make sure the problem is not my setup, so then I noticed the xhci_hcd firmware warning, googled and stumbled on posts about missing Renesas upd72020x-fw, installed, rebooted, didn't work, removed, rebooted, didn't work, kept wondering about whether I am missing firmware, ended up looking into this post topic and got annoyed of seeing the "just ignore it" answer everywhere without any way of checking if I actually have a missing firmware.
Maybe I just got a faulty one, but not being able to make sure that the problem is not my machine kind of bothered me.
Either way, your answer to the firmware problem was quite clarifying, so thanks for that (maybe it can be linked in the wiki?).

edit: for those stumbling here, so expected speeds are more like ~470MB/s and 1.1GB/s from dd and I indeed get ~470MB/s and 1.0GB/s for *reading* bs=1M via dd (and quite close with bs=4K), 5Gbps and 10Gbps are before error-correction overhead (8/10 and 128/132 respectively?) and it seems that maybe you have to account that dd uses powers of 1024 while USB specs maybe use powers of 1000 (not sure, I have not managed to confirm this online), writing is where it is unreasonably slow

Last edited by dragomang87 (Today 18:43:29)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,209

Re: [SOLVED] possible missing firmware message and how to make sure

So the actual issue that made you wonder was indeed such a renesas case, did you rebuild the initramfs after installing the firmware? Because this would be a situation where the last point would apply. So if you install the firmware but xhci_hcd being one of the cases that's almost certain to land in the initramfs, that new firmware would only get loaded when generating the image after installing it.

But there are quite a few other factors at play in such scenarios, so check the cable you're using, check whether the same USB bus might get lower perf because too many devices are connected to the same bus, potentially stealing bandwith

dragomang87

Member

Registered: 2014-09-19

Posts: 8

Re: [SOLVED] possible missing firmware message and how to make sure

yeah, I use aura so rebuild was automatic, same cable as fanxiang, 10Gb/s port, only device connected, sometimes got ~110MB/s, but still ...

weirdly, `| grep -i ren` outputs nothing from neither lspci, lsusb nor dmesg, so I thought I did not have Renesas anyway and I just missed some other firmware, I'll keep testing and just send it back once I get tired

maybe I will indeed install all firmware and try in fallback, let's see

Last edited by dragomang87 (2024-11-14 18:34:00)

dragomang87

Member

Registered: 2014-09-19

Posts: 8

Re: [SOLVED] possible missing firmware message and how to make sure

marked solved, the speed problems I'll leave for another topic