Arch Linux

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

Here is the boot from 5.4 https://gist.github.com/Alex-BGoode/af4 … eb6a0f4dea

Since I am not able to boot with 5.10 I am not sure how to get the corresponding logs. I get thrown into a emergency shell while booting:

ERROR: resume: hibernation device '/dev/mapper/arch-swap' not found
ERROR: device '/dev/mapper/arch-root' not found. Skipping fsck.
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ]# _

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

From the emergency shell can you execute the following and post if there are any matches other than for the command line

dmesg | grep -i nvme

Edit:
Also check in /dev if the nvme device nodes are present.

Last edited by loqs (2021-02-20 01:11:54)

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

I would have to type everything from the photo I took (and I am happy if that helps), but maybe the crucial line is

    nvme nvme0: Removing after probe failure status: -19

Edit:
The device nodes in /dev (i.e. nvme0n1p1 etc) are not present

Edit2:

Output of dmesg | grep -i nvme

Command line: BOOT_IMAGE=/vmlinuz-linux-lts root=/dev/mapper/arch-root rw cryptdevice=/dev/nvme0n1p3:arch-encrypted resume=/dev/mapper/arch-swap i915.enable_dc=0 i915.enable_fbc=0 i915.enable_psr=0 audit=0 loglevel=3 quiet

Kernel command line: BOOT_IMAGE=/vmlinuz-linux-lts root=/dev/mapper/arch-root rw cryptdevice=/dev/nvme0n1p3:arch-encrypted resume=/dev/mapper/arch-swap i915.enable_dc=0 i915.enable_fbc=0 i915.enable_psr=0 audit=0 loglevel=3 quiet

nvme nvme0: pci function 0000:04:00.0

nvme nvme0: Removing after probe failure status: -19

cryptdevice=/dev/nvme0n1p3:arch-encrypted

Last edited by alex_bgoode (2021-02-20 01:25:15)

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

If you leave the linux-lts package on 5.4 so you have a working kernel than try 5.5 to 5.10 of the linux package from the ALA,  which series introduced the issue?
If they are all bad try 5.4 of the linux package.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

Should i install the latest version for every kernel. For example linux-5.5.13.arch2-1-x86_64.pkg.tar.zst  for 5.5?

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

The last version should have the most fixes so that seems a good choice.  If you start in the middle with 5.7 then you can reduce the number of kernels to check.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

I tried linux-5.7.12.arch1-1 which did not work.

Next was 5.6.15-arch1-1which is working.

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

What about 5.7.7 if that works try 5.7.8.
Edit:
Also what is the NVME device?  Does the system use a Silicon Motion controller?

Last edited by loqs (2021-02-20 02:05:59)

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

5.7.7 and 5.7.8 do not work.

The device is a WDC PC SN720 SDAQNTW-512G-1001 which leads to this: https://www.westerndigital.com/products … -sn720-ssd
I don't know about a Silicon Motion controller. Can I find out somehow?

Edit:

I tested all version going down from 5.7.8 and no 5.7 version is working. So 5.6.15 seems to be the last working version.

Last edited by alex_bgoode (2021-02-20 10:37:55)

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

Would you be willing to bisect between 5.6 and 5.7 to find the causal commit?  I can walk you through the process.

I believe WDC use their own in house custom controller.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

I can do that. I would need some help though. I have never done anything with a linux kernel other than pacman -S linux

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

5.6 no longer builds with the current gcc so it took some time getting the package building again.  I also disabled building kernel documentation to reduce build time.
You need to install gcc9 and gcc9-libs from the ALA.

pacman -U https://archive.archlinux.org/packages/g/gcc9/gcc9-9.3.0-4-x86_64.pkg.tar.zst https://archive.archlinux.org/packages/g/gcc9-libs/gcc9-libs-9.3.0-4-x86_64.pkg.tar.zst

To build the package you will also need base-devel installed and to reduce build time I recomend enabling parallel compilation

git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/linux"
cd packages/trunk
git checkout d0b96b4828f482cc02a57ab05b14c3d3d00b0782
curl -o PKGBUILD.diff https://gist.githubusercontent.com/loqs/e058b4e7914217ea61df2de57b5ef10a/raw/335316494accae7a79f1f2731f4763a090759e0e/PKGBUILD.diff #download patch for PKGBUILD
git apply -v PKGBUILD.diff
gpg --search-keys ABAF11C65A2970B130ABE3C479BE3E4300411886 #obtain Linus's key used to verify git tag
makepkg -rsi #build and install package

See if the locally built 5.6 works.  Will cover how to bisect after confirming the PKGBUILD works.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

Thank you for the instructions and sorry for taking so long!

I was now able to build the 5.6 kernel as described (had to add a newline to the .diff file).
The kernel boots (although WiFi is not working, probably unrelated?) I am ready to do follow up instructions.

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

cd packages/trunk
curl -o PKGBUILD.diff https://gist.githubusercontent.com/loqs/cddaaf203c358a34ddf2ed9781149234/raw/f0d0d23d0717b01a06f6f760c6add0fd7e65d1ac/PKGBUILD.diff #download patch adding pkgver function and setting pkgver to 5.7
git apply -v PKGBUILD.diff
cd src/linux
rm .scmversion
cd ../..
makepkg -rsi #build and install package

Check the locally built 5.7 is bad.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

5.7 does not boot

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

cd packages/trunk/src/linux
git bisect start
git bisect bad v5.7
git bisect good v5.6
cd ../..
makepkg -rsif

Boot and test new kernel version.

cd packages/trunk/src/linux
git bisect $result #where result is good or bad
cd ../..
makepkg -rsif

Boot and test new kernel,  repeating this last code block until git has found the cause.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

After

cd packages/trunk/src/linux
git bisect start
git bisect bad v5.7
git bisect good v5.6

I get the output

6499c53216d11948deb2e39ccc9ca9824b8f38ab was both good and bad

Is that expected behavior?

Last edited by alex_bgoode (2021-02-27 16:52:41)

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

No,  downgrade the git package

# pacman -U /var/cache/pacman/pkg/git-2.29.2-1-x86_64.pkg.tar.zst # assumes the old version is in the package cache

Then try the following again

git bisect start
git bisect bad v5.7
git bisect good v5.6

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

Output changed to

Bisecting: 7542 revisions left to test after this (roughly 13 steps)
[50a5de895dbe5df947b3a695777db5b2c313e065] Merge tag 'for-linus-hmm' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

That is the expected output.  Please continue from the ../.. in post #41.

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,301

Re: nvme not detected

Has anyone with this issue tried dracut? Someone on IRC is having this issue, but can see the drive from a recent Arch install ISO, making me think the issue is with the initramfs.

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

https://bbs.archlinux.org/viewtopic.php?id=263567 might be another case but I put that down to difference in kernel versions.  Did the user on IRC say if they had the NVME device in AHCI or RAID mode?

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,301

Re: nvme not detected

I didn't think to check that, as I wouldn't expect it to show up on the ISO if it was in RAID mode.

alex_bgoode

Member

Registered: 2021-02-20

Posts: 14

Re: nvme not detected

The process is extremely slow right now. I expect I wasn't able to correctly configure parallel compilation. I typed

MAKEFLAGS="-j$(nproc)"

in the console before

makepkg -rsif

Is there something I have missed?

Last edited by alex_bgoode (2021-02-27 21:31:28)

loqs

Member

Registered: 2014-03-06

Posts: 18,208

Re: nvme not detected

Check the output of `top` there should be one instance of cc per CPU core.