[SOLVED] Cannot boot because nvme device cannot be found

bjcubsfan · 2019-12-30 16:23:33

After completing to the reboot step in the installation guide my system goes to rootfs and displays this message:

:: running early hook [udev]
Starting version 244.1-1-arch
:: running hook [udev]
:: Triggering uevents...
Waiting 10 seconds for device /dev/disk/by-partnuid/a158dd01-636f-4dcb-b6fc-118427be3bd8 ...
ERROR: device 'PARTUUID=a158dd01-636f-4dcb-b6fc-118427be3bd8' not found. Skipping fsck.
:: mounting 'PARIUUID=a158dd01-636f-4dcb-b6fc-118427be3bd8' on real root
mount: /new_root: can't find PARTUUID=a158dd01-636f-4dcb-b6fc-118427be3bd8.
You are non being dropped into an emergency shell.
sh: can't access tty: Job control turned off
[rootfs ]#

It seems that although it was entirely usable when I was installing, the NVME SSD is not visible to the system during boot after completing installation. What is the key difference in this instance between the install media and the system that I have set up? It's also odd that the EFI which I used as /boot is on the SSD and it can read that.

At the "rootfs" prompt I executed a "blkid", but it only showed the 4 HDDs ("sd*" devices) and not the nvme.

In the system configuration of my Dell Precision 7920, I have tried AHCI and OFF for the "Sata opertion" settings and neither work to show the NVME SSD. It doesn't seem like something in the settings can be the issue since the drive is visible when I boot off of the installation media.

I added "i915", "nvme", and "nvme_core" to modules in "/etc/mkinitcpio.conf", but that did not make the drive visible.

What else should I do? How can I troubleshoot this?

Last edited by bjcubsfan (2020-01-03 16:18:54)

V1del · 2019-12-30 16:57:38

What exact device is this? See e.g. https://bbs.archlinux.org/viewtopic.php … 6#p1876266 which afaik still hasn't had the fix as part of a stable release, though it should be in the next bump.

bjcubsfan · 2019-12-30 17:38:15

Here is what it says in "fdisk -l":

Disk /dev/nvme0n1: 953.89 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: SSDPEKKF010T8 NVMe INTEL 1024GB

I see HP mentioned there, but it sounds like it's a particular controller chip issue. Is there a way to tell if my device has that chip?

I did try booting using fallback and it did not work either.

bjcubsfan · 2019-12-30 19:42:15

I installed the 5.3.1 kernel and booted. I still get the same error. I think it's therefore not the problem in that other post.

loqs · 2019-12-30 20:08:06

Is the issue still present using the fallback initrd? If so can you dump the output of dmesg produced from emergency shell to one of the hard-disks then post its output?

bjcubsfan · 2019-12-30 20:30:17

The issue is still present with the fallback initrd.

Here is the dmesg produced from the the emergency shell. Do you see any hints there?

loqs · 2019-12-30 21:18:35

No hints from that output. Please post the dmesg from live media where the NVME device was working for comparison.

CarbonChauvinist · 2019-12-30 21:21:50

bjcubsfan wrote:

... it sounds like it's a particular controller chip issue. Is there a way to tell if my device has that chip?

Looks like you can run `nvme list` to get something similar to the following output:

Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     HBSE18454500029      HP SSD EX900 120GB                       1         120,03  GB / 120,03  GB    512   B +  0 B   R0802B0

Then you can also run `lscpi -vvv` and focus on the line for the nvme controller as shown below in example:

...
04:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
	Subsystem: Silicon Motion, Inc. Device 2263
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
....

The Subsytem: Silicon Motion, Inc. Device 2263 (rev 03) appears to the pertinent bit - if that's your same controller you most likely are experiencing the same issue.

Lastly, you can run `nvme id-ns /dev/nvme0 -n 1` in order to confirm for sure, which will produce output like so:

NVME Identify Namespace 1:
nsze    : 0xdf94bb0
ncap    : 0xdf94bb0
nuse    : 0xdf94bb0
...
anagrpid: 0
endgid  : 0
nguid   : 00000000000000000000000000000000
eui64   : 0100000000000000
lbaf  0 : ms:0   lbads:9  rp:0 (in use)

With the pertinent bits being:

nguid   : 00000000000000000000000000000000
eui64   : 0100000000000000

Which apparently

are broken identifications! You will get by with a device like this if you've just one in the system, but we'll observe other problems if more than one is present.

All example outputs and quotes were gleaned from following bugzilla report, which V1del linked to earlier in his post. And per the same bug report it appears the patch/fix is slated for the next tagged 5.4 branch release.

And as predicted, the fix was applied to 5.5-rc2, and 5.4 stable-queue automatically picked it up: https://git.kernel.org/pub/scm/linux/ke … cec31be7b5
The 5.4 branch will rebase with those patches in its queue for the next tagged release.

Edit: --Ninjaed by loqs. But, also though it may not be relevant at all I'm curious, why are you passing this kernel parameter?

add_efi_memmap

Any changes without it?

Last edited by CarbonChauvinist (2019-12-30 21:24:13)

bjcubsfan · 2019-12-30 22:17:03

Here is the dmesg output when I boot with the installation media. I see the working nvme, but not sure what makes it appear there and not in the normal boot.

nvme list gives me:

Node             SN                   Model                                    Namespace Usage
              Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     BTHH85010SHQ1P0E     SSDPEKKF010T8 NVMe INTEL 1024GB          1           1.02  TB /
  1.02  TB    512   B +  0 B   D06N

`lcpci -vvv`shows:

10000:01:00.0 Non-Volatile memory controller: Intel Corporation SSD Pro 7600p/760p/E 6100p Series (rev 03) (prog-if 02 [NVM Express])
        Subsystem: Intel Corporation SSD Pro 7600p/760p/E 6100p Series
        Physical Slot: 87
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

I think that means it's not that bug, but thanks for the troubleshooting tips that allowed me to confirm that.

bjcubsfan · 2019-12-30 22:46:01

I did not intentionally have the "add_efi_memmap" kernel parameter. It must have been in the example rEFInd configuration that I modified. I took it out and there was no change.

loqs · 2019-12-31 00:04:41

Please post the contents of /etc/mkinitcpio.conf used to build the initrd.

bjcubsfan · 2019-12-31 15:13:58

Below is the mkinitcpio.conf. The only change I made was to add some modules at the top that I read solved nvme issue for someone. I could probably remove them now, since they didn't work for me.

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=(i915 nvme_core)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

loqs · 2019-12-31 19:46:00

Both the live media and the installed system are currently using 5.3.1-arch1-1-ARCH. What if you copy the initrd from the live media to the installed system and try using that?

bjcubsfan · 2019-12-31 21:44:16

I copied over the initrd from the live media, but it still didn't work. I am truly baffled.

CarbonChauvinist · 2019-12-31 22:24:43

Any difference if you switch to systemd init instead of busybox?

Also, I know we've verified that your drive is not the previously identified controller. But I still think it's useful to confirm what the `nguid` and `eui64` values are for your drive. Can you run `nvme id-ns /dev/nvme0 -n 1` i and confirm their values please?

bjcubsfan · 2019-12-31 23:14:00

I replaced "udev" with "systemd" in my /etc/mkinitcpio.conf. It looks different booting since it's using systemd now, but it still times out after not finding the root device.

Here is the output of "nvme id-ns /dev/nvme0 -n 1":

NVME Identify Namespace 1:
nsze    : 0x773bd2b0
ncap    : 0x773bd2b0
nuse    : 0x773bd2b0
nsfeat  : 0
nlbaf   : 0
flbas   : 0
mc      : 0
dpc     : 0
dps     : 0
nmic    : 0
rescap  : 0
fpi     : 0x80
dlfeat  : 1
nawun   : 0
nawupf  : 0
nacwu   : 0
nabsn   : 0
nabo    : 0
nabspf  : 0
noiob   : 0
nvmcap  : 0
nsattr  : 0
nvmsetid: 0
anagrpid: 0
endgid  : 0
nguid   : 00000000000000000000000000000000
eui64   : 5cd2e42291b056af
lbaf  0 : ms:0   lbads:9  rp:0 (in use)

bjcubsfan · 2020-01-03 16:18:19

I finally solved this!

For me the working combination is:

- Change SATA mode to AHCI in system configuration of my Dell Precision 7920 (BIOS system settings)
- Enable Intel VMD in the system configuration
- add "vmd" module to /etc/mkinitcpio.conf
- add nvme_load=YES

I did change all of those things at once, so one or more may not be necessary, but maybe this solution will help someone else.

jypma · 2021-05-21 08:52:52

(sorry for bouncing, but I encountered the same issue)

I'm trying to get an Intel SSD 660p to work in a Thingpad P15v, with zfs, and encountered the same issue. A previous NVME SSD was working fine.

For me, adding the "vmd" module made the difference.

Curiously, when booting the system with both SSDs, the "vmd" module wasn't loaded and the Intel SSD worked nonetheless.
Could it make sense to add the "vmd" module by default?

Arch Linux

#1 2019-12-30 16:23:33

[SOLVED] Cannot boot because nvme device cannot be found

#2 2019-12-30 16:57:38

Re: [SOLVED] Cannot boot because nvme device cannot be found

#3 2019-12-30 17:38:15

Re: [SOLVED] Cannot boot because nvme device cannot be found

#4 2019-12-30 19:42:15

Re: [SOLVED] Cannot boot because nvme device cannot be found

#5 2019-12-30 20:08:06

Re: [SOLVED] Cannot boot because nvme device cannot be found

#6 2019-12-30 20:30:17

Re: [SOLVED] Cannot boot because nvme device cannot be found

#7 2019-12-30 21:18:35

Re: [SOLVED] Cannot boot because nvme device cannot be found

#8 2019-12-30 21:21:50

Re: [SOLVED] Cannot boot because nvme device cannot be found

#9 2019-12-30 22:17:03

Re: [SOLVED] Cannot boot because nvme device cannot be found

#10 2019-12-30 22:46:01

Re: [SOLVED] Cannot boot because nvme device cannot be found

#11 2019-12-31 00:04:41

Re: [SOLVED] Cannot boot because nvme device cannot be found

#12 2019-12-31 15:13:58

Re: [SOLVED] Cannot boot because nvme device cannot be found

#13 2019-12-31 19:46:00

Re: [SOLVED] Cannot boot because nvme device cannot be found

#14 2019-12-31 21:44:16

Re: [SOLVED] Cannot boot because nvme device cannot be found

#15 2019-12-31 22:24:43

Re: [SOLVED] Cannot boot because nvme device cannot be found

#16 2019-12-31 23:14:00

Re: [SOLVED] Cannot boot because nvme device cannot be found

#17 2020-01-03 16:18:19

Re: [SOLVED] Cannot boot because nvme device cannot be found

#18 2021-05-21 08:52:52

Re: [SOLVED] Cannot boot because nvme device cannot be found

Board footer