Arch Linux

ESPNSTI

Member

Registered: 2016-06-21

Posts: 11

[SOLVED] ZFS on Root, zfs.ko kernel module missing

Hi,

Yesterday, I updated zfs-linux and linux from 4.8.7 to 4.8.8.
I didn't notice any errors, but there must have been some, because on reboot, the zfs.ko kernel module is now missing and "modprobe zfs" fails.

How do I recover from this?
Do I get a 4.8.8 zfs.ko (how/from where?) and somehow copy it into initramfs-linux.img?
Should I boot from usb, zpool import, then chroot, and then try to downgrade linux and install a downgraded zfs-linux?

Thanks,
     ESPNSTI

Last edited by ESPNSTI (2016-11-19 01:27:11)

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,791

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

What is the output of uname -a   ?  Verify that it is not still 4.8.7

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

ESPNSTI

Member

Registered: 2016-06-21

Posts: 11

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

I'm not at the PC at the moment, so I can't run uname -a, however when I was looking for the zfs.ko module, the usr/lib/modules/4.8.8-2-ARCH directory was present, so it's not on 4.8.7 anymore.

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,791

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

What I suspect is that everything has been upgraded to 4.8.8 -- including the kernel modules.  But, if the bootloader is still finding the old kernel, that old kernel will not find its (4.8.7) modules as they have been removed.  This can happen if the boot partition had not been mounted when the kernel update was processed.  This would cause the new kernel and intitrd to be in /boot without having actually updated the boot partition.  The easy check is to see what version of the kernel is running -- the one that was available to the boot loader.

Last edited by ewaller (2016-11-18 17:01:05)

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

ESPNSTI

Member

Registered: 2016-06-21

Posts: 11

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

I will check this when I get home.
Thanks.

mouseman

Member

From: Outta nowhere

Registered: 2014-04-04

Posts: 291

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

Ran into the same issue.

$ uname -a
Linux atlantis 4.8.8-2-ARCH #1 SMP PREEMPT Thu Nov 17 14:51:03 CET 2016 x86_64 GNU/Linux

$ sudo modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/4.8.8-2-ARCH

The last lines in my pacman.log:

[2016-11-18 19:59] [ALPM] upgraded spl-utils-linux-git (0.7.0_rc2_r2_gcbba714_4.8.7_1-1 -> 0.7.0_rc2_r2_gcbba714_4.8.8_1-1)
[2016-11-18 19:59] [ALPM] upgraded spl-linux-git (0.7.0_rc2_r2_gcbba714_4.8.7_1-1 -> 0.7.0_rc2_r2_gcbba714_4.8.8_1-1)
[2016-11-18 19:59] [ALPM-SCRIPTLET] >>> Updating SPL module dependencies
[2016-11-18 19:59] [ALPM-SCRIPTLET] depmod: WARNING: could not open /lib/modules/4.8.8-1-ARCH/modules.order: No such file or directory
[2016-11-18 19:59] [ALPM-SCRIPTLET] depmod: WARNING: could not open /lib/modules/4.8.8-1-ARCH/modules.builtin: No such file or directory
[2016-11-18 19:59] [ALPM] upgraded syncthing (0.14.10-1 -> 0.14.11-1)
[2016-11-18 19:59] [ALPM] upgraded threadweaver (5.27.0-1 -> 5.28.0-1)
[2016-11-18 19:59] [ALPM] upgraded vim-runtime (8.0.0055-1 -> 8.0.0086-1)
[2016-11-18 19:59] [ALPM] upgraded vim (8.0.0055-1 -> 8.0.0086-1)
[2016-11-18 19:59] [ALPM] upgraded zfs-utils-linux-git (0.7.0_rc2_r38_g0df15db_4.8.7_1-1 -> 0.7.0_rc2_r42_g7ca2505_4.8.8_1-1)
[2016-11-18 19:59] [ALPM-SCRIPTLET] >>> WARNING: The systemd unit files for archzfs have changed!
[2016-11-18 19:59] [ALPM-SCRIPTLET] >>>          You may need re-enable the zfs systemd service.
[2016-11-18 19:59] [ALPM-SCRIPTLET] >>>          See https://github.com/archzfs/archzfs/issues/72
[2016-11-18 19:59] [ALPM] upgraded zfs-linux-git (0.7.0_rc2_r38_g0df15db_4.8.7_1-1 -> 0.7.0_rc2_r42_g7ca2505_4.8.8_1-1)
[2016-11-18 19:59] [ALPM-SCRIPTLET] >>> Updating ZFS module dependencies
[2016-11-18 19:59] [ALPM-SCRIPTLET] depmod: WARNING: could not open /lib/modules/4.8.8-1-ARCH/modules.order: No such file or directory
[2016-11-18 19:59] [ALPM-SCRIPTLET] depmod: WARNING: could not open /lib/modules/4.8.8-1-ARCH/modules.builtin: No such file or directory
[2016-11-18 19:59] [ALPM] transaction completed
[2016-11-18 19:59] [ALPM] running '80-linux.hook'...
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Starting build: 4.8.8-2-ARCH
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Image generation successful
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Starting build: 4.8.8-2-ARCH
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2016-11-18 19:59] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2016-11-18 19:59] [ALPM-SCRIPTLET] ==> Image generation successful
[2016-11-18 19:59] [ALPM] running 'glib-compile-schemas.hook'...
[2016-11-18 19:59] [ALPM] running 'gtk-update-icon-cache.hook'...
[2016-11-18 19:59] [ALPM] running 'update-desktop-database.hook'...

Appreciate any help getting ZFS back up.

Thanks!

ESPNSTI

Member

Registered: 2016-06-21

Posts: 11

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

What is the output of uname -a   ?  Verify that it is not still 4.8.7

Finally had time to check it, it's 4.8.8.

Linux (none) 4.8.8-2-ARCH #1 SMP PREEMPT Thu Nov 17 14:51:03 CET 2016 x86_64 GNU/Linux

Last edited by ESPNSTI (2016-11-18 22:56:58)

ESPNSTI

Member

Registered: 2016-06-21

Posts: 11

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

Ok, I have this fixed, it turned out not to be quite as difficult as I had thought it would be.

I downloaded Release 2016.09.03 to match the kernel version of zfs-archiso-linux, and created a bootable usb from that.

After booting from usb, I enabled zfs:

dirmngr </dev/null
echo "[archzfs]" >> /etc/pacman.conf
echo "SigLevel = Required DatabaseOptional TrustedOnly" >> /etc/pacman.conf
echo "Server = http://demizerone.com/\$repo/\$arch" >> /etc/pacman.conf
pacman-key -r 0EE7A126
pacman-key --lsign-key 0EE7A126
pacman -Syy
pacman -S archzfs-archiso-linux

Then I followed most of the Emergency chroot repair with archzfs wiki:

modprobe zfs
zpool import -a -R /mnt
mount /dev/sdb2 /mnt/boot
arch-chroot /mnt /bin/bash
pacman -Qi linux
uname -r
depmod -a 4.8.8-2-ARCH

I did not run "mkinitcpio -p linux".

After that I looked at the Downgrading the kernel wiki:

pacman -U linux-4.8.7-1-x86_64.pkg.tar.xz zfs-linux-0.6.5.8_4.8.7_1-1-x86_64.pkg.tar.xz zfs-utils-linux-0.6.5.8_4.8.7_1-1-x86_64.pkg.tar.xz spl-linux-0.6.5.8_4.8.7_1-1-x86_64.pkg.tar.xz spl-utils-linux-0.6.5.8_4.8.7_1-1-x86_64.pkg.tar.xz
exit
umount /mnt/boot
zfs umount -a

After that I rebooted and it came up without issue.

Now it looks like the zfs linux package is maybe fixed, so you may be able to update that instead of downgrading the kernel when in the chroot.

I have upgraded again after the reboot and didn't have any issues, so I'm on 4.8.8-2 again.

All in all it wasn't that bad, I just hadn't done this before and I wasn't sure if this was the right approach to fix it.

mouseman

Member

From: Outta nowhere

Registered: 2014-04-04

Posts: 291

Re: [SOLVED] ZFS on Root, zfs.ko kernel module missing

Yeah confirmed, updating fixed the issue.