Arch Linux

addadi

Member

Registered: 2015-10-12

Posts: 2

Kernel update killed my ZFS pool. I hacked back in via Jellyfin

So this happened to me a few weeks ago. Headless Arch server at home, ZFS for bulk storage, I'm abroad. Routine pacman -Syu, kernel goes from 6.15 to 6.18, reboot. System comes up fine. Pings respond. But SSH rejects every key I throw at it. Permission denied (publickey).

Long story short: DKMS tried to rebuild the ZFS module for the new kernel, failed silently, and the system booted into a state where my ZFS pool didn't exist. My /home was on ZFS. No /home = no authorized_keys. SSH was doing exactly what it should — looking for keys in a file that wasn't there.

No monitor, no IPMI, weeks away from getting home. The only service still responding was Jellyfin on port 8096. I ended up writing a C# plugin (I don't know C#, an LLM did the heavy lifting), publishing it through GitHub Actions, and using it as a remote shell through curl. 23 plugin builds from blank repo to root shell. Full story here (https://addadi.github.io/2026/04/17/how … ia-player/).

Pinning my kernel now. Lesson learned.

Last edited by addadi (2026-05-02 13:19:57)

cryptearth

Member

Registered: 2024-02-03

Posts: 2,104

Re: Kernel update killed my ZFS pool. I hacked back in via Jellyfin

what's your reason to run dkms in the first place? do you run a custom kernel? if so: instead of rely on dkms fork archzfs and setup for your custom kernel - this way you get the safety net of if zfs fails to build pacman will refuse the kernel because ot can't satisfy zfs
also: /home (or /home/user) isn't a smart play as you discovered - i do it this way: have /home and /home/user on root and have a dataset mounted to a folder inside my home
this way ssh still works even if the pool fails

cool using zfs - but sorry for pointing out the obvious: your setup has several pitfalls youshould fix to avoid such in the future

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 14,962

Re: Kernel update killed my ZFS pool. I hacked back in via Jellyfin

Moderator Note

Moving to System Administration

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,724

Re: Kernel update killed my ZFS pool. I hacked back in via Jellyfin

DKMS tried to rebuild the ZFS module for the new kernel, failed silently,

This does not happen. Build failures ALWAYS throw an error, you just ignored it.

addadi

Member

Registered: 2015-10-12

Posts: 2

Re: Kernel update killed my ZFS pool. I hacked back in via Jellyfin

what's your reason to run dkms in the first place? do you run a custom kernel? if so: instead of rely on dkms fork archzfs and setup for your custom kernel - this way you get the safety net of if zfs fails to build pacman will refuse the kernel because ot can't satisfy zfs
also: /home (or /home/user) isn't a smart play as you discovered - i do it this way: have /home and /home/user on root and have a dataset mounted to a folder inside my home
this way ssh still works even if the pool fails

cool using zfs - but sorry for pointing out the obvious: your setup has several pitfalls youshould fix to avoid such in the future

I for sure had a wrong setup. no doubt about it. The focus of the post is how I've used Jellyfin addon to diagnose and fix the mistake.

DKMS tried to rebuild the ZFS module for the new kernel, failed silently,

This does not happen. Build failures ALWAYS throw an error, you just ignored it.

That's probably true, I didn't notice it or overlooked it. I edited the post.

cryptearth

Member

Registered: 2024-02-03

Posts: 2,104

Re: Kernel update killed my ZFS pool. I hacked back in via Jellyfin

The focus of the post is how I've used Jellyfin addon to diagnose and fix the mistake.

please avoid blog-like posts as this way this topic currently doesn't has no meaningful content other than "I messed up by overlooking the warning signs amd had a bad setup in the first place"
to give this any helpful content please tell how you resolved your current setup
also: please avoid full-quotes

clfarron4

Member

From: London, UK

Registered: 2013-06-28

Posts: 2,171

Website

Re: Kernel update killed my ZFS pool. I hacked back in via Jellyfin

DKMS tried to rebuild the ZFS module for the new kernel, failed silently, and the system booted into a state where my ZFS pool didn't exist. My /home was on ZFS.

What version of zfs-dkms are you running? The ZFS DKMS would fail with 6.18 if you're still running 2.3.5 (we're now on 2.4.1 which supports kernel 6.19).

And it definitely tells you when the DKMS module fails to build. I have these kinds of lines in my pacman.log when ZFS fails to build:

==> dkms install --no-depmod zfs/2.4.1 -k 7.0.3-arch1-2
==> WARNING: `dkms install --no-depmod zfs/2.4.1 -k 7.0.3-arch1-2' exited 1

Also, there has been some behind the scenes changes of how the Arch-ZFS repositories are maintained, meaning you will definitely need to update your repository sources if you're using those.

Last edited by clfarron4 (2026-05-03 13:32:49)

---

Claire is fine.
Problems? I have dysgraphia, so clear and concise please.
My public GPG key for package signing
My x86_64 package repository