You are not logged in.

Pages: 1

#1 2023-10-20 06:18:18

Lockheed
Member
Registered: 2010-03-16
Posts: 1,550

[solved] ZFS on root stopped booting - GLIBC error and kernel panic

I have multiple machines with the same setup and one of them, after a reboot, developed a strange issue being unable to boot:
https://i.imgur.com/fP54adf.png
https://i.imgur.com/6nyMED2.jpg
This is on linux-lts

I chrooted into the installation, updated the kernel, recreated mkinitcpio, tried installing regular linux kernel + zfs module and boot from it. Nothing helped.

Last edited by Lockheed (2023-10-23 10:00:54)

Offline

#2 2023-10-20 10:29:39

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 24,960

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

Don't post images directly, link to them.

you need to rebuild your zfs-utils against the new glibc, likely the kernel as well.

Moving to AUR issues...

Last edited by V1del (2023-10-20 10:32:24)

Offline

#3 2023-10-20 18:58:15

Lockheed
Member
Registered: 2010-03-16
Posts: 1,550

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

I reinstalled zfs-utils (it was not building anything, maybe because I have a zfs repo in pacman.conf) and reinstalled the kernels, which triggered mkinitcpio. The problem continues.

Offline

#4 2023-10-20 19:48:33

loqs
Member
Registered: 2014-03-06
Posts: 18,732

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

What is the output of `pacman -Qi glibc`?  When did you last perform a full system update?

Last edited by loqs (2023-10-20 19:48:47)

Offline

#5 2023-10-20 20:13:19

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 71,689

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

glibc is at 2.38 since early august, boot the install iso, mount the installed system at /mnt and run "pacman -root /mnt -Qikk glibc"
If that looks good, arch-chroot into the system and regenerate the initramfs, don't forget to mount the /boot partition and pay attention to any errors that show up.

Edit: so, that would have been a really awesome moment for an F5…

Last edited by seth (2023-10-20 20:13:59)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#6 2023-10-22 16:48:40

Lockheed
Member
Registered: 2010-03-16
Posts: 1,550

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

# pacman -Qikk glibc
Name            : glibc
Version         : 2.37-3
Description     : GNU C Library
Architecture    : x86_64
URL             : https://www.gnu.org/software/libc
Licenses        : GPL  LGPL
Groups          : None
Provides        : None
Depends On      : linux-api-headers>=4.10  tzdata  filesystem
Optional Deps   : gd: for memusagestat
                  perl: for mtrace [installed]
Required By     : aalib  alsa-lib  argon2  attr  audit  base  bash  binutils  bison  bluez-libs  brotli  btrfs-progs  bzip2  cdparanoia  coreutils  cups  dav1d  dbus-glib  dbus-python  debugedit  device-mapper  diffutils  dotnet-host  dotnet-runtime
                  duktape  expat  fakeroot  file  findutils  flex  fribidi  fuse2  fuse3  gawk  gcc-libs  gdbm  ghostscript  giflib  gnupg  gperftools  grep  gsm  gzip  hyphen  icu  ijs  iproute2  jack2  jansson  json-c  kbd  keyutils  kmod  krb5
                  l-smash  less  libass  libassuan  libasyncns  libatasmart  libatomic_ops  libavc1394  libblockdev  libbpf  libburn  libcaca  libcamera  libcamera-ipa  libcap  libcap-ng  libcdio-paranoia  libcups  libdatrie  libdv  libedit  libelf
                  libepoxy  libevdev  libexif  libexttextcat  libffi  libfreeaptx  libgpg-error  libgtop  libice  libidn  libimobiledevice  libjpeg-turbo  libksba  liblc3  libldac  libmnl  libndp  libnfnetlink  libnghttp2  libnl  libnsl  libogg
                  libomxil-bellagio  libopenmpt  libp11-kit  libpaper  libpcap  libpciaccess  libpgm  libplist  libraw1394  libsamplerate  libsasl  libseccomp  libsndfile  libsodium  libstemmer  libtasn1  libthai  libtheora  libtiff  libtommath  libtool
                  libunistring  libusb  libusbmuxd  libutempter  libverto  libx11  libxau  libxcrypt  libxcvt  libxdmcp  libxklavier  libxshmfence  libyaml  light-locker  lilv  lpsolve  lz4  lzo  m4  make  mdadm  mkinitcpio-busybox  mpfr  mtdev  ncurses
                  neon  nettle  npth  nspr  ocl-icd  opencore-amr  openssh  openssl  opus  orc  pacman  pam  patch  pciutils  perl  pinentry  pixman  pkgconf  poppler  popt  portaudio  procps-ng  python-pydantic  qpdf  raptor  rasqal  rav1e  readline
                  sbc  sdl2  sed  serd  shadow  sord  sratom  startup-notification  sudo  svt-av1  systemd-libs  tailscale  tar  tdb  twolame  udisks2  util-linux-libs  vid.stab  volume_key  vulkan-icd-loader  wavpack  wayland  wget  which  xorg-xrdb
                  xvidcore  xxhash  zeromq  zimg  zlib  zstd
Optional For    : tzdata
Conflicts With  : None
Replaces        : None
Installed Size  : 47.20 MiB
Packager        : Frederik Schwan <freswa@archlinux.org>
Build Date      : Sun Apr 30 12:24:27 2023
Install Date    : Wed Aug 23 15:42:13 2023
Install Reason  : Installed as a dependency for another package
Install Script  : Yes
Validated By    : Signature

So looked fine, but I reinstalled it anyway. Then, I regenerated initramfs (for the 10th time).
No change.

Offline

#7 2023-10-22 16:51:58

loqs
Member
Registered: 2014-03-06
Posts: 18,732

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

loqs wrote:

When did you last perform a full system update?

Why is your system still using glibc 2.37-3?

Offline

#8 2023-10-22 20:02:14

Lockheed
Member
Registered: 2010-03-16
Posts: 1,550

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

loqs wrote:
loqs wrote:

When did you last perform a full system update?

Why is your system still using glibc 2.37-3?

Daily since I am trying to solve that issue.
my pacman.conf additions:

[archzfs]
Server = http://archzfs.com/$repo/x86_64
[zfs-linux-lts]
Server = http://kernels.archzfs.com/$repo/

Offline

#9 2023-10-22 20:47:28

loqs
Member
Registered: 2014-03-06
Posts: 18,732

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

Please post the output of:

pacman-conf

Offline

#10 2023-10-23 09:26:10

Lockheed
Member
Registered: 2010-03-16
Posts: 1,550

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

# pacman-conf
[options]
RootDir = /
DBPath = /var/lib/pacman/
CacheDir = /var/cache/pacman/pkg/
HookDir = /etc/pacman.d/hooks/
GPGDir = /etc/pacman.d/gnupg/
LogFile = /var/log/pacman.log
HoldPkg = pacman
HoldPkg = glibc
Architecture = x86_64
CheckSpace
ParallelDownloads = 1
CleanMethod = KeepInstalled
SigLevel = PackageRequired
SigLevel = PackageTrustedOnly
SigLevel = DatabaseOptional
SigLevel = DatabaseTrustedOnly
LocalFileSigLevel = PackageOptional
LocalFileSigLevel = PackageTrustedOnly
[archzfs]
Usage = All
Server = http://archzfs.com/archzfs/x86_64
[zfs-linux]
Usage = All
Server = http://kernels.archzfs.com/zfs-linux/
[zfs-linux-lts]
Usage = All
Server = http://kernels.archzfs.com/zfs-linux-lts/
[zfs-linux-zen]
Usage = All
Server = http://kernels.archzfs.com/zfs-linux-zen/
[core]
Usage = All
Server = https://archive.archlinux.org/repos/2023/07/01/core/os/x86_64
[extra]
Usage = All
Server = https://archive.archlinux.org/repos/2023/07/01/extra/os/x86_64
[community]
Usage = All
Server = https://archive.archlinux.org/repos/2023/07/01/community/os/x86_64
[multilib]
Usage = All
Server = https://archive.archlinux.org/repos/2023/07/01/multilib/os/x86_64

Offline

#11 2023-10-23 09:31:34

seth
Member
From: Don't DM me only for attention
Registered: 2012-09-03
Posts: 71,689

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

Server = https://archive.archlinux.org/repos/2023/07/01/core/os/x86_64
Server = https://archive.archlinux.org/repos/2023/07/01/extra/os/x86_64
Server = https://archive.archlinux.org/repos/2023/07/01/community/os/x86_64
Server = https://archive.archlinux.org/repos/2023/07/01/multilib/os/x86_64

You're keeping the system in July, but there's no such limitation on the archzfs.com repos…

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#12 2023-10-23 09:33:28

Lockheed
Member
Registered: 2010-03-16
Posts: 1,550

Re: [solved] ZFS on root stopped booting - GLIBC error and kernel panic

I don't know why that's the case. I certainly did not manually put those changes in.
I can see it's because my mirror file contains just a single line with this old reference. I will fix it and report back.

EDIT:
Problem solved. Thanks!

Last edited by Lockheed (2023-10-23 10:00:40)

Offline

Pages: 1

Board footer

Powered by FluxBB