Arch Linux

troytjh

Member

Registered: 2019-03-18

Posts: 4

[SOLVED] Kernel panic - not syncing

I have been using Archlinux for about a year and a half with few issues in recent months. Earlier today I performed a reboot to switch the kernel from 5.8.5 to 5.8.6. The result was the computer frozen on the splash screen, with the motherboard logo. This occurred after selecting the boot option in GRUB. This was a custom-built kernel with the OpenRGB patch, though I also had the stock kernel installed. I attempted to boot with the stock kernel at this point with the same result. I prepared a bootable USB to investigate the issue further. One possibility was droidcam, which I had installed recently to use my phone's camera as a webcam. It installs v4l2loopback and v4l2loopback-dkms which is an extra module added to the kernel. I uninstalled the application and added modules but received the same result. I also installed Linux-lts while in the chroot, believing that the issue may be with version 5.8.6. The result was the same. I also regenerated the grub config at different points during this process and compared the drive UUIDs to the UUIDs present in /etc/fstab. I added --verbose nosplash debug to the grub entry to attempt to get more logging information and observed a Kernel panic - not syncing error. I read through the General troubleshooting arch wiki on kernel panics. I was unable to get to a shell through the three types of kernel parameters under "reboot into root shell and fix problem". I have not found a way to get the full log at boot, as a result of the kernel panic. The end portion of the kernel panic is:

[    1.220508]          HOME=/
[    1.221479]          TERM=linux
[    1.222430]          BOOT=/vmlinuz-linux-custom
[    1.223390]  Run /etc/init as init process
[    1.224336]      with arguments:
[    1.225270]          /etc/init
[    1.226174]          --verbose
[    1.227051]          nosplash
[    1.227901]      with environment:
[    1.228733]          HOME=/
[    1.229535]          TERM=linux
[    1.230317]          BOOT_IMAGE=/vmlinuz-linux-custom
[    1.231100]  Run /bin/init as init process
[    1.231885]      with arguments:
[    1.232626]          /bin/init
[    1.233332]          --verbose
[    1.234031]          nosplash
[    1.234705]      with environment:
[    1.235391]          HOME=/
[    1.236033]          TERM=linux
[    1.236666]          BOOT_IMAGE=/vmlinuz-linux-custom
[    1.237314]  Run /bin/sh as init process
[    1.237950]      with arguments:
[    1.238573]          /bin/sh
[    1.239172]          --verbose
[    1.239769]          nosplash
[    1.240343]      with environment:
[    1.240909]          HOME=/
[    1.241461]          TERM=linux
[    1.241993]          BOOT_IMAGE=/vmlinuz-linux-custom
[    1.242546]  Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    1.243832]  CPU: 8 PID: 1 Comm: swapper/0 Not tainted 5.8.7-arch1-1-custom #11
[    1.244531] Hardware name: System manufacturer System Product Name/ROG MAXIMUS X CODE, BIOS 2301 02/25/2020
[    1.245306]  Call Trace:
[    1.246798]      dump_stack+0x6b/0x88
[    1.246798]      ? rest_init+0xa0/0xbf
[    1.247544]      panic+0x112/0x2e8
[    1.248279]      ? rest_init+0xa0/0xbf
[    1.249011]      kernel_init+0xfc/0x101
[    1.250502]      ret_from_fork+0x22/0x30
[    1.250502]  Kernel Offset: 0x4c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.251385]  ---[ end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

I can normally point to a resent change that may have caused this severe of an issue, though I'm at a loss with this one. Would anybody have any suggestions for a potential cause of the issue or a method to gain more information on what is occuring.

Last edited by troytjh (2020-09-06 23:06:25)

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,517

Website

Re: [SOLVED] Kernel panic - not syncing

I have no idea why it's failing, but what is failing is clear: the kernel cannot find your init.  Are you using systemd?  Have you made any changes to systemd/init?  You could try adding the kernel parameter "init=/usr/lib/systemd/systemd" but as this seems to be failing before the switch to the new root, then determining whether/where systemd's init binary is in your initramfs would be necessary.

From a live medium, you can chroot in and check your initramfs contents with the following:

bsdtar tf /boot/initramfs-linux-custom.img

Look for "init" and the above-listed systemd binary path.

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

troytjh

Member

Registered: 2019-03-18

Posts: 4

Re: [SOLVED] Kernel panic - not syncing

Yes, the system is using systemd.
/usr/lib/systemd appears to be present in the initramfs

VERSION
bin
buildconfig
config
dev/
etc/
etc/fstab
etc/initrd-release
etc/ld.so.cache
etc/ld.so.conf
etc/modprobe.d/
etc/mtab
hooks/
hooks/udev
init
init_functions
lib
lib64
new_root/
proc/
run/
sbin
sys/
tmp/
usr/
usr/bin/
usr/bin/[
usr/bin/arch
usr/bin/ash
usr/bin/awk
usr/bin/base64
usr/bin/basename
usr/bin/bc
usr/bin/blkdiscard
usr/bin/blkid
usr/bin/busybox
usr/bin/bzip2
usr/bin/cat
usr/bin/chgrp
usr/bin/chmod
usr/bin/chown
usr/bin/chroot
usr/bin/clear
usr/bin/cp
usr/bin/cpio
usr/bin/cttyhack
usr/bin/cut
usr/bin/dd
usr/bin/depmod
usr/bin/df
usr/bin/dirname
usr/bin/dmesg
usr/bin/du
usr/bin/echo
usr/bin/env
usr/bin/expr
usr/bin/fallocate
usr/bin/false
usr/bin/fatattr
usr/bin/free
usr/bin/fsck
usr/bin/fsck.ext2
usr/bin/fsck.ext3
usr/bin/fsck.ext4
usr/bin/fsfreeze
usr/bin/fstrim
usr/bin/fsync
usr/bin/getopt
usr/bin/grep
usr/bin/gzip
usr/bin/halt
usr/bin/head
usr/bin/hexdump
usr/bin/hexedit
usr/bin/i2ctransfer
usr/bin/ifconfig
usr/bin/init
usr/bin/insmod
usr/bin/install
usr/bin/ip
usr/bin/ipaddr
usr/bin/iplink
usr/bin/ipneigh
usr/bin/iproute
usr/bin/iprule
usr/bin/iptunnel
usr/bin/kbd_mode
usr/bin/kill
usr/bin/killall
usr/bin/kmod
usr/bin/less
usr/bin/link
usr/bin/ln
usr/bin/loadfont
usr/bin/loadkmap
usr/bin/losetup
usr/bin/ls
usr/bin/lsmod
usr/bin/lsscsi
usr/bin/lzop
usr/bin/md5sum
usr/bin/mkdir
usr/bin/mkfifo
usr/bin/mknod
usr/bin/mkpasswd
usr/bin/mktemp
usr/bin/modinfo
usr/bin/modprobe
usr/bin/mount
usr/bin/mountpoint
usr/bin/mv
usr/bin/nc
usr/bin/netstat
usr/bin/nologin
usr/bin/nproc
usr/bin/nsenter
usr/bin/nslookup
usr/bin/nuke
usr/bin/openvt
usr/bin/partprobe
usr/bin/paste
usr/bin/pgrep
usr/bin/pidof
usr/bin/ping
usr/bin/ping6
usr/bin/poweroff
usr/bin/printf
usr/bin/ps
usr/bin/pwd
usr/bin/readlink
usr/bin/realpath
usr/bin/reboot
usr/bin/resume
usr/bin/rm
usr/bin/rmdir
usr/bin/rmmod
usr/bin/route
usr/bin/run-init
usr/bin/sed
usr/bin/seq
usr/bin/setfattr
usr/bin/setfont
usr/bin/sh
usr/bin/sha1sum
usr/bin/sha256sum
usr/bin/sha512sum
usr/bin/shuf
usr/bin/sleep
usr/bin/sort
usr/bin/stat
usr/bin/strings
usr/bin/switch_root
usr/bin/sync
usr/bin/systemd-tmpfiles
usr/bin/tac
usr/bin/tail
usr/bin/tar
usr/bin/tcpsvd
usr/bin/tee
usr/bin/telnet
usr/bin/test
usr/bin/tftp
usr/bin/touch
usr/bin/true
usr/bin/truncate
usr/bin/ts
usr/bin/udevadm
usr/bin/udhcpc
usr/bin/udhcpc6
usr/bin/umount
usr/bin/uname
usr/bin/uniq
usr/bin/unlink
usr/bin/unshare
usr/bin/unzip
usr/bin/uptime
usr/bin/vi
usr/bin/wc
usr/bin/wget
usr/bin/which
usr/bin/xxd
usr/bin/xz
usr/bin/yes
usr/lib/
usr/lib/modprobe.d/
usr/lib/modprobe.d/bluetooth-usb.conf
usr/lib/modprobe.d/nvdimm-security.conf
usr/lib/modprobe.d/systemd.conf
usr/lib/modules/
usr/lib/modules/5.8.7-arch1-1-custom/
usr/lib/modules/5.8.7-arch1-1-custom/kernel/
usr/lib/modules/5.8.7-arch1-1-custom/kernel/cdrom.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/crc-itu-t.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/crc16.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/crc32c-intel.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/crc32c_generic.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/ext4.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/firewire-core.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/firewire-sbp2.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/hid-generic.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/hid.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/jbd2.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/mbcache.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/mmc_block.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/mmc_core.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/sr_mod.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/uas.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/usb-storage.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/usbhid.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/virtio_blk.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/xhci-hcd.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/xhci-pci-renesas.ko
usr/lib/modules/5.8.7-arch1-1-custom/kernel/xhci-pci.ko
usr/lib/modules/5.8.7-arch1-1-custom/modules.alias.bin
usr/lib/modules/5.8.7-arch1-1-custom/modules.builtin.alias.bin
usr/lib/modules/5.8.7-arch1-1-custom/modules.builtin.bin
usr/lib/modules/5.8.7-arch1-1-custom/modules.dep.bin
usr/lib/modules/5.8.7-arch1-1-custom/modules.devname
usr/lib/modules/5.8.7-arch1-1-custom/modules.softdep
usr/lib/modules/5.8.7-arch1-1-custom/modules.symbols.bin
usr/lib/systemd/
usr/lib/systemd/systemd-udevd
usr/lib/udev/
usr/lib/udev/ata_id
usr/lib/udev/rules.d/
usr/lib/udev/rules.d/50-udev-default.rules
usr/lib/udev/rules.d/60-persistent-storage.rules
usr/lib/udev/rules.d/64-btrfs.rules
usr/lib/udev/rules.d/80-drivers.rules
usr/lib/udev/scsi_id
usr/lib64
usr/local/
usr/local/bin
usr/local/lib
usr/local/sbin
usr/sbin
var/
var/run

specifying the init also fails. The new message is:

sd 0:0:0:0: [sda] Writer cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdb] 4096-byte physical blocks
sd 2:0:0:0: [sdb] Write Protect is off
sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdb] Attached SCSI disk
    sda: sda1
sd 0:0:0:0: [sda] Attached SCSI disk
Freeing unused decrypted memory: 2040k
Freeing unused kernel image (initmem) memory: 1644k
Write protecting the kernel read-only data: 24576k
Freeing unused kernel image (text/rodata gap) memory: 2044k
Freeing unused kernel image (rodata/data gap) memory: 276k
x86/mm: Checked W+X mappings: passed, no W+X pages found.
x86/mm: Checking user space page tables
x86/mm: Checking W+X mappings: passed, no W+X pages found.
Run /init as init process
    with arguments:
        /init
    with environment:
        HOME=/
        TERM=linux
        BOOT_IMAGE=/vmlinuz-linux-custom
Failed to execute /init (error -2)
Run /usr/lib/systemd/systemd as init process
    with arguments:
        /usr/lib/systemd/systemd
    with environment:
        HOME=/
        TERM=linux
        BOOT_IMAGE=/vmlinuz-linux-custom
Kernel panic - not syncing: Requested init /usr/lib/systemd/systemd failed (error -2).
CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.8.7-arch1-1-custom #11
Hardware name: System manufacturer System Product Name/ROG MAXIMUS X CODE, BIOS 2301 02/25/2020
Call Trace:
    dump_stack+0x6b/0x88
    ? rest_init+0xa0/0xbf
    panic+0x112/0x2e8
    ? rest_init+0xbf/0xbf
    ? rest_init+0xbf/0xbf
    kernel_init+0xb0/0x101
    ret_from_fork+0x22/0x30
Kernel Offset: 0x1440000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: Requested init /usr/lib/systemd/systemd failed (error -2). ]---

sd likely refers to the secondary harddrive, configured with bcache. The main drive is nvme. I have not made any changes to systemd/init that I'm aware of.

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,517

Website

Re: [SOLVED] Kernel panic - not syncing

Odd, now init is found, but it fails.  Though this looks like an encrypted system, right?.  I don't have any experience with full disk encryption.  Perhaps someone who does can comment on those messages.

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

troytjh

Member

Registered: 2019-03-18

Posts: 4

Re: [SOLVED] Kernel panic - not syncing

The system is not encrypted. The behavior is strange. I may end up performing a clean install with some of the configs from the current install.

troytjh

Member

Registered: 2019-03-18

Posts: 4

Re: [SOLVED] Kernel panic - not syncing

Thank you for your time in helping me diagnose this issue. I'm unsure what caused the issue, but a clean install appears to have fixed the issue. Everythings running correctly now. Again Thank you.