Error and no keyboard response with init=/bin/bash

iith4ahm · 2021-01-11 17:36:32

I'm thinking of trying to boot a system with just the kernel and maybe a bootloader, and a single process of my own creation. As a first step, I'm just trying to set init for the kernel with

init=/bin/bash

in the bootloader on my current full-fledged system (arch with I think either gummiboot or systemd-boot). Unfortunately I get an error and I can't interact in any way (not even key echoing)

:: running early hook [udev]
Starting version 247.2-1-arch
:: running hook [udev]
:: Triggering uevents...
:: performing fsck on '/dev/nvmc0n1p2'
/dev/nvmc0n1p2: clean, 422674/15564800 files, 47331349/62252544 blocks
:: mounting /dev/nvme0n1p2' on real root
:: running cleanup hook [udev]
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
[root@archlinux /]#

(copied by hand, so it may not be correct character for character)

Searching the internet I found these:
https://lists.gnu.org/archive/html/bug- … 00097.html
https://stackoverflow.com/questions/320 … -ioctl-for (not me, but similar error. I can't even enter a command like it looks like he could)

But it's not clear to me exactly what's going on in my case or how to fix it. Are /dev/console and /dev/tty set up by the kernel or init? Where do the virtual consoles Ctrl-Alt-F# fit into the process? It sounds like init=/bin/bash is a fairly common thing to do, for example to fix a forgotten root password, so I'm wondering if this issue is more with a (usb) keyboard detection or driver. Are the keyboards typically set up with systemd/udev and I'm bypassing that? Do I need to add a keyboard driver to my initramfs?

Does anyone know what's going on? Thank you

Last edited by iith4ahm (2021-01-11 17:38:10)

seth · 2021-01-11 22:00:56

https://forums.gentoo.org/viewtopic-t-1 … art-0.html
https://wiki.gentoo.org/wiki/Custom_Ini … ob_control

The input likely fails because no sysfs/dev/devpts is setup (see the init script in the gentoo thread)

iith4ahm · 2021-01-12 15:46:25

The input likely fails because no sysfs/dev/devpts is setup (see the init script in the gentoo thread)

Thanks Seth. What do you mean by "no sysfs/dev/devpts" is setup? I leaned that I can run commands in the init script and they will show on the screen - I just can't interact with the keyboard. /dev, /sys, and /proc seem to be populated when the script is run. Does that mean they are "set up"?

If I run (from the gentoo script, but I swapped out an echo command for tty)

#!/bin/bash
export PATH=/bin:/sbin:/usr/bin:/usr/sbin
export HOME=/root

mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

tty

bash

I get the original output with the addition of

mount: /sys: none already mounted on /proc.
mount: /dev: none already mounted on /proc
not a tty

Running the bash program and redirecting stdin, stdout and stderr to /dev/tty7 got rid of the bash error, but I still could not do anything. I tried doing the same thing and running it in the background with

/bin/bash </dev/tty7 >/dev/tty7 2>&1 &

but that caused a "kernel panic - not syncing: Attempted to kill init!". I'm guessing init is not supposed to finish.

Also, if it could be related to what's going on:
When I boot my system up normally, it doesn't go right into a display manager or a console. It leaves me with the output of the kernel and/or systemd. I can't type anything. If I press Ctl-Alt-1 it does nothing just like when I run my custom init. However, if I first press Ctl-Alt-2 I do get a virtual terminal, and then when I press Ctl-Alt-1 gnome starts up. It's been this way for a while. The difference with my init script is that I can't Ctl-Alt-anything.

Last edited by iith4ahm (2021-01-12 15:47:04)

seth · 2021-01-13 12:33:52

What do you mean by "no sysfs/dev/devpts" is setup?
…
/dev, /sys, and /proc seem to be populated when the script is run.

That. Unfortunately.

When I boot my system up normally, it doesn't go right into a display manager or a console. It leaves me with the output of the kernel and/or systemd. I can't type anything. If I press Ctl-Alt-1 it does nothing just like when I run my custom init. However, if I first press Ctl-Alt-2 I do get a virtual terminal, and then when I press Ctl-Alt-1 gnome starts up.

Could be either an entropy issue or the GPU stalls.

Did you try

setsid sh -c 'exec sh </dev/tty1 >/dev/tty1 2>&1'

(do NOT fork the command!)
The important part is setsid and you may have to pass absolute paths.

iith4ahm · 2021-01-13 20:24:52

#!/bin/bash
setsid sh -c 'exec sh </dev/tty1 >/dev/tty1 2>&1'

gives me

sh-5.1#

with the same unresponsive keyboard. If I try to run that command with init= instead of from a bash script I get

ERROR: Root device mounted successfully, but setsid does not exist.
Bailing out, you are on your own. Good luck.

sh: can't access tty: job control turned off
[rootfs ]# _

Again with no ability to interact. I'm assuming the kernel would have needed a full path name.

Could be either an entropy issue or the GPU stalls.

Do you think this could be the cause of everything? There are a lot of yellow NVIDIA message in journalctl. I will update below:

Extract:

Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE) Screen 0 deleted because of no matching config section.
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (II) UnloadModule: "modesetting"
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE) Device(s) detected, but none match those in the config file.
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE)
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: Fatal server error:
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE) no screens found(EE)
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE)
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: Please consult the The X.Org Foundation support
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]:          at http://wiki.x.org
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]:  for help.
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE)
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[495]: (EE) Server terminated with error (1). Closing log file.
Jan 13 14:25:36 lava /usr/lib/gdm-x-session[493]: Unable to run X server

Do you think if I sort out these driver/X issues my init=/bin/bash problem will take care of itself?

But why are X and systemd running in the first place? Aren't they in userspace, and I'm bypassing them by passing the init kernel parameter? -Oh, since I'm using systemd-boot, does that let systemd start doing its thing right away and not wait for the kernel to start it? If I switch bootloaders should I get a systemd/X free system, and maybe the kernel will get me a responsive tty?

Last edited by iith4ahm (2021-01-13 21:13:05)

seth · 2021-01-13 23:13:09

It should rid you of the error message?

About the keyboard: you could have the script print "lsmod" and compare that to a normal session (in case you rely on some explicitly loaded modules for the keyboard)

Is this an integrated keyboard or an external one? Do you have a replacement option (ie. another keyboard to attach)?

iith4ahm · 2021-01-21 00:54:07

Thank you seth.

I installed Arch Linux in virtualized QEMU drive, and I can set init=/bin/bash just fine and the keyboard works (although I still get the errors from bash about inappropriate ioctl). So I'll blame the root problem on some driver/nvidia issue.

If I try to set init=/home/init (my custom script) I get an error about the file not existing and ls shows the following. /dev/sda2 (I think my main partition) is mounted to /new_root instead of /. Is this expected?

ERROR: Root device mounted successfully, but /home/init does not exist.
Bailing out, you are on your own. Good luck.

sh: can't access tty; job control turned off
[rootfs ]# ls
VERSION        dev        init_functions   proc       sys
bin            etc        lib              root       tmp
buildconfig    hooks      lib64            run        usr
config         init       new_root         sbin       var
[rootfs ]# mount
none on / type rootfs (rw)
proc on ...
sys...
...
/dev/sda2 on /newroot type ext4 (rw,relatime)

Also, follow up question: Even the base installation of arch (or presumably most distros) has tons of files. This makes me feel like I don't understand everything that my system does. Is it reasonable to compile or find some binaries for bash, pam, sshd, nginx, and python and a few others, to have a web server with only a kernel and a few binaries? Then a simple init script to start them? Would this be worth it for a simple secure server I can wrap my head all the way around, or am I better off just using debian and "apt upgrade"ing regularly and not worrying about if I understand it all?

Last edited by iith4ahm (2021-01-21 03:23:42)

seth · 2021-01-21 08:15:59

/home is probably on an extra mount?

The failing input could be due to a missing kernel module that requires explicit loading.

seth before already wrote:

About the keyboard: you could have the script print "lsmod" and compare that to a normal session (in case you rely on some explicitly loaded modules for the keyboard)

Can you enter commands blindly (and just not get a visual reaction)?

Is it reasonable to compile or find some binaries for bash, pam, sshd, nginx, and python and a few others, to have a web server with only a kernel and a few binaries?

You probably want busybox in that case, but "it depends".
Whatever isn't there doesn't have to be hardened, but you'll also have to maintain the system, care for security updates, ensure your custom setup and config doesn't create weaknesses.

We're kinda close to violating board rules, but I'd rather take a look at sth. like alpinelinux before trying to invent YAD (yet another distro) - unless you have a *very* specific and unique usecase.

iith4ahm · 2021-01-26 18:25:33

Thank you again seth. If it'd be helpful for others, or if someone knows how to go about fixing it I welcome continuing this thread, but otherwise the virtualization solution is satisfactory for me for now. If that is the case is it best to mark as solved?

As mentioned before, In the virtualization solution I can't run scripts in my home directory. Perhaps this has to do with the bootloader/kernel/init process here: https://www.kernel.org/doc/html/latest/ … nitrd.html Maybe init is getting called on the initrd instead of the root filesystem. Perhaps /bin/bash may still exist in the initrd and that's why that works but not /home/... /init. Perhaps the kernel is trying to call my init script from the initrd, which does not have my home directory. If I use an initrd maybe I need to put a "primer" init there which calls pivot_root and then sets up my home init script to run. I don't understand why I don't have this problem on my non-virtualized system. Perhaps again something to do with the drivers. Computers are complicated.

Regarding previous questions:

Can you enter commands blindly (and just not get a visual reaction)?

As far as I know, no. I tried to carefully type "touch /home/.../asdfghjkl" with no luck checking after rebooting. ...unless it was in the initrd or something... to be honest I'm not sure I have it all straight. But if there's a command in my /home/.../init script and I set the kernel parameter to run the script, the standard output gets put to screen.

About the keyboard: you could have the script print "lsmod" and compare that to a normal session ... Is this an integrated keyboard or an external one

It is an external usb keyboard. lsmod put in an init script returns

Module              Size        Used by
ext4                933888      1
crc32c_generic       16384      0
crc16                  ...      1 ext4
mbcache                         1 ext4
jbd2                            1 ext4
crc32c_intel                    2
xhci_pci                        0
xhci_pci_renesas                1 xhci_pci

But when booted up all the way returns much more.

Module                  Size  Used by
isofs                  53248  1
cdrom                  77824  1 isofs
nvidia_drm             65536  3
nvidia_modeset       1232896  5 nvidia_drm
joydev                 28672  0
mousedev               24576  0
nvidia              34086912  180 nvidia_modeset
usbhid                 65536  0
usblp                  28672  0
uas                    32768  0
usb_storage            81920  2 uas
snd_hda_codec_realtek   143360  1
cfg80211              987136  0
edac_mce_amd           32768  0
snd_hda_codec_generic    98304  1 snd_hda_codec_realtek
ledtrig_audio          16384  1 snd_hda_codec_generic
snd_hda_codec_hdmi     73728  1
rfkill                 28672  4 cfg80211
snd_hda_intel          57344  3
8021q                  40960  0
snd_intel_dspcfg       24576  1 snd_hda_intel
soundwire_intel        45056  1 snd_intel_dspcfg
garp                   16384  1 8021q
soundwire_generic_allocation    16384  1 soundwire_intel
mrp                    20480  1 8021q
soundwire_cadence      36864  1 soundwire_intel
stp                    16384  1 garp
llc                    16384  2 stp,garp
snd_hda_codec         172032  4 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec_realtek
kvm                   933888  0
snd_hda_core          106496  5 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek
snd_hwdep              16384  1 snd_hda_codec
soundwire_bus          90112  3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence
nls_iso8859_1          16384  1
vfat                   24576  1
irqbypass              16384  1 kvm
fat                    86016  1 vfat
snd_soc_core          323584  1 soundwire_intel
drm_kms_helper        274432  1 nvidia_drm
crct10dif_pclmul       16384  1
wmi_bmof               16384  0
snd_compress           32768  1 snd_soc_core
crc32_pclmul           16384  0
ac97_bus               16384  1 snd_soc_core
ghash_clmulni_intel    16384  0
r8169                  98304  0
snd_pcm_dmaengine      16384  1 snd_soc_core
aesni_intel           372736  0
snd_pcm               147456  8 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,soundwire_intel,snd_compress,snd_soc_core,snd_hda_core,snd_pcm_dmaengine
ccp                   118784  0
cec                    69632  1 drm_kms_helper
realtek                24576  1
snd_timer              45056  1 snd_pcm
crypto_simd            16384  1 aesni_intel
sp5100_tco             20480  0
mdio_devres            16384  1 r8169
syscopyarea            16384  1 drm_kms_helper
cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
snd                   114688  16 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek,snd_timer,snd_compress,snd_soc_core,snd_pcm
glue_helper            16384  1 aesni_intel
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
rapl                   16384  0
pcspkr                 16384  0
libphy                151552  3 r8169,mdio_devres,realtek
k10temp                16384  0
i2c_piix4              28672  0
rng_core               16384  1 ccp
soundcore              16384  1 snd
fb_sys_fops            16384  1 drm_kms_helper
wmi                    36864  1 wmi_bmof
gpio_amdpt             20480  0
pinctrl_amd            32768  0
mac_hid                16384  0
gpio_generic           20480  1 gpio_amdpt
acpi_cpufreq           28672  0
drm                   569344  6 drm_kms_helper,nvidia_drm
crypto_user            20480  0
fuse                  163840  3
agpgart                53248  1 drm
bpf_preload            16384  0
ip_tables              32768  0
x_tables               53248  1 ip_tables
ext4                  933888  1
crc32c_generic         16384  0
crc16                  16384  1 ext4
mbcache                16384  1 ext4
jbd2                  151552  1 ext4
crc32c_intel           24576  2
xhci_pci               20480  0
xhci_pci_renesas       20480  1 xhci_pci

We're kinda close to violating board rules

I'm going to assume you're refering to keeping the forms arch-only.

mandraxx · 2021-11-15 20:01:20

Hi,

Got the same issue trying to build a custom ISO image.
Putting an eye in the generated initialramfs generated with mkinitcpio, I found that udev must be started to get keyboard working.

This piece of shell is working for me as init script opening an interactive terminal :

#!/bin/bash

mount -t proc       proc        /proc   -o nosuid,noexec,nodev
mount -t sysfs      sys         /sys    -o nosuid,noexec,nodev
mount -t devtmpfs   dev         /dev    -o mode=0755,nosuid
mount -t tmpfs      run         /run    -o nosuid,nodev,mode=0755
mount -t tmpfs      tmpfs       /tmp
mkdir -m755 /run/initramfs

kmod static-nodes --format=tmpfiles --output=/run/tmpfiles.d/kmod.conf
systemd-tmpfiles --prefix=/dev --create --boot
/usr/lib/systemd/systemd-udevd --daemon --resolve-names=never
udevadm trigger --action=add --type=subsystems
udevadm trigger --action=add --type=devices
udevadm settle

/usr/bin/setsid bash -c "exec /bin/bash -i <dev/console >/dev/console 2>&1"

Trilby · 2021-11-15 21:50:27

Udev is not strictly needed, but the relevant devices need to be initialized and modules need to be loaded. Doing this manually (a find command with -exec on the /sys/ filesystem) in an init script is challenging. Udev does all this for you, but it's a beast. Mdev is tiny, efficient, and handles this pretty well.

But depending on which method you take, you might also need to compile your own kernel (e.g., if you need hotplugging enabled).

FWIW, here's an init I was working on with mdev some time ago:

#!/bin/ash

mount -t proc      proc    /proc     -o nosuid,noexec,nodev
mount -t sysfs     sys     /sys      -o nosuid,noexec,nodev
mount -t tmpfs     run     /run      -o mode=0755,nosuid,nodev

mkdir -p /dev/pts /dev/shm
mount -t devpts    devpts  /dev/pts  -o mode=0620,gid=5,nosuid,noexec
mount -t tmpfs     shm     /dev/shm  -o mode=1777,nosuid,nodev

ln -sf /proc/self/fd  /dev/fd
ln -sf fd/0           /dev/stdin
ln -sf fd/1           /dev/stdout
ln -sf fd/2           /dev/stderr

echo /bin/mdev > /proc/sys/kernel/hotplug
touch /dev/mdev.log
mdev -s
find /sys -name 'modalias' -type f -exec cat '{}' \+ | \
	sort -u | \
	xargs -I MOD -P 0 modprobe MOD >/dev/null 2>&1

mount -o remount,ro /
#fsck -ATP -t vfat,ext4 -p
mount -o remount,rw /
mount -a

cat /var/random.seed > /dev/urandom
cat /etc/hostname > /proc/sys/kernel/hostname
ip link set up dev lo

Last edited by Trilby (2021-11-15 21:52:21)

Arch Linux

#1 2021-01-11 17:36:32

Error and no keyboard response with init=/bin/bash

#2 2021-01-11 22:00:56

Re: Error and no keyboard response with init=/bin/bash

#3 2021-01-12 15:46:25

Re: Error and no keyboard response with init=/bin/bash

#4 2021-01-13 12:33:52

Re: Error and no keyboard response with init=/bin/bash

#5 2021-01-13 20:24:52

Re: Error and no keyboard response with init=/bin/bash

#6 2021-01-13 23:13:09

Re: Error and no keyboard response with init=/bin/bash

#7 2021-01-21 00:54:07

Re: Error and no keyboard response with init=/bin/bash

#8 2021-01-21 08:15:59

Re: Error and no keyboard response with init=/bin/bash

#9 2021-01-26 18:25:33

Re: Error and no keyboard response with init=/bin/bash

#10 2021-11-15 20:01:20

Re: Error and no keyboard response with init=/bin/bash

#11 2021-11-15 21:50:27

Re: Error and no keyboard response with init=/bin/bash

Board footer