You are not logged in.

#1 2019-01-04 21:50:30

magiusche
Member
Registered: 2008-12-05
Posts: 37

[Solved] chroot: failed to run command "/bin/bash": Input/output error

Hi guys, and first of all, sorry for my poor english.
I'll go straight to the point.
I've a Cloud VPS hosted to OVH.
I need to recover it after a kernel panic so i ran into Rescue Mode.
Basically OVH create a new VPS (Linux rescue-pro 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64) in witch there is a mounted SDB1 (my original disc with ARCHLinux).
I Just wanted to execute chroot to reintall all pacman packages!
But when i try to use chroot (as described here: https://wiki.archlinux.org/index.php/Chroot ) i got this error: chroot: failed to run command "/bin/bash": Input/output error

The structure is

/ (debian x86_64)
/mnt/sdb1 (archlinux x86_64)

I've tried theese steps:

STEP A

  1. Download https://www.archlinux.org/packages/?nam … ll-scripts

  2. extract arch-chroot and place it into /mnt/sdb1/home/arch/arch-chroot

  3. call /mnt/sdb1/home/arch/arch-chroot

  4. returns:

    root@rescue-pro:~# /mnt/sdb1/home/arch/arch-chroot /mnt/sdb1 /bin/bash
    chroot: failed to run command "/bin/bash": Input/output error
  5. calling /mnt/sdb1/bin/bash works without errors!

STEP B

  1. follow the steps described in https://wiki.archlinux.org/index.php/Ch … ing_chroot

  2. but when i call chroot the return is

    root@rescue-pro:~# chroot /mnt/sdb1 /bin/bash
    chroot: failed to run command "/bin/bash": Input/output error

I've check disk with fsck and badblocks and all seems normal.

Do you guys have any suggestions?

Thx in advance!

Last edited by magiusche (2019-01-06 23:35:26)


Hope i help you!
and
VICE.VERSA

Offline

#2 2019-01-04 22:42:30

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

locale issue?
Check the output of "locale" on the rescue system and try to remember the locales of your arch system (some_SOME-UTF-8?)
In case they differ, try to export the correct LC_ALL to chroot.

Offline

#3 2019-01-04 23:39:22

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

seth wrote:

locale issue?
Check the output of "locale" on the rescue system and try to remember the locales of your arch system (some_SOME-UTF-8?)
In case they differ, try to export the correct LC_ALL to chroot.

first.. thx for anwere :-)

outpout of locale in the Rescue (debian)

root@rescue-pro:~# locale
LANG=it_IT.UTF-8
LANGUAGE=
LC_CTYPE="it_IT.UTF-8"
LC_NUMERIC="it_IT.UTF-8"
LC_TIME="it_IT.UTF-8"
LC_COLLATE="it_IT.UTF-8"
LC_MONETARY="it_IT.UTF-8"
LC_MESSAGES="it_IT.UTF-8"
LC_PAPER="it_IT.UTF-8"
LC_NAME="it_IT.UTF-8"
LC_ADDRESS="it_IT.UTF-8"
LC_TELEPHONE="it_IT.UTF-8"
LC_MEASUREMENT="it_IT.UTF-8"
LC_IDENTIFICATION="it_IT.UTF-8"
LC_ALL=

(i've already tried to change only LANG to it_IT.UTF-8 (like the arch)

root@rescue-pro:~# cat /mnt/sdb1/etc/locale.conf
LANG="it_IT.UTF-8"

LC_TIME="it_IT.UTF-8"
root@rescue-pro:~# cat /mnt/sdb1/etc/locale.gen
# Created by cloud-init v. 0.7.6 on Sun, 16 Dec 2018 22:30:33 +0000
it_IT.UTF-8 UTF-8
it_IT ISO-8859-1
it_IT@euro ISO-8859-15

But

root@rescue-pro:~# /mnt/sdb1/home/arch/arch-chroot /mnt/sdb1/
chroot: impossibile eseguire il comando "/bin/bash": Input/output error

Same results :-(

Do you know where i can find a sort of log more detailed about error of the last chroot fail?

Last edited by magiusche (2019-01-04 23:42:03)


Hope i help you!
and
VICE.VERSA

Offline

#4 2019-01-05 08:27:04

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

You could strace it.
Could it be that there's simply some junk in /mnt/sdb1/etc/* or  /mnt/sdb1/root/.* sourced by bash?

Offline

#5 2019-01-05 15:11:22

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 27,288
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Seth, wouldn't that result in bash itself returning the error?  It looks like chroot is not even able to start bash (although a strace would really determine this one way or the other).


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#6 2019-01-05 16:12:18

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Omg... i've just used strace but it is out of mine comprehension!
I'll post resutl here... can you please help me to figure out something usefull from it?
thx in advance!

strace /mnt/sdb1/home/arch/arch-chroot /mnt/sdb1/

results: https://pastebin.com/mJREz2AC

Last edited by magiusche (2019-01-05 16:12:37)


Hope i help you!
and
VICE.VERSA

Offline

#7 2019-01-05 16:27:12

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

make it "strace -f chroot …" to cover child processes and skip the arch-chroot noise.

Offline

#8 2019-01-05 16:41:53

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 27,288
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

straces are a bit arcane, but there are small details you can learn to pick up on pretty quickly.  Most strace output is still beyond my comprehension, but I know enough to be able to filter what is and is not comprehensible to me, and I know how to look for a few expected bits.  For example, in this strace I'm looking for the execve line calling bash - but it's not there.  Seth almost certainly has the reason for that: the line I want to see (and the lines immediately after it) are in a child process which was not included in this trace, but can be obtained as noted above.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#9 2019-01-05 22:49:34

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Ok! I've tried this (in the meantime my MasterCooler CRASHED!!!! so i'm writing from my Laptop)

strace -f chroot /mnt/sdb1 /bin/bash &> /mnt/sdb1/home/arch/chroot_strace.txt

Results: https://pastebin.com/yWnZMug7

Again.. thx in advance for the help!


Hope i help you!
and
VICE.VERSA

Offline

#10 2019-01-05 23:26:45

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

It's from the execve in chroot, bash is never started.
Let's see whether it's specific to that binary:

strace -f chroot /mnt/sdb1 /bin/false &> /mnt/sdb1/home/arch/chroot_stracemore.txt

Offline

#11 2019-01-06 10:38:29

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Here the results... thx again

https://pastebin.com/9rSUUe1Q


Hope i help you!
and
VICE.VERSA

Offline

#12 2019-01-06 13:41:08

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

execve("/bin/false", ["/bin/false"], [/* 14 vars */]) = -1 EIO (Input/output error)

Same problem, entirely unrelated to the bash binary.

Can you do a chroot to root, ie.

chroot / /bin/bash

?

Offline

#13 2019-01-06 14:27:09

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 10,894
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Please also post the output of

file /usr/bin/chroot /mnt/sdb1/usr/bin/bash

and

smartctl -a /dev/sdb

Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#14 2019-01-06 14:34:45

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

If the execve of the rescue system doesn't throw ENOEXEC for what I suspect WorMzy suspects, I'll throw a tantrum…

Offline

#15 2019-01-06 14:41:08

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 27,288
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

I have seen reports of execve failing with EIO due to the wrong path.  It is notable here that the chroot is trying /bin/bash and /bin/false rather than /usr/bin/bash (please try `chroot /mnt/sdb1 /usr/bin/bash`).  There could be a problem with the /bin symlink on sdb1 and/or chroot/exec could be failing to follow the symlink properly.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#16 2019-01-06 14:53:23

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

done...and fails.

after reboot i've ran:

strace -f chroot / /bin/bash &> /mnt/sdb1/home/arch/chroot_stracemore.txt

here the results: https://pastebin.com/KnREutmN


Hope i help you!
and
VICE.VERSA

Offline

#17 2019-01-06 14:56:31

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

WorMzy wrote:

Please also post the output of

file /usr/bin/chroot /mnt/sdb1/usr/bin/bash

and

smartctl -a /dev/sdb

here the results:

root@rescue-pro:~# file /usr/bin/chroot /usr/sbin/chroot /mnt/sdb1/usr/bin/bash
/usr/bin/chroot:        cannot open `/usr/bin/chroot' (No such file or directory)
/usr/sbin/chroot:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=4c35305a438b3e07f1dc443353ddb5d154902d53, stripped
/mnt/sdb1/usr/bin/bash: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=42602c973215ba5b8ab5159c527e72f38e83ee52, stripped

and

root@rescue-pro:~# smartctl -a /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-6-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               QEMU
Product:              QEMU HARDDISK
Revision:             2.5+
User Capacity:        21.474.836.480 bytes [21,4 GB]
Logical block size:   512 bytes
LU is thin provisioned, LBPRZ=0
Rotation Rate:        5400 rpm
Device type:          disk
Local Time is:        Sun Jan  6 14:55:58 2019 UTC
SMART support is:     Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

Device does not support Self Test logging

Hope i help you!
and
VICE.VERSA

Offline

#18 2019-01-06 14:59:34

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Trilby wrote:

I have seen reports of execve failing with EIO due to the wrong path.  It is notable here that the chroot is trying /bin/bash and /bin/false rather than /usr/bin/bash (please try `chroot /mnt/sdb1 /usr/bin/bash`).  There could be a problem with the /bin symlink on sdb1 and/or chroot/exec could be failing to follow the symlink properly.

No. it fails even with this:

root@rescue-pro:~# cd /mnt/sdb1/
root@rescue-pro:/mnt/sdb1# mount -t proc /proc proc/
root@rescue-pro:/mnt/sdb1# mount --rbind /sys sys/
root@rescue-pro:/mnt/sdb1# mount --rbind /dev dev/
root@rescue-pro:/mnt/sdb1# chroot /mnt/sdb1 /usr/bin/bash
chroot: impossibile eseguire il comando "/usr/bin/bash": Input/output error

Hope i help you!
and
VICE.VERSA

Offline

#19 2019-01-06 15:13:19

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 27,288
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

OK, I'm pretty much stumped.  But so far we have been really focused on all the common or most like sources of issues.  I don't think we've really accounted for where this all started: something is wrong on the sdb1 system.

So lets gather some data on that.  First, can you fsck sdb1?  After that, mount it and check the output of `ldd /mnt/sdb1/usr/bin/bash`.  Depending on the results, you may need to download some arch package (.pkg.tar.xz files) and "manually" install (unpack them) into sdb1 to replace broken components.

Last edited by Trilby (2019-01-06 15:14:12)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#20 2019-01-06 15:16:07

seth
Member
Registered: 2012-09-03
Posts: 31,978

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

magiusche wrote:

done...and fails.

after reboot i've ran:

strace -f chroot / /bin/bash &> /mnt/sdb1/home/arch/chroot_stracemore.txt

here the results: https://pastebin.com/KnREutmN

execve("/bin/bash", ["/bin/bash"], [/* 14 vars */]) = 0
…
open("/dev/tty", O_RDWR|O_NONBLOCK)     = 3
…
ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(2, TCGETS, 0x7ffd73490af0)        = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(-1, TIOCGPGRP, 0x7ffd73490b5c)    = -1 EBADF (Bad file descriptor)

Not the same error.

To be very sure, can you re-confirm

calling /mnt/sdb1/bin/bash works without errors!

?

Offline

#21 2019-01-06 15:21:11

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 27,288
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Oops, I missed that there was a new situation in the strace.  It would be good to confirm that calling /mnt/.../bash works, but if it does it may support my suspicion as running it directly would use the libs on the debian system rather than the arch system (I think).

Come to think of it, I'm not sure if we'll get what I'm looking for from `ldd /mnt/.../bash` as the debian-based ldd may resolve the libraries from it's own.  But at very least we can check the state of each of the needed libs on the sdb1 system (perhaps starting with `file /mnt/sdb1/usr/lib/...`).

Last edited by Trilby (2019-01-06 15:22:47)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#22 2019-01-06 15:21:49

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Trilby wrote:

OK, I'm pretty much stumped.  But so far we have been really focused on all the common or most like sources of issues.  I don't think we've really accounted for where this all started: something is wrong on the sdb1 system.

So lets gather some data on that.  First, can you fsck sdb1?  After that, mount it and check the output of `ldd /mnt/sdb1/usr/bin/bash`.  Depending on the results, you may need to download some arch package (.pkg.tar.xz files) and "manually" install (unpack them) into sdb1 to replace broken components.

Ok i will.
Now i have to go to repair my Desktop (CoolMaster crashed) and i need to reassemble it all.
After that.. maybe tonight i will start to do what you suggested.

Btw i've already performed a fsck sdb1 (at very beginning of my voyage) and it found some trouble but them should be full repaired.

I guess too i need to donwload some package manually and reinstall it (but i need some assisteance there).

Thx again for all and see you later.


Hope i help you!
and
VICE.VERSA

Offline

#23 2019-01-06 19:20:34

magiusche
Member
Registered: 2008-12-05
Posts: 37

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

Trilby wrote:

Oops, I missed that there was a new situation in the strace.  It would be good to confirm that calling /mnt/.../bash works, but if it does it may support my suspicion as running it directly would use the libs on the debian system rather than the arch system (I think).

Come to think of it, I'm not sure if we'll get what I'm looking for from `ldd /mnt/.../bash` as the debian-based ldd may resolve the libraries from it's own.  But at very least we can check the state of each of the needed libs on the sdb1 system (perhaps starting with `file /mnt/sdb1/usr/lib/...`).

ok, this is the output of

file /mnt/sdb1/usr/lib/* &> /mnt/sdb1/home/arch/chroot_stracemore.txt

https://pastebin.com/s2PtQwzg

should i try to start fresh arch install via arch install scripts overwrite the esiting one in sdb1? (i need to save etc and mariadb file)
I dunno if this is possibile starting from debian!

Last edited by magiusche (2019-01-06 19:22:03)


Hope i help you!
and
VICE.VERSA

Offline

#24 2019-01-06 19:27:16

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 27,288
Website

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

I meant the libs linked to by bash, not just all of them, but I can find them in the list and it confirms my suspicion:

/mnt/sdb1/usr/lib/libdl-2.28.so:                            empty
/mnt/sdb1/usr/lib/libc-2.28.so:                             empty
/mnt/sdb1/usr/lib/ld-2.28.so:                               empty

Many of your essential libs are totally borked.  Ncurses being "ASCII text" is particularly odd.

You'll need to manually copy over the appropriate libs for *at least* these libs before you can even chroot to use bash.  But from there you will not be able to do much else.  I didn't check all the libs pacman links to, for example.

EDIT: I removed libncurses from the list, apparently `file` reads it as ASCII text on my system too, so that may be fine (edit 3: yup, it just reads in libncursesw).  So glibc seems borked at least.

EDIT 2: your borked packages seem to be limted to glibc, libunwind, python, and slang.  Only glibc is critical to get back into your system and run pacman.  So manually download and extract the glibc package and copy it's contents over to the appropriate locations in sdb1.  Then you should be able to chroot and run pacman ensuring to reinstall the other three broken packages.

Last edited by Trilby (2019-01-06 19:35:45)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#25 2019-01-06 19:49:03

loqs
Member
Registered: 2014-03-06
Posts: 15,358

Re: [Solved] chroot: failed to run command "/bin/bash": Input/output error

eschwartz also provides a static version of pacman precompiled at https://pkgbuild.com/~eschwartz/repo/x86_64-extracted/

Offline

Board footer

Powered by FluxBB