[Solved] Commands not doing anything in chroot

honey_badger3 · 2020-06-11 04:53:34

I recently upgraded my install on my laptop and was unable to boot because I was getting a premature end of file error when grub was loading the kernel. I have fixed this issue before in the past by booting my live USB and reinstalling the kernel in a chrooted environment.
The issue I am having is that entering some commands doesn't do anything, It just returns to prompt for the next command. For example, the basic stuff like cd and ls works, but when trying to run any pacman commands or opening files in any text editor (vim/nano), just doesn't do anything. I have tried both arch-chroot and manually chrooting by mounting dev and proc but the results are the same. Any idea what could be broken and how to fix it?

Last edited by honey_badger3 (2020-06-15 15:19:55)

Xyne · 2020-06-11 05:35:30

Is there any space left on the disk?
Are you seeing any error messages with dmesg or journalctl?
If there are any hard disk errors, run a test with smartctl.

To exclude a chroot error, try to install packages with pacstrap or open files on the mounted system from outside the chroot.

eschwartz · 2020-06-11 13:59:07

honey_badger3 wrote:

The issue I am having is that entering some commands doesn't do anything, It just returns to prompt for the next command. For example, the basic stuff like cd and ls works, but when trying to run any pacman commands or opening files in any text editor (vim/nano), just doesn't do anything.

Are you saying you literally type in a command, e.g.

$ ls /usr
bin/  include/  lib/  lib32/  lib64@  local/  sbin@  share/  src/
$ pacman -Syu
$ echo $?
0

and some commands work as expected, while others produce no output and no error messages and simply, silently, fail to do anything?

That echo $? command will let you see if the previous command failed with an unusual status. Normally if a command simply fails to exist, it should do something like this:

$ fdsfsd
bash: fdsfsd: command not found
$ echo $?
127

If you have a custom command_not_found_handle function in bash, it will override the usual error message and provide its own error handling and messaging. But the return code should still be 127 for commands which bash determined does not exist.

Also try "type -a pacman" to see which pacman program is being found, if any.

If there were in fact disk errors, I would expect that to manifest as the command reporting an error, not as it silently doing nothing. Even if bash did not find the command because of disk errors, I would expect bash to report an error. Completely silent errors sounds very weird.

honey_badger3 · 2020-06-11 17:17:16

Thanks for the quick responses.

@eschwartz, what you described is exactly what is happening. The exit status for those commands that are not giving any output comes up as 0. They don't return any errors and they produce no output.
I ran "type -a pacman" and this was the output:

pacman is /usr/bin/pacman.

@Xyne, I do have some disc space left, not sure how much exactly but more than 5GB on the 32GB partition I have my system installed on.
I ran smartctl -H on the disc and the health status returned as OK, so I am thinking I don't have any disc errors.
I am able to open files from outside the chrooted environment so I can modify them if need be.

I should also clarify that I am using the LTS kernel. I tried using the pacstrap command as you suggested and here are the commands I ran.

pacstrap /mnt linux-lts

I got a ton of errors along the lines of "File /usr/lib/modules/5.4.45-1-lts<module-name> already exists in filesystem" and the command failed. Sorry if that's not the exact error, I forgot to capture it exactly.
I then tried to force the install with:

pacstrap /mnt linux-lts --overwrite '*'

This seemed to work and the installation proceeded. However, on running post transaction hooks to build the new initramfs, here are the errors I got:

==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default  
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img  
==> ERROR: inavlid kernel specified: '/boot/vmlinuz-linux-lts'  
error: command failed to execute correctly

Hopefully I typed that correctly. I did not capture the output of my commands but instead I am manuallyt typing what I'm seeing on the screen. Let me know if there are other things I can check or logs you'd like to see.
Sadly, checking dmesg and journalctl doesn't work because those commands aren't doing anything either.

Thanks!

seth · 2020-06-11 20:09:06

I ran smartctl -H on the disc and the health status returned as OK

Yeah, no - that value is worthless.
Post the output of "smartctl -a" to see whether something fishy is going on with that drive.

not sure how much exactly

Check "df -h"?

Also: Is your /boot a different partition or do you boot from the root partition?

honey_badger3 · 2020-06-11 23:25:25

@seth, thanks for clarifying. I ran "df -h" and it shows that I have 6.2G available.
I also ran "smartctl -a" and I get and I get an error that says
Error Counter logging not supported
Device does not support Self Test logging.

And yes, I boot from the root partition. I created a swap partition and the rest of the drive the root partition.

I am thinking of using pacstrap to install linux instead of linux-lts. Is that a good idea or will I create some conflict with lts modules?

honey_badger3 · 2020-06-11 23:48:40

So I made some progress though the main issue still exists. I backed up the pacman cache on a separate flash drive and then cleared it with "rm -rf /mnt/var/lib/pacman/local/*" and then installed linux-lts with pacstrap and the install proceeded successfully.
I managed to boot into my main system but as before, most commands don't return any output or errors. I don't have a login manager so when my system starts I log in at tty and then run startx to launch my window manager (i3). As with other commands, running startx doesn't do anything either.
I think therefore that the problem is beyond chroot as you guys suggested. It's possible there might be drive issues because I'm running on a 7 yr old laptop from 2013 but that's the only fact I'm basing that hunch on. Are there other symptoms I could check to confirm this is the issue?
Any further suggestions on finding out what they may be?

Last edited by honey_badger3 (2020-06-11 23:53:41)

eschwartz · 2020-06-12 02:05:51

honey_badger3 wrote:

I should also clarify that I am using the LTS kernel. I tried using the pacstrap command as you suggested and here are the commands I ran.
pacstrap /mnt linux-lts
I got a ton of errors along the lines of "File /usr/lib/modules/5.4.45-1-lts<module-name> already exists in filesystem" and the command failed. Sorry if that's not the exact error, I forgot to capture it exactly.

This seems rather odd, pacstrap will internally use the chroot's metadata so there should be no conflict. This implies write errors of some sort, since pacman's database doesn't have info on the files which are installed. Following that trail of thought...

Is it possible some of your binaries are actually zero-byte files? From the install ISO, what happens if you execute

pacman --sysroot /path/to/mount -Qkk

find /path/to/mount/usr/bin -type f -empty

What happens if you try to use the "file" command on some of these files? It might not work from within the broken system, but you did say the ls command works, so does ls -l report a suspicious file size such as "0"? Other possibilities are "cat /path/to/file", or if you need something which relies on nothing but pure shell builtins, you can use:

echo "$(</path/to/file)"

(because $(</path/to/file) is a shell builtin replacement for $(cat /path/to/file), and then you pass the cat'ed contents as an argument to the echo builtin.)

One possibility is that these non-working programs are empty files, in which case the shell tries to treat them as scripts without a shebang, executes them with /bin/sh, and successfully runs zero commands. This was an old implementation of the /bin/true command, in fact albeit somewhat slow because it forked a whole shell interpreter: http://trillian.mit.edu/~jc/humor/ATT_C … _true.html

honey_badger3 · 2020-06-12 04:40:45

@eschwartz, I think you're onto something. I ran

pacman --sysroot /path/to/mount -Qkk

and I put the output in this pastebin.
I also ran the find command to find empty files and I got quite a long list. I checked some of them, including pacman with the file command and got

/mnt/usr/bin/pacman: empty

Is that something fixable by reinstalling my entire package list with pacstrap from the iso, or is it more complicated than that?

eschwartz · 2020-06-12 04:47:59

Interestingly, pacman isn't reported on that list. What does "pacman --sysroot /mnt -Qi pacman" report? Is it pretending to be uninstalled???

This might go some way towards explaining why everything is broken. Reinstalling all broken packages with zero-byte files should do the trick, but given some of them don't even seem to be installed as far as pacman knows, retrieving the package list might be... complicated. There are tools to parse it from /var/log/pacman.log...

You'll want to investigate how this happened to begin with, too. One possibility is that all these packages were upgraded but you had a forced poweroff in mid transaction, interrupting file writes.

honey_badger3 · 2020-06-12 13:38:35

So that command returned "error: package 'pacman' not found".
I think you're right, now that you mentioned it I recall what I did after updating. I ran pacman -Syu and allowed the command to complete. After it was done, I immediately ran "systemctl reboot" and it started the reboot process and appeared to have turned off but it really wasn't. The screen was blank/black but I could see the backlight was still on. It stayed like that for about a minute or two and I got impatient and I forced a shutdown by holding the power button. Does the file writing still continue even after the command has completed? If so, definitely a self inflicted wound on my part by forcing the shutdown.
I'll give a shot at reinstalling the packages. Any ideas how I can leverage regular expressions to parse the logs and generate a list of the packages so I can reinstall them? Or any other tools I can use for that?

Last edited by honey_badger3 (2020-06-12 13:39:39)

loqs · 2020-06-12 13:49:09

Changes to the filesystem are not guaranteed to be written to the underlying storage until sync or equivalent has been called or the filesystem has been unmounted.

eschwartz · 2020-06-12 16:19:37

$ cat /etc/pacman.d/hooks/zz-sync.hook 
[Trigger]
Operation = Install
Operation = Upgrade
Operation = Remove
Type = Package
Target = *

[Action]
Description = Force syncing all files to disk
Depends = coreutils
When = PostTransaction
Exec = /usr/bin/sync

I find this to be a useful safety net.

Anyway, you can use the following script from pacman-contrib:

paclog-pkglist /var/log/pacman.log

which in theory should list every package and pkgver which is currently installed, according to the upgrade logs. If this does not match, then you need to fix something.

diff -u <( pacman --sysroot /mnt -Q | sort) <(paclog-pkglist /mnt/var/log/pacman.log | sort)

should give you an overview of what is inconsistent here.

Xyne · 2020-06-12 19:32:06

eschwartz wrote:

$ cat /etc/pacman.d/hooks/zz-sync.hook 
[Trigger]
Operation = Install
Operation = Upgrade
Operation = Remove
Type = Package
Target = *

[Action]
Description = Force syncing all files to disk
Depends = coreutils
When = PostTransaction
Exec = /usr/bin/sync

I find this to be a useful safety net.

Good idea. I hadn't thought about it before, but I would have assumed that pacman sync'd automatically after making changes to the disk.

honey_badger3 · 2020-06-14 22:37:38

Thanks @eschwartz for your suggestions. I looked for that script and it led me to this page. I followed the steps there and got most of the core systems working now. I just have some wifi problems but I think I can figure those out later since at least I can use my wired connection. I'll consider this one solved!

Last edited by honey_badger3 (2020-06-14 22:41:26)

eschwartz · 2020-06-14 22:57:26

Yay.

Please edit the first post in the thread, and mark it as solved by prepending "[Solved]" to the thread title.

Arch Linux

#1 2020-06-11 04:53:34

[Solved] Commands not doing anything in chroot

#2 2020-06-11 05:35:30

Re: [Solved] Commands not doing anything in chroot

#3 2020-06-11 13:59:07

Re: [Solved] Commands not doing anything in chroot

#4 2020-06-11 17:17:16

Re: [Solved] Commands not doing anything in chroot

#5 2020-06-11 20:09:06

Re: [Solved] Commands not doing anything in chroot

#6 2020-06-11 23:25:25

Re: [Solved] Commands not doing anything in chroot

#7 2020-06-11 23:48:40

Re: [Solved] Commands not doing anything in chroot

#8 2020-06-12 02:05:51

Re: [Solved] Commands not doing anything in chroot

#9 2020-06-12 04:40:45

Re: [Solved] Commands not doing anything in chroot

#10 2020-06-12 04:47:59

Re: [Solved] Commands not doing anything in chroot

#11 2020-06-12 13:38:35

Re: [Solved] Commands not doing anything in chroot

#12 2020-06-12 13:49:09

Re: [Solved] Commands not doing anything in chroot

#13 2020-06-12 16:19:37

Re: [Solved] Commands not doing anything in chroot

#14 2020-06-12 19:32:06

Re: [Solved] Commands not doing anything in chroot

#15 2020-06-14 22:37:38

Re: [Solved] Commands not doing anything in chroot

#16 2020-06-14 22:57:26

Re: [Solved] Commands not doing anything in chroot

Board footer