Arch Linux

FixedWing

Member

Registered: 2015-03-14

Posts: 7

[Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

Greetings.

I am getting the following on boot:

***************************************
Starting Version 218

A password is required to access the MyStorage volume:
Enter passphrase for /dev/sda3
/dev/mapper/MyStorage-rootvol: clean, 210151/983040 files, 2284613/3932160 block
[ 78.084720] systemd-fsck [280]: fsck: /sbin/fsck.ext4: execute failed: Exec format error
[ 78.085215] systemd-fsck [287]: fsck: /sbin/fsck.ext2: execute failed: Exec format error
****************************************

I then end up at a login prompt but if I try to login, I get “Login incorrect”. Sometimes Getty will stop and restart on tty1. Then I get returned to the login prompt.

This came about after upgrading with Pacman (which included “upgraded e2fsprogs (1.42.12-1 -> 1.42.12-2)”) a few days ago. Pacman completed successfully but on reboot the system froze forcing a hard reset.

I've booted to a USB and run fsck on the boot partition (the only ext2 partition). Ditto on the root and home volumes. All fine. I've also mounted all three and can access the data.

I would have thought it was something to do with the e2fs upgrade but it obviously scanned the root volume fine and I haven't been able to find any similar reports online.

I've searched online for ideas and I've also searched for logs which might give me some indication of what the cause is but at this point, I've reached my limits.

I'd just nuke the data and start again but I really want to understand what happened here.

Any thoughts on what caused this or suggestions on how to proceed?

Thank you.

Stephen

Last edited by FixedWing (2015-03-16 01:40:20)

Head_on_a_Stick

Member

From: The Wirral

Registered: 2014-02-20

Posts: 9,003

Website

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

I would have thought it was something to do with the e2fs upgrade

Have you tried using the Arch live ISO to chroot into your system and downgrade that package?
https://wiki.archlinux.org/index.php/Do … g_packages

Last edited by Head_on_a_Stick (2015-03-15 12:39:43)

---

Jin, Jîyan, Azadî

FixedWing

Member

Registered: 2015-03-14

Posts: 7

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

Have you tried using the Arch live ISO to chroot into your system and downgrade that package?

I had not. So this morning, I did.

I get an error message:

pacman: error while loading shared libraries: /usr/lib/libcom_err.so.2: file too short

libcom_err.so.2 actually points to libcom_err.so.2.1 and 2.1 has 0 bytes and is dated 2 March (as are the links to it). When I look on another machine, the file on that x64 machine is 14,648 bytes with the same date.

But am I correct that this in and of itself would not cause my boot problems and is likely symptomatic other bigger problems?

Thank you Head.

Stephen

Last edited by FixedWing (2015-03-15 14:01:19)

Head_on_a_Stick

Member

From: The Wirral

Registered: 2014-02-20

Posts: 9,003

Website

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

pacman: error while loading shared libraries: /usr/lib/libcom_err.so.2: file too short

That file is part of the e2fsprogs package so just installing the old version should re-create it:

# pacman -U /var/cache/pacman/pkg/e2fsprogs-<old version number>.pkg.tar.xz

Perhaps there is corruption in your filesystem.

---

Jin, Jîyan, Azadî

FixedWing

Member

Registered: 2015-03-14

Posts: 7

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

That file is part of the e2fsprogs package so just installing the old version should re-create it:

# pacman -U /var/cache/pacman/pkg/e2fsprogs-<old version number>.pkg.tar.xz

Perhaps there is corruption in your filesystem.

Well, it appears that Pacman downloaded an e2fsprogs package of 0 bytes in size and then proceeded to install it anyway, overwriting every file in the system with an "updated" file 0 bytes in length. And then it logged that it had successfully upgraded the package. Wow. Just wow.

The error message I previously quoted was produced when I ran Pacman trying to do as you suggest. The problem is that Pacman also needs libcom_err.so.2.1 and so is refusing to run.

I tried to use Pacstrap to download e2fsprogs and reinstall the package. That runs but also refuses to overwrite the files because they already exist.

So how can I force a reinstall of e2fsprogs?

Stephen

Last edited by FixedWing (2015-03-16 01:00:25)

FixedWing

Member

Registered: 2015-03-14

Posts: 7

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

Well, I was able to manually reinstall e2fsprogs and that got pacman running again but I also got a bunch or errors about other packages. I was then able to boot the system and log in. But when I tried to reboot, it froze in the same way as previously. There is obviously a lot more that's wrong with this system now.

Anyway, thanx for the help!

Stephen

Head_on_a_Stick

Member

From: The Wirral

Registered: 2014-02-20

Posts: 9,003

Website

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

The problem is that Pacman also needs libcom_err.so.2.1 and so is refusing to run.

https://wiki.archlinux.org/index.php/Pa … ond_repair

However, it may be simplest to just re-install in your case -- it depends whether you want to use the troubleshooting & repairing as a learning process or if you just want your system up & running again ASAP...

---

Jin, Jîyan, Azadî

FixedWing

Member

Registered: 2015-03-14

Posts: 7

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

https://wiki.archlinux.org/index.php/Pa … ond_repair

However, it may be simplest to just re-install in your case -- it depends whether you want to use the troubleshooting & repairing as a learning process or if you just want your system up & running again ASAP...

All fixed and working just like nothing happened.

I did use the advice at the referred link plus a few others on archlinux.org and elsewhere. Yes, an absolutely wonderful learning experience!

I manually reinstalled e2fsprogs. That got Pacman working again and I was able to boot into the system. Then I used Pacman to reinstall e2fsprogs properly plus the other seven packages which were also installed during the same Pacman session despite their being corrupted.

What I really don't get is how Pacman could accept a package with 0 bytes and install it? How could such a package possible pass the security check? When I reinstalled the packages, Pacman of course refused to install the corrupt packages in the cache and deleted them. So why didn't that happen initially? I can only think that a corrupt file in that process terminated prematurely and that Pacman wasn't robust enough to detect this so simply continued on, now skipping the scans and installing the corrupt packages. So just to be sure it wasn't a corrupt file in Pacman itself, I also forced a reinstall of that package as well. I've upgraded packages since without issue so I have to assume that whatever the issue was is now gone.

Anyway, thanx for the help!

Stephen

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

What I really don't get is how Pacman could accept a package with 0 bytes and install it? How could such a package possible pass the security check?

It won't, the problem is in disk caching. Simply put, the files were OK as far as pacman was concerned and were installed just fine, but in the underlying system, they only got as far as the buffer and never written to the actual disk.

This is why you use the famous "Safely eject hardware" option in Windows. It makes sure the buffers are flushed.

FixedWing

Member

Registered: 2015-03-14

Posts: 7

Re: [Solved -sorta] systemd-fsck []: fsck: /sbin/fsck.ext4: execute failed

It won't, the problem is in disk caching. Simply put, the files were OK as far as pacman was concerned and were installed just fine, but in the underlying system, they only got as far as the buffer and never written to the actual disk.

Ah, that makes perfect sense! It would also explain why I didn't see the sort of errors I was expecting when I scanned the volumes with fsck.

Thank you.

Stephen

Last edited by FixedWing (2015-03-16 14:59:31)