[Solved] Upgrade just crashed my system

loqs · 2016-10-05 18:19:55

Fixed wrote:

loqs wrote:
Strange blkid seems to show an entry that matches the UUID that can not be found by init.
Can I change the UUID on the stage of rEFInd?

Yes but to what? If the UUID is correct then it should not be changed.
One reason text is preferred over screen shots is to avoid having to interpret simlar looking characters such as '0' and 'O'
So please post the output requested as text

Fixed · 2016-10-05 19:36:10

loqs wrote:

Few sanity checks
From the live/install media what is the output of
# blkid

ptpb.pw/lGCf

loqs wrote:

From within the chroot
$ pacman -Qkk linux

https://ptpb.pw/axvd

loqs wrote:

There should be no mention of /boot/vmlinuz-linux in the output

There is none.

loqs wrote:

Also please post the entries from /var/log/pacman.log for your last update.

ptpb.pw/juUt

seth · 2016-10-05 22:19:07

Bug or broken installation?
Tried to update/re-install the kernel (and possible related packages with extra modules) from the chrooted session?
Downgrade the kernel?

Trilby · 2016-10-05 22:35:35

None of the refind documentation on our wiki shows using UUIDs, only PARTUUIDs which are different. I don't know if UUIDs are not supported or problematic with refind, but this does give an option to try: use root=PARTUUID=d2e9f07e-fa87-4c03-9cb0-3aa6c9050853

jasonwryan · 2016-10-05 22:41:12

refind supports UUIDs: http://www.rodsbooks.com/efi-bootloaders/refind.html

loqs · 2016-10-05 22:45:44

Should not the root paramter be passed unaltered to the initrd anyway?

ERROR: device 'UUID=2e996be4-41bf-4ecd-8bca-c0e4c5067971' not found. Skipping fsck.
ERROR: unable to find root device  'UUID=2e996be4-41bf-4ecd-8bca-c0e4c5067971' .

Are those errors not coming from the initrd not the botoloader?

Fixed wrote:

System is frozen at the mooment, doesn't accept any entry.

If Fixed could access the rescue shell would ask for blkid / fsck / mount from that but the previous quote prevents it but again would seem to point to an initrd issue.

iodine · 2016-10-06 06:17:15

It's weird, because my kernel upgrade (4.7.5 -> 4.7.6) also led to a crash yesterday, but in my case vmlinuz-linux was missing afterwards. Your kernel seems to be there, but I would do a kernel downgrade/re-install as suggested here earlier.

Fixed · 2016-10-06 19:55:06

iodine wrote:

It's weird, because my kernel upgrade (4.7.5 -> 4.7.6) also led to a crash yesterday, but in my case vmlinuz-linux was missing afterwards. Your kernel seems to be there, but I would do a kernel downgrade/re-install as suggested here earlier.

I downgraded kernel to 4.7.5, I've got:

::running hook [keymap]
::loading keymap...done.
ERROR: device 'UUID=2e996be4-41bf-4ecd-8bca-c0e4c5067971' not found. Skipping fsck.
ERRORr: Unable to find root device 'UUID=2e996be4-41bf-4ecd-8bca-c0e4c5067971'.
You are being dropped to a recovery shell
Type 'exit' to try and continue booting
sh: can't access tty: job control turned off
[rootfs ]#

Shell is frozen now.

seth · 2016-10-06 20:38:49

See https://bbs.archlinux.org/viewtopic.php?id=165696
A rEFInd issue?

=> Try https://bbs.archlinux.org/viewtopic.php … 4#p1659054 or simply "root=/dev/sda4"

seth · 2016-10-06 20:40:46

Oh, and please post your entire refind-linux.conf somewhere (sorry if I just missed it)

loqs · 2016-10-06 20:46:19

[2016-10-04 10:01] [PACMAN] Running 'pacman -Syu'
[2016-10-04 10:01] [PACMAN] synchronizing package lists
[2016-10-04 10:01] [PACMAN] starting full system upgrade
[2016-10-04 10:02] [ALPM] transaction started
[2016-10-04 10:02] [ALPM] upgraded qt4 (4.8.7-9 -> 4.8.7-10)
[2016-10-04 10:02] [ALPM] transaction completed
[2016-10-04 10:02] [ALPM] running 'gtk-update-icon-cache.hook'...
[2016-10-04 10:02] [ALPM] running 'update-desktop-database.hook'...

As the linux package was not part of the full upgarde it was already up to date and I assume was was version 4.7.6-1.
Did you ever successfully boot with this kernel? If not what was the last kernel that did successfully boot?
Would suggest as the downgrade was unsuccessful installing the linux-lts package alongside the linux package to see if that boots.
Depending on how the bootloader is configured you may have to manually create an entry for it.
Also you seem to be confusing quote tags and code tags

philo · 2016-10-07 09:48:40

Referring to post #33.

I recently had the same message of errors. I went into arch-chroot and run:

sudo arch-chroot /mnt/arch  pacman -S --force mkinitcpio systemd linux linux-lts

and was back on my feet . Not sure if this applicable here.

Last edited by philo (2016-10-07 09:51:08)

Fixed · 2016-10-07 10:21:44

Hello Everyone,

And thanks for involvement into investigation.

Maybe I've found some traces of what caused this screw-up.

I used this installation for 3 years on MacBook Air without any major glitch.

The troubling update appeared for sure on Oct. 4th, since I update daily. On October 4th I hit 'pacman -Syu', which passed the phase of downloading packages OK, but completely crashed the system during installation. The only thing I recall, it was connected with upgrading some libraries mentioning systemd.

Today, I tried to downgrade several of them, and found that corresponding packages are missing or corrupt.

For example, that's what I see after chroot;

[root#archiso /]# pacman -U /var/cache/pacman/pkg/systemd-231-3-x86_64.pkg.tar.xz

https://ptpb.pw/RlNe

One thing which confuses me, is the date of the package on the disk: Oct.2nd, but crush happened on Oct. 4th.

Attempt to install earlier version of the package systemd-231-1 reports:

https://ptpb.pw/415b

Same story of corrupted packages (the latest versions):

systemd-sysvcompat-231-3-x86-64.pkg.tar.xz
libsystemd-231-3-x86_64.pkg.tar.xz

The package

lib32-systemd

is not corrupted though.

Edit1: In old "good" times of systemv, I was able to go to /var/log/messages and screen the log of console. This could help to identify which specific package update trashed the system.

Last edited by Fixed (2016-10-07 10:25:34)

Fixed · 2016-10-07 10:35:56

philo wrote:

Referring to post #33.
I recently had the same message of errors. I went into arch-chroot and run:
sudo arch-chroot /mnt/arch  pacman -S --force mkinitcpio systemd linux linux-lts
and was back on my feet . Not sure if this applicable here.

After using your suggestion I have the same:

::running hook [keymap]
::loading keymap...done.
ERROR: device 'UUID=2e996be4-41bf-4ecd-8bca-c0e4c5067971' not found. Skipping fsck.
ERRORr: Unable to find root device 'UUID=2e996be4-41bf-4ecd-8bca-c0e4c5067971'.
You are being dropped to a recovery shell
     Type 'exit' to try and continue booting
sh: can't access tty: job control turned off
[rootfs ]#

Shell is frozen now

philo · 2016-10-07 11:59:49

Do you have a post-transaction hook for systemd update? If yes, remove it first while in arch-chroot.

EDIT- That hook would be in: /usr/share/libalpm/hooks/

Last edited by philo (2016-10-07 12:30:32)

loqs · 2016-10-07 12:28:59

Tips_and_tricks#Reinstalling_all_packages

# pacman -Qnq | pacman -S -

That will obviously fail due to the existing files in the filesystem but it may be able to be used to create a list of packages that have files already in the filesystem

# pacman -Qnq | pacman -S - | grep "exists in filesystem" | cut -d : -f 1 | unique damagedpackages

I can not test this as I do not have any packages in this condition.
Would also recommend compiling a list of packages that pacman detects as having been modified

$ pacman -Qkk

You will need to use your judgment about which modifications are due to corruption and which are not.
Both of the above methods may also fail due to corruption of the pacman database files.

Fixed · 2016-10-07 12:47:29

philo wrote:

Do you have a post-transaction hook for systemd update? If yes, remove it first while in arch-chroot.
EDIT- That hook would be in:
/usr/share/libalpm/hooks/

I don't see any post-transaction hooks in

/usr/share/libalpm/hooks/

:

look at the hooks in this directory:

https:///ptpb.pw/j9IA

Last edited by Fixed (2016-10-07 13:02:53)

philo · 2016-10-07 13:47:02

You do not have a post-transaction hook for systemd upgrades unless you created one yourself.

In your post #38, "https://ptpb.pw/415b" shows that you need the argument "--force"" inserted in the command suggested in my post #37.

loqs · 2016-10-07 13:54:14

philo wrote:

In your post #38, "https://ptpb.pw/415b" shows that you need the argument "--force"" inserted in the command suggested in my post #37.

You are suggesting using the --force option without a warning of the dangers that entails, on a a system where the state of the pacman database is unknown, where the list of packages that contain untracked files is unknown and the state of the filesystem itself is unknown and the root cause of the issue is unknown.
Without even checking if Fixed has backups first in case your suggestion makes the situation worse.
Edit:
Leaving the previous reasons aside at the very least --force should not be used with the linux package as Fixed report being able to successfully downgrade that in post #33 and not with linux-lts or mkinitcpio packages as there is nothing in this thread that I can find indicating pacman can not operate on those packages without the --force flag.
Edit2:

philo wrote:

Do you have a post-transaction hook for systemd update? If yes, remove it first while in arch-chroot.

How could it be a post-transaction hook

Errors occurred, no packages were upgraded.

Not

....
[ALPM] transaction started
....
[ALPM] transaction completed

Only after which would post-transaction hooks be run.

Last edited by loqs (2016-10-07 14:29:07)

Fixed · 2016-10-08 08:49:46

Everyone,

I really appreciate your involvement and help with the problem.

I can spend days, but not weeks resolving the issue, since Arch is the tool in my research, and the problem is obviously beyond standard methods of system recovery.

I am sorry for not able to continue to stay in this investigation, as am in the beginning of one of the biggest project of my life.

Yesterday I saved my \home directory and reformatted the disk, and now in another pain of trying to set up dual boot of new OSX "Sierra" and Arch.

Fixed

philo · 2016-10-08 12:57:28

Fixed wrote:

Yesterday I saved my /home directory and reformatted the disk, and now in another pain of trying to set up dual boot of new OSX "Sierra" and Arch.Fixed

So this thread can be considered as closed.

Last edited by philo (2016-10-08 12:59:07)

ewaller · 2016-10-08 15:15:46

Fixed,
Please do not used the [CLOSED] tag in your title. Closed implies that a moderator has closed and locked the thread.
https://wiki.archlinux.org/index.php/Co … ow_to_post (second bullet from end)
Thanks.

philo · 2016-10-09 18:33:12

So it was caused by a bug in systemd 231-1 (not 231-3)?
https://bugs.archlinux.org/task/51275

Last edited by philo (2016-10-09 18:46:54)

Arch Linux

#26 2016-10-05 18:19:55

Re: [Solved] Upgrade just crashed my system

#27 2016-10-05 19:36:10

Re: [Solved] Upgrade just crashed my system

#28 2016-10-05 22:19:07

Re: [Solved] Upgrade just crashed my system

#29 2016-10-05 22:35:35

Re: [Solved] Upgrade just crashed my system

#30 2016-10-05 22:41:12

Re: [Solved] Upgrade just crashed my system

#31 2016-10-05 22:45:44

Re: [Solved] Upgrade just crashed my system

#32 2016-10-06 06:17:15

Re: [Solved] Upgrade just crashed my system

#33 2016-10-06 19:55:06

Re: [Solved] Upgrade just crashed my system

#34 2016-10-06 20:38:49

Re: [Solved] Upgrade just crashed my system

#35 2016-10-06 20:40:46

Re: [Solved] Upgrade just crashed my system

#36 2016-10-06 20:46:19

Re: [Solved] Upgrade just crashed my system

#37 2016-10-07 09:48:40

Re: [Solved] Upgrade just crashed my system

#38 2016-10-07 10:21:44

Re: [Solved] Upgrade just crashed my system

#39 2016-10-07 10:35:56

Re: [Solved] Upgrade just crashed my system

#40 2016-10-07 11:59:49

Re: [Solved] Upgrade just crashed my system

#41 2016-10-07 12:28:59

Re: [Solved] Upgrade just crashed my system

#42 2016-10-07 12:47:29

Re: [Solved] Upgrade just crashed my system

#43 2016-10-07 13:47:02

Re: [Solved] Upgrade just crashed my system

#44 2016-10-07 13:54:14

Re: [Solved] Upgrade just crashed my system

#45 2016-10-08 08:49:46

Re: [Solved] Upgrade just crashed my system

#46 2016-10-08 12:57:28

Re: [Solved] Upgrade just crashed my system

#47 2016-10-08 15:15:46

Re: [Solved] Upgrade just crashed my system

#48 2016-10-09 18:33:12

Re: [Solved] Upgrade just crashed my system

Board footer