[SOLVED] I broke OpenSSL and then the whole system.

nphuracm · 2022-11-12 01:52:04

I'm posting this to Newbie Corner because I really think this is a stupid mistake.

So my Arch system hasn't been updated for a while now due to insufficient disk space. Recently I was dealing with this 3DS emulator called Citra and while it complained about OpenSSL version, like some 'version OPENSSL_3_0_0' not found $#!+, I updated OpenSSL only.

And it borked everything.
Essential commands from sudo to pacman stopped working, all complaining that either "libssl.so.1.1" or "version OPENSSL_1_1_0" were not found. Not even the reboot command would work and I had to reset the system. Used archiso to try fixing but no avail - just ends up throwing an error when doing some PackageKit work saying No such file or directory and lots, LOTS of module not found errors during the automated mkinitramfs (I suppose that's the name? Can't remember it quite well.)

And now when I boot up Arch, although it does go into boot process (thank god I didn't mount my ESP when in archiso), it panics halfway - boot freezes after a short period and keyboard began to flash CAPS LOCK and SCROLL LOCK.

There is a chance though, that it's a major change in OSSL itself that broke my whole system - a situation that my Arch install 'rolled' itself into some ditch. But considering the obsolescence of my packages, this case seems unlikely.

I would really, REALLY appreciate it if someone helps me out.

Last edited by nphuracm (2022-11-12 16:41:55)

jasonwryan · 2022-11-12 02:05:04

https://wiki.archlinux.org/index.php/Sy … nsupported

And god only knows why you think you should be using Packagekit from the rescue image...

nphuracm · 2022-11-12 02:47:20

jasonwryan wrote:

https://wiki.archlinux.org/index.php/Sy … nsupported
And god only knows why you think you should be using Packagekit from the rescue image...

Well it just popped up and I just think 'oh no theres an error it looks bad' so I described it here.

So, according to the wiki page, I have made a huge, HUGE mistake and that there is no fix to my system now... right? Or is there still some hope to save it?

Scimmia · 2022-11-12 02:49:15

Boot an install disk and use pacman's --root option to update the entire system.

jasonwryan · 2022-11-12 02:51:43

From a live image, do afull update using pacman --root to the mounted root.

#edit: correct option

Last edited by jasonwryan (2022-11-12 02:52:40)

Trilby · 2022-11-12 02:59:31

Note that if there is (still) a disk space limitation, the above-recommended pacman command might fail. But that's because this above-mentioned command is the solution to one problem, and it sounds like you created this problem just to avoid solving a different problem (which itself may have been a symptom of yet some other problem).

So the full solution may be an iterative process of finding the solution to the first hurdle only to reveal an error from the second. But in any case, pacman from the iso with --root is step one (I was thinking this should be --sysroot, and the man page still supports that view, but I'll defer to those above). If this still fails, come back with the full error / output.

Scimmia · 2022-11-12 03:01:32

In general, --sysroot is preferred, but I've run into some situations where it fails on broken systems, so I generally suggest --root for this.

nphuracm · 2022-11-12 03:27:42

So I did the following:

pacman -Syyu --root /mnt

and

pacman -Syyu --sysroot /mnt

The first variant comes with an insufficient space error complaining about the space in airootfs.
The second one complains '/etc/mtab' not found.

I'm considering about formatting a USB drive to some fs that makes Linux systems happy and use mount --bind so that the downloaded packages goes there without eating the hard disk or airootfs space. I also have another disk on the PC for data storage that has quite a lot of space left for this but since it's NTFS I doubt that it would play nice with pacman. You know, the colons.

Last edited by nphuracm (2022-11-12 03:33:31)

jasonwryan · 2022-11-12 04:00:14

Scimmia wrote:

In general, --sysroot is preferred, but I've run into some situations where it fails on broken systems, so I generally suggest --root for this.

Yep. I realised when I saw your post that it was more likely a safer option.

cfr · 2022-11-12 04:23:56

nphuracm wrote:

So I did the following:
pacman -Syyu --root /mnt
and
pacman -Syyu --sysroot /mnt
The first variant comes with an insufficient space error complaining about the space in airootfs.

You can use ---cachedir to specify the cache directory on the target system, but if you don't have enough disk space there, that obviously won't work.

nphuracm · 2022-11-12 08:48:18

Okay, it seems like the upgrade succeeded. I did not really pay attention to all of the pacman logs present on the screen though.
So I rebooted the system and now it turns out to be an emergency shell. It looks like that systemd broke.

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
?? root ?????
(?? Control-D ???):

I can't type anything here, and when I hit 'enter' it just gives me a '?????' message and repeats the ?? root ????? lines above.

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
?? root ?????
(?? Control-D ???):
?????

?? root ?????
(?? Control-D ???):

Pressing Ctrl-D does not help either. It just gives the following messages and the emergency mode prompt again:

Reloading system manager configuration
Starting default.target

Last edited by nphuracm (2022-11-12 08:50:03)

Trilby · 2022-11-12 13:27:18

I requested full (and implied verbatim) output:

Trilby wrote:

come back with the full error / output.

You gave vague paraphrases:

nphuracm wrote:

The first variant comes with an insufficient space error complaining about the space in airootfs.
The second one complains '/etc/mtab' not found.

And have now descended to not even paying attention to the output yourself:

nphuracm wrote:

I did not really pay attention to all of the pacman [output].

How exactly do you think anyone here could help? Don't expect anyone here to put more effort into fixing your system than you can be bothered with.

Last edited by Trilby (2022-11-12 13:28:44)

nphuracm · 2022-11-12 13:50:57

Trilby wrote:

I requested full (and implied verbatim) output:
Trilby wrote:
come back with the full error / output.
You gave vague paraphrases:
nphuracm wrote:
The first variant comes with an insufficient space error complaining about the space in airootfs.
The second one complains '/etc/mtab' not found.
And have now descended to not even paying attention to the output yourself:
nphuracm wrote:
I did not really pay attention to all of the pacman [output].
How exactly do you think anyone here could help? Don't expect anyone here to put more effort into fixing your system than you can be bothered with.

Oops, sorry.. I didn't quite read closely to your post. My deepest apologies.
And now... how should I reproduce this whole process now, may I ask? Or is it that I totally fucked this whole thing up by making this extra stupid mistake of losing outputs?

By the way... I said pacman 'succeeded' because no error messages like "error occurred, no packages were updated" or something like that, right after the last line of pacman describing post-installation operations is the shell prompt. It *seemed* to run smoothly. I do suspect warnings from the output but I never went through the output, and just rebooted excitedly in the hope of the system would work once again. Sorry for the even more vague description but these are all I could provide after making this fault.

Trilby · 2022-11-12 13:59:33

Everything that happened will be in the pacman.log. I'm pretty sure that this is stored on the target system even when running on the iso if you used the --root flag (but I have admittedly limited first-hand experience needing to do this).

So you complete pacman.log and journal logs could be useful. These should be posted to a file sharing site and linked to from here. They will be huge. That's fine. Anyone here capable of helping is more than capable of filtering text data to get what we want at any point. What we can't do is fill in the blanks of pre-filtered or absent data.

nphuracm · 2022-11-12 14:05:27

https://0x0.st/o6Vs.log

journalctl -xb comes with one of these outputs:
using chroot:

Failed to get boot id: No such file or directory

using arch-chroot:

-- No entries --

So no journal

There are messages that are timestamped to be happened today... so it's hopeful I guess.
Oh yeah, by the way there is quite a lot of garbage in my system files...

Last edited by nphuracm (2022-11-12 14:18:37)

Scimmia · 2022-11-12 14:19:07

What distro is this? You're installing a lot of package from repos that Arch doesn't provide.

nphuracm · 2022-11-12 14:27:54

Scimmia wrote:

What distro is this? You're installing a lot of package from repos that Arch doesn't provide.

It IS Arch. Just Arch. Installed step-by-step from the vanilla Archiso. I have read the rules on the forums that tells me to not post here for help if I use derived distros.
I do use a custom repo though - called archlinuxcn, for it comes with quite a lot of prebuilt AUR packages which I think would be convenient. I'm thinking about AUR also when I come across the phrase 'package from repos that Arch doesn't provide.‘
AND, I also have Tencent's (crappy) Linux QQ client on my Arch install, which package is provided on the company's website rather than Arch repos.
Not really sure if that is in the scope of 'not being 100% Arch' though.

Last edited by nphuracm (2022-11-12 14:37:35)

Scimmia · 2022-11-12 14:36:59

So *why* did you get dropped to emergency mode? There should have been an error before that that you cut out.

I see all of your dkms modules failed to build, you may have to chroot and reinstall the kernel headers to retrigger it if that's what's causing the failure.

nphuracm · 2022-11-12 14:59:27

pacman log: https://0x0.st/o6VM.log

And as for emergency mode, I have no idea by just looking at the log the system pours out (loglevel=5), and that there is no way known to me to capture the whole log to a file. But these look interesting to me, although not necessarily relevant:

[  OK  ] Found device FUJITSU_MHY2160BH EFIBOOT.
         Starting File System Check on /dev/disk/by-uuid/[REDACTED]...
[  OK  ] Finished File System Check on /dev/disk/by-uuid/[REDACTED].
         Mounting /boot...
[FAILED] Failed to mount /boot.
See 'systemctl status boot.mount' for details.
[DEPEND] Dependency failed for Local File Systems.

The dependency failure might have led systemd to decide to go into emergency mode, ~~and for whatever reason the keyboard became non-functional and I stuck here for eternity.~~

EDIT: It turns out that strange input prompt was a login prompt that wants my root password. I wonder how do I upload journals.

Last edited by nphuracm (2022-11-12 15:12:05)

Scimmia · 2022-11-12 15:01:31

So your /boot failed to mount and your keyboard doesn't work. Let me guess, you didn't mount the /boot partition into /mnt when you used pacman --root?

nphuracm · 2022-11-12 15:14:02

Scimmia wrote:

So your /boot failed to mount and your keyboard doesn't work. Let me guess, you didn't mount the /boot partition into /mnt when you used pacman --root?

Umm, turns out it does work - I edited my last post. That strange ?? root ????? prompt where I could not type in is actually a login prompt - it wants my root password. I typed my root pw into it and I went into my familiar shell again.

I did mount my ESP to /mnt/boot when doing that upgrade.

Last edited by nphuracm (2022-11-12 15:38:14)

Scimmia · 2022-11-12 15:51:06

So what does uname -r give you at that shell?

What filesystem is your /boot?

Last edited by Scimmia (2022-11-12 16:15:15)

nphuracm · 2022-11-12 16:16:18

Scimmia wrote:

So what does uname -r give you at that shell?
What filesystem is your /boot?

5.19.13-arch1-1

My /boot is FAT16.

Okay, something bad happened. It was supposed to be 6.0 or something.

Last edited by nphuracm (2022-11-12 16:16:54)

Scimmia · 2022-11-12 16:18:32

Yeah, that means your /boot wasn't mounted when you updated the kernel. Go back to the install disk, mount everything, arch-chroot in, then reinstall the kernel and headers.

nphuracm · 2022-11-12 16:38:21

Scimmia wrote:

Yeah, that means your /boot wasn't mounted when you updated the kernel. Go back to the install disk, mount everything, arch-chroot in, then reinstall the kernel and headers.

Well, I reinstalled the kernel, and the mkinitramfs output during pacman seems fine. But the DKMS output in the headers' install output is still that trash.
But when I reboot it, it booted into X11 normally. Success.

Last edited by nphuracm (2022-11-12 16:41:37)

Arch Linux

#1 2022-11-12 01:52:04

[SOLVED] I broke OpenSSL and then the whole system.

#2 2022-11-12 02:05:04

Re: [SOLVED] I broke OpenSSL and then the whole system.

#3 2022-11-12 02:47:20

Re: [SOLVED] I broke OpenSSL and then the whole system.

#4 2022-11-12 02:49:15

Re: [SOLVED] I broke OpenSSL and then the whole system.

#5 2022-11-12 02:51:43

Re: [SOLVED] I broke OpenSSL and then the whole system.

#6 2022-11-12 02:59:31

Re: [SOLVED] I broke OpenSSL and then the whole system.

#7 2022-11-12 03:01:32

Re: [SOLVED] I broke OpenSSL and then the whole system.

#8 2022-11-12 03:27:42

Re: [SOLVED] I broke OpenSSL and then the whole system.

#9 2022-11-12 04:00:14

Re: [SOLVED] I broke OpenSSL and then the whole system.

#10 2022-11-12 04:23:56

Re: [SOLVED] I broke OpenSSL and then the whole system.

#11 2022-11-12 08:48:18

Re: [SOLVED] I broke OpenSSL and then the whole system.

#12 2022-11-12 13:27:18

Re: [SOLVED] I broke OpenSSL and then the whole system.

#13 2022-11-12 13:50:57

Re: [SOLVED] I broke OpenSSL and then the whole system.

#14 2022-11-12 13:59:33

Re: [SOLVED] I broke OpenSSL and then the whole system.

#15 2022-11-12 14:05:27

Re: [SOLVED] I broke OpenSSL and then the whole system.

#16 2022-11-12 14:19:07

Re: [SOLVED] I broke OpenSSL and then the whole system.

#17 2022-11-12 14:27:54

Re: [SOLVED] I broke OpenSSL and then the whole system.

#18 2022-11-12 14:36:59

Re: [SOLVED] I broke OpenSSL and then the whole system.

#19 2022-11-12 14:59:27

Re: [SOLVED] I broke OpenSSL and then the whole system.

#20 2022-11-12 15:01:31

Re: [SOLVED] I broke OpenSSL and then the whole system.

#21 2022-11-12 15:14:02

Re: [SOLVED] I broke OpenSSL and then the whole system.

#22 2022-11-12 15:51:06

Re: [SOLVED] I broke OpenSSL and then the whole system.

#23 2022-11-12 16:16:18

Re: [SOLVED] I broke OpenSSL and then the whole system.

#24 2022-11-12 16:18:32

Re: [SOLVED] I broke OpenSSL and then the whole system.

#25 2022-11-12 16:38:21

Re: [SOLVED] I broke OpenSSL and then the whole system.

Board footer