[solved] Boot stuck at black screen with after systemd

seth · 2021-01-31 10:04:53

Is the phone still attached and do you use it for tethering?

Erotaku · 2021-01-31 10:09:23

seth wrote:

Is the phone still attached and do you use it for tethering?

I detached the phone's USB cable the last attempt. Also yes, I use that phone to tether the internet connection to my PC.

Last edited by Erotaku (2021-01-31 10:21:52)

seth · 2021-01-31 14:41:39

Can you boot "nomodeset"?
You're not facing anything like this w/ the install medium, right?

Chroot into the installed system and run

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

and post the output…

Erotaku · 2021-01-31 16:38:53

seth wrote:

Can you boot "nomodeset"?
You're not facing anything like this w/ the install medium, right?
Chroot into the installed system and run
find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
and post the output…

I used 'nomodeset' as kernel parameter, and nothing changed.
My live usb installation ISO is perfectly fine without any issue, yes. I have used it like 30 times in the past two days so far haha.

Here is the output, doc:

dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.resolve1.service    | system
dirmngr.socket                           | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gpg-agent-browser.socket                 | sockets.target.wants
gpg-agent-extra.socket                   | sockets.target.wants
gpg-agent-ssh.socket                     | sockets.target.wants
gpg-agent.socket                         | sockets.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
NetworkManager.service                   | multi-user.target.wants
p11-kit-server.socket                    | sockets.target.wants
pulseaudio.socket                        | sockets.target.wants
remote-fs.target                         | multi-user.target.wants
systemd-resolved.service                 | multi-user.target.wants

Erotaku · 2021-02-01 15:06:47

So, I have been thinking what to do next. As annoyingly as it would be to scratch all my progress and start over, I would still do it if it would fix my issue at hand. However, I will be just repeating every step I have taken in my current installation. Which means what ever is happening now will just happen again right away.

I am honestly stuck right now. If you guys are giving up, I would understand. But at least point me to another direction and place that can solve my problem. I would so very much appreciate that.

seth · 2021-02-01 15:40:00

Disable NM and systemd-resolved, but that's voodoo territorry w/ exactly no reason I could state beyond "it's there"…

Erotaku · 2021-02-01 18:15:30

seth wrote:

Disable NM and systemd-resolved, but that's voodoo territorry w/ exactly no reason I could state beyond "it's there"…

Done... Nothing changed. Honestly, the fact that I can see the tty login screen and can not type anything, that in and of itself is just abnormal to me. No matter what error is taking place, why can I not type into a non frozen login screen?

Anyway, there is another bizarre thing I have noticed. I had completely uninstalled systemd-resolved service with a pacman -Rns, and turned off its service before that. How was the service still on? Anyway, I went and deleted the whole of /run/systemd/resolve folder too. That did not change anything either.

I have pretty much turned off everything at this point, and it's still not functioning as normal. I have reversed almost everything I have done the last day before the problem started manifesting too.

Oh well, if you think a fresh installation will make this nightmare somehow go away, you need only to tell me.

Erotaku · 2021-02-05 17:38:36

I GOT IT! So, it turns out I was correct at what I sniffed at earlier, and you guys totally missed it even after giving you a hint. I told myself "what the hell. No one can help me, and my system is bricked. What is the worst that could happen" and I started thinking what I could try next.

It turns out that it was, indeed, the user groups what caused the whole issue from the start and everything else is fine. All I had to do was simply remove my user from video, audio and storage groups. For any future solution-seeking people, the only group you need to be in when you install your arch is just wheel.

seth · 2021-02-05 17:56:25

How can your user assignment to any group inhibit the systerm start if you're according to post #21 don't autologin at the console (nor through other measures like lightdm which you ruled out to be a factor in your initial post)?

What *exactly* did you do to resolve the situation?

Erotaku · 2021-02-05 19:15:30

Well, here is the way I can logically see it. As I have mentioned many times before, the system is clearly not frozen or crashing on start up. After you guys took a look at all the logs I have sent your way, it didn't seem like there were any errors.

So, there is a login screen that is not responding to my keystrokes. It's not a monitor/graphical glitch either. Then I went and read in the wiki that including groups such as audio was something that was needed in the past, and not anymore. In fact, it can potentially impede with the system's functions. So it was easy to figure out that this should by my next step.

As for what I exactly did? gpasswd -a myusername audio (then video and storage). That's pretty much it. I restarted, and I was able to type and log in. I re-enabled network manager, lightdm, vpn and fixed the dns file I deleted earlier. All is working flawlessly. That's all there is to it

Thank you very much both of you for trying to help me. You guys spent quite a bit of effort and time into this. I wouldn't have been able to figure it out on my own.

seth · 2021-02-05 20:53:17

gpasswd -a myusername audio (then video and storage). That's pretty much it. I restarted, and I was able to type and log in

But that would *add* you to those groups…

While it's true that those groups are no longer required since logind will use ACLs to grant to access to the affected devices and that gaining access to those devices by other means can have undesired side-effects (the audio device is given as example) this does in no way, shape or form explain how it could have any impact *previous* to logging in a that user.
Even iff you had actually removed him from those groups (instead of apparently simply re-adding him)

I suspect that something™ else happened before that reboot (some update) or - if absolutely not - there was another issue w/ /etc/group (maybe inhibiting some system daemon) and that got cured by the gpasswd invocation.

Erotaku · 2021-02-05 21:20:12

seth wrote:

gpasswd -a myusername audio (then video and storage). That's pretty much it. I restarted, and I was able to type and log in
But that would *add* you to those groups…
While it's true that those groups are no longer required since logind will use ACLs to grant to access to the affected devices and that gaining access to those devices by other means can have undesired side-effects (the audio device is given as example) this does in no way, shape or form explain how it could have any impact *previous* to logging in a that user.
Even iff you had actually removed him from those groups (instead of apparently simply re-adding him)
I suspect that something™ else happened before that reboot (some update) or - if absolutely not - there was another issue w/ /etc/group (maybe inhibiting some system daemon) and that got cured by the gpasswd invocation.

Ah, my mistake there! "gpasswd -d user group" is the one I did. I copy pasted the wrong command above.

Otherwise, you definitely present good and sound explanation/analysis, from what little I can tell. And that's the thing, I really don't know much. I was purely driven by the wiki (as a good arch user should be doing ), gut feeling and google research.

That being said, before I did what I did with groups, I specifically remembered to try and boot in just to make sure it's really still not booting up. I had a feeling my fix would work and I didn't want to assume anything false. It sure as hell did not boot. The moment I changed the user groups, everything started working as intended. I can never explain that myself.

Last edited by Erotaku (2021-02-05 21:21:32)

seth · 2021-02-05 21:27:31

You could re-add yoru user to individual groups and see whether any of them re-breaks the boot process.
If none does, the fix was merely a side-effect.
If one does, we at least have another lead to get to the bottom of this.

Erotaku · 2021-02-05 21:38:16

Oh, that's very interesting! I will definitely do this tomorrow when I wake up. I will quote you and reply back with my findings.

Last edited by Erotaku (2021-02-05 21:39:42)

Erotaku · 2021-02-06 09:45:56

seth wrote:

You could re-add yoru user to individual groups and see whether any of them re-breaks the boot process.
If none does, the fix was merely a side-effect.
If one does, we at least have another lead to get to the bottom of this.

Mate, it's all but confirmed. The user groups is the one and only gear responsible for the machine coming to a halt. I just re-added back the three groups in question. Upon restarting, the black screen with an underscore was back and boot was stuck once again. I don't know what's going on under the hood, but it can't be anything else anymore.

The only influencing part I could think of is Wayland. Most of you don't use it, and it's wildly different in how it operates compared to X11 to a relative degree. Maybe this is something the arch linux team could explore, or at least know about. After checking many many threads regarding this problem, the only people who were able to identify it were the folks at gentoo forums.

seth · 2021-02-06 09:48:13

Which one is the critical one and do you in any way, shape or form autologin or run a service as your user (w/o logging in)?

Erotaku · 2021-02-06 12:49:18

seth wrote:

Which one is the critical one and do you in any way, shape or form autologin or run a service as your user (w/o logging in)?

My sanity is depleting... What the hell is going!

I opted out of the storage group = not booting. Then audio group = not booting. It has to be video group now, right? Well, no. With all three groups out of the equation, it's still back to the black screen... HOW! The moment I re-added the groups back, it stopped working. How is removing the groups again not fixing the issue?

Okay, so here are more things I have done (and noticed happening) in an effort to get to the bottom of this insanity:

After removing the three groups in question, I then went and also opted out of autologin group as well, and disabled lightdm. I still got stuck at the login screen without being able to type anything
Booting into LTS kernel with lightdm and autologin group enabled gave me the same results. HOWEVER, with lightdm and autologin disabled? I can type and login my user in the tty.
From there, I started lightdm with systemctl. I was greeted with another black screen.
I rebooted and went into LTS kernel again. This time and after suspecting lightdm, I tried launching my window manager with a direct "sway" console command. The system crashed/froze right away.
Now onto the most important observation: Today I had updated my system prior to all the groups changing games I had done this morning. Part of the updates were the linux and LTS kernel. I distinctly remember reading the version 5.10.13.arch from the new package. However, when I do boot into the current kernel, it's showing me 5.10.12.arch version number.

That has to be it? I don't know anymore. Why is it booting into an old kernel version though? I even went back into the LTS kernel (which is also one version behind) and I checked with a pacman -Q linux and linux-lts. They are in fact updated and one version higher than what's being loaded with systemd.

Send help...

seth · 2021-02-06 14:57:06

The autologin group is (only?) relevant for lightdm autologins.
So I suspect that you configured lightm (X11) to autologin your user and run a wayland session from there?

=> Disable any autologin feature and see https://wiki.archlinux.org/index.php/Li … TTY_output
If this is effectively a single user system and you intend to use a wayland session, it's probably a good idea to skip any (X11) Dm altogether (but eg. autologin into a locked session, but AFTER! we solved this one here)

reading the version 5.10.13.arch from the new package. However, when I do boot into the current kernel, it's showing me 5.10.12.arch version number.

You forgot to mount the /boot partition before updating or are in fact booting from the root partition.
You've to fi that because otherwise you'll not be able to load kernel modules that are not in the initramfs.
Speaking of which: do you have similar issues when booting the failsafe/fallback initramfs?

Erotaku · 2021-02-06 17:53:24

seth wrote:

The autologin group is (only?) relevant for lightdm autologins.
So I suspect that you configured lightm (X11) to autologin your user and run a wayland session from there?

I checked my lightdm config file. I only have three lines uncommented. Two of them were just the username and window manager for autologin group. The third one is "session-wrapper=/etc/lightdm/Xsession". I don't know what to do with that one.

What I did do however, was reinstalling my linux and LTS kernel in a live usb chrooted environment after mounting all my partitions. Aaand it's booting fine now.... Now that I think about it, every time the booting problem started happening, I would have updated the system beforehand. I am pretty sure the first incident had a kernel update prior like this time.

So, now that the system is working again. I checked what's mounted, and sure enough the VFAT partition is not mounted. Why is that the case? Is this a systemd-boot related thing also? If you could point me in the right direction regarding this topic, then that would be great. I will try to figure something out for 5th time from the systemd-boot wiki page.

I will also try re-adding the other groups and repeat the earlier experiment I did this morning as well.
Edit: Yep. I just readded all the other groups that I thought were the problem. The system is still booting fine.
Edit2: My fstab file does not have an entry for my VFAT partition. I am 110% sure that I had mounted my /boot folder on my VFAT partition before generating my fstab file during the installation.
Edit3: I created an entry in the fstab file manually with the following:

UUID=E310-4D7D					/boot		vfat		defaults	0 2

I restarted the system, and everything seems to be working in order and mounted correctly. Is "defaults" okay under options for a boot entry? I hope so.

Last edited by Erotaku (2021-02-06 18:43:44)

seth · 2021-02-06 21:00:31

Is "defaults" okay under options for a boot entry? I hope so.

Yes, is.

Now that I think about it, every time the booting problem started happening, I would have updated the system beforehand.

This thread might turn out to be the longest walk to "forgot to mount /boot?" ever

Erotaku · 2021-02-06 21:17:49

seth wrote:

This thread might turn out to be the longest walk to "forgot to mount /boot?" ever

Honestly, it's still hard to wrap my head around the behaviour the system was exhibiting just because of this. It feels like an error message should have been more a lot more "in your face" than an unresponsive login screen.

But yea, I never considered this because the current arch installation was a second attempt to make sure I had partitioned and mounted everything as perfectly as possible. I was very sure I had achieved just that... but apparently I was wrong there too.

I can't thank you enough for your time, patience and effort though. You stuck with me for a few days by now. I really appreciate that. I learnt a heckin' lot in the process as a newbie too. What a painful, stressing and sanity draining journey though... I would be ashamed to admit how many hours I have spent on all of this in total.

seth · 2021-02-06 21:52:35

The more typical patterns are that modules fail to load and you lack USB support etc. or that you're being dropped into an emergency shell.

You could compare

lsinitcpio /boot/initramfs-linux.img | grep modules

with

lsmod

to see which modules are loaded that are not provided by the initramfs (since those will fail to load)
Maybe there's something obvious - maybe you don't have the keyboard hook in mkintcpio.conf?

Arch Linux

#26 2021-01-31 10:04:53

Re: [solved] Boot stuck at black screen with after systemd

#27 2021-01-31 10:09:23

Re: [solved] Boot stuck at black screen with after systemd

#28 2021-01-31 14:41:39

Re: [solved] Boot stuck at black screen with after systemd

#29 2021-01-31 16:38:53

Re: [solved] Boot stuck at black screen with after systemd

#30 2021-02-01 15:06:47

Re: [solved] Boot stuck at black screen with after systemd

#31 2021-02-01 15:40:00

Re: [solved] Boot stuck at black screen with after systemd

#32 2021-02-01 18:15:30

Re: [solved] Boot stuck at black screen with after systemd

#33 2021-02-05 17:38:36

Re: [solved] Boot stuck at black screen with after systemd

#34 2021-02-05 17:56:25

Re: [solved] Boot stuck at black screen with after systemd

#35 2021-02-05 19:15:30

Re: [solved] Boot stuck at black screen with after systemd

#36 2021-02-05 20:53:17

Re: [solved] Boot stuck at black screen with after systemd

#37 2021-02-05 21:20:12

Re: [solved] Boot stuck at black screen with after systemd

#38 2021-02-05 21:27:31

Re: [solved] Boot stuck at black screen with after systemd

#39 2021-02-05 21:38:16

Re: [solved] Boot stuck at black screen with after systemd

#40 2021-02-06 09:45:56

Re: [solved] Boot stuck at black screen with after systemd

#41 2021-02-06 09:48:13

Re: [solved] Boot stuck at black screen with after systemd

#42 2021-02-06 12:49:18

Re: [solved] Boot stuck at black screen with after systemd

#43 2021-02-06 14:57:06

Re: [solved] Boot stuck at black screen with after systemd

#44 2021-02-06 17:53:24

Re: [solved] Boot stuck at black screen with after systemd

#45 2021-02-06 21:00:31

Re: [solved] Boot stuck at black screen with after systemd

#46 2021-02-06 21:17:49

Re: [solved] Boot stuck at black screen with after systemd

#47 2021-02-06 21:52:35

Re: [solved] Boot stuck at black screen with after systemd

Board footer