[SOLVED] Struggling to optimize system startup times

Frontear · 2023-10-02 23:11:12

I've been trying to reduce startup time to the best of my ability. Using systemd-analyze, I've found that graphical.target takes about 2s to boot up, which I was hoping to streamline down to under a second. I noted systemd-analyze critical-chain was reporting my .device for my boot partition (sometimes blames root partition, or -.mount -> fsck@root -> dev/disk-by-uuid etc etc, so its not specifically the boot partition) apparently takes an extra 800ms to load up, which seems somewhat high given I'm running off an SSD. I've been trying various things to see how I can improve these timings to no avail, at least nothing for the registering of the device itself under systemd. This is before fsck or mount operations occur, so it's further back from userspace and I'm not totally sure what I could be doing.

Output of systemd-analyze (I brought down graphical.target to about 1.6 by disabling some things like backlight saving):

Startup finished in 6.144s (firmware) + 466ms (loader) + 401ms (kernel) + 648ms (initrd) + 1.601s (userspace) = 9.261s 
graphical.target reached after 1.601s in userspace.

Output of systemd-analyze critical-chain:

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @1.601s
└─ly.service @1.601s
  └─systemd-user-sessions.service @1.567s +19ms
    └─network.target @1.564s
      └─wpa_supplicant.service @1.553s +11ms
        └─dbus.service @730ms +15ms
          └─basic.target @723ms
            └─sockets.target @723ms
              └─dbus.socket @723ms
                └─sysinit.target @722ms
                  └─systemd-timesyncd.service @672ms +49ms
                    └─systemd-tmpfiles-setup.service @579ms +68ms
                      └─local-fs.target @565ms
                        └─boot.mount @507ms +57ms
                          └─systemd-fsck@dev-disk-by\x2duuid-362A\x2d1483.service @449ms +41ms
                            └─dev-disk-by\x2duuid-362A\x2d1483.device @584542y 2w 2d 20h 1min 49.191s +807ms

Also on a sidenote, I'm not sure what's up with the timings claiming 500 000 years, but that's besides the point because it obviously isn't working. Additionally, I couldn't find a good way to reduce firmware timings at all. I don't use fallback, and I've disabled most of the extraneous things on my laptop in the BIOS, like bluetooth, the microphone, the webcam, the touchscreen, which has indeed also prevented the kernel from loading these modules and firmware, and yet I have 6s of just firmware loading. I've also set my BIOS to "minimal" hardware initialization, where it will skip some things in favor of speeding the boot process, but it actually doesn't make a difference. This is a Dell Laptop so I've got pretty little customization BIOS wise beyond that. Couldn't find any other tips to reduce firmware timings, would appreciate any if there are.

Last edited by Frontear (2023-10-04 16:01:31)

cloverskull · 2023-10-03 04:23:02

I sped up my boot time by (IIRC) about 20% simply by changing from a busybox initrd to systemd. Once I did that, I removed the fsck hook and mounted my root filesystem read only on the kernel commandline. This may or may not be helpful to you but I recall seeing a pretty decent speedup in boot times after those changes.

Frontear · 2023-10-03 06:24:38

cloverskull wrote:

I sped up my boot time by (IIRC) about 20% simply by changing from a busybox initrd to systemd. Once I did that, I removed the fsck hook and mounted my root filesystem read only on the kernel commandline. This may or may not be helpful to you but I recall seeing a pretty decent speedup in boot times after those changes.

Haha, this is useful, but I've already actually done these things, when following the tips for Silent Boot, thanks though!

-thc · 2023-10-03 07:38:28

Try

systemd-analyze plot > Desktop/boot.svg

and view it with a Browser or Inkscape. Is the 800 ms delay really holding up the boot process (red with the next unit below it starting after the complete 800 ms have passed)?

I consider 1.6 s userspace time very fast.

Last edited by -thc (2023-10-03 07:38:54)

Frontear · 2023-10-03 22:17:32

-thc wrote:

Try
systemd-analyze plot > Desktop/boot.svg
and view it with a Browser or Inkscape. Is the 800 ms delay really holding up the boot process (red with the next unit below it starting after the complete 800 ms have passed)?
I consider 1.6 s userspace time very fast.

Good point, I didn't even remember systemd has a graph plot thing.

It does half half of it being red, so it does hold up the boot process, but so do all the other drives, which makes sense since systemd needs to process all the devices. I've uploaded the boot.svg to http://0x0.st/HW10.svg, this is really hard to read.

-thc · 2023-10-04 05:33:58

Your link results in an error:

Process 23 stopped
* thread #1: tid = 23, 0x00007f1aca9c4c70, name = 'fhost'
    frame #0:
Process 23 stopped
* thread #8: tid = 23, 0x00007f1b3bf97b80 fhost`get(path='/HW10.svg') + 27 at fhost.c:139, name = 'fhost/responder', stop reason = invalid address (fault address: 0x30)
    frame #0: {3:#018x} fhost`get(path='/HW10.svg') + 27 at fhost.c:139
   136   get(SrvContext *ctx, const char *path)
   137   {
   138       StoredObj *obj = ctx->store->query(shurl_debase(path));
-> 139       switch (obj->type) {
   140           case ObjTypeFile:
   141               ctx->serve_file_id(obj->id);
   142               break;
(lldb) q

seth · 2023-10-04 06:11:26

https://0x0.st/HW1O.svg - this is why we don't transcribe things manually and if we do, we don't use shitty monospace fonts where you cannot

distinguish 0 and O

No idea whatthe OP is after, but the problem is obviously the 6s in firmware which is either a problem w/ the UEFI or an encryted system and the OP slow to type their PW?

-thc · 2023-10-04 07:11:16

@seth: No, the TO used "systemd-analyze critical-chain" which implied the 800 ms for the NVMe device are needlessly slowing down the boot process. I consider 5 to 10 seconds in the firmware perfectly normal.

Frontear wrote:

It does half half of it being red, so it does hold up the boot process, but so do all the other drives, which makes sense since systemd needs to process all the devices. I've uploaded the boot.svg to http://0x0.st/HW10.svg, this is really hard to read.

Red means "activating" and if the next bar immediately below it has to wait until it's predecessor is finished starting - that would be a holdup.

In your case the TTY and disk devices take 1000 and 800 ms to activate - but all the while from "initrd-root-device.target" to "sys-devices-LNXSYSTM:00-LNXSYBUS:00-INTC6000:00-tpm-tpm0.device" systemd starts other units. The "systemd-fsck@dev-disk-by*" units mark the point at which the disk devices need to be ready - which they are.

The real wait times in your boot process are:
~ 200 ms between the TTY and the disk devices activating
~ 350 ms for the Networkmanger

Frontear · 2023-10-04 07:44:39

-thc wrote:

@seth: No, the TO used "systemd-analyze critical-chain" which implied the 800 ms for the NVMe device are needlessly slowing down the boot process. I consider 5 to 10 seconds in the firmware perfectly normal.
Frontear wrote:
It does half half of it being red, so it does hold up the boot process, but so do all the other drives, which makes sense since systemd needs to process all the devices. I've uploaded the boot.svg to http://0x0.st/HW10.svg, this is really hard to read.
Red means "activating" and if the next bar immediately below it has to wait until it's predecessor is finished starting - that would be a holdup.
In your case the TTY and disk devices take 1000 and 800 ms to activate - but all the while from "initrd-root-device.target" to "sys-devices-LNXSYSTM:00-LNXSYBUS:00-INTC6000:00-tpm-tpm0.device" systemd starts other units. The "systemd-fsck@dev-disk-by*" units mark the point at which the disk devices need to be ready - which they are.
The real wait times in your boot process are:
~ 200 ms between the TTY and the disk devices activating
~ 350 ms for the Networkmanger

Not too surprised by NetworkManager, are there any tips for reducing that 200ms between tty? I wonder if I could disable majority of those TTYs since I dont use em. As for NetworkManager, i’ll probably consult the wiki, but do you any tips for that one as well?

-thc · 2023-10-04 08:31:11

Frontear wrote:

Not too surprised by NetworkManager, are there any tips for reducing that 200ms between tty? I wonder if I could disable majority of those TTYs since I dont use em. As for NetworkManager, i’ll probably consult the wiki, but do you any tips for that one as well?

Not really - my system boots up in a different order and the 31 tty's create no holdup at all. On the other hand my "systemd-udev-trigger-service" needs nearly 300 ms and yours don't.

As for the NetworkManager - mine only needs 97 ms - but I have WiFi disabled .

Frontear · 2023-10-04 16:00:52

-thc wrote:

Frontear wrote:
Not too surprised by NetworkManager, are there any tips for reducing that 200ms between tty? I wonder if I could disable majority of those TTYs since I dont use em. As for NetworkManager, i’ll probably consult the wiki, but do you any tips for that one as well?
Not really - my system boots up in a different order and the 31 tty's create no holdup at all. On the other hand my "systemd-udev-trigger-service" needs nearly 300 ms and yours don't.
As for the NetworkManager - mine only needs 97 ms - but I have WiFi disabled .

Seems fair, thanks a lot for the input, Ill see what I can do for firmware loading times and network manager, solved!

Arch Linux

#1 2023-10-02 23:11:12

[SOLVED] Struggling to optimize system startup times

#2 2023-10-03 04:23:02

Re: [SOLVED] Struggling to optimize system startup times

#3 2023-10-03 06:24:38

Re: [SOLVED] Struggling to optimize system startup times

#4 2023-10-03 07:38:28

Re: [SOLVED] Struggling to optimize system startup times

#5 2023-10-03 22:17:32

Re: [SOLVED] Struggling to optimize system startup times

#6 2023-10-04 05:33:58

Re: [SOLVED] Struggling to optimize system startup times

#7 2023-10-04 06:11:26

Re: [SOLVED] Struggling to optimize system startup times

#8 2023-10-04 07:11:16

Re: [SOLVED] Struggling to optimize system startup times

#9 2023-10-04 07:44:39

Re: [SOLVED] Struggling to optimize system startup times

#10 2023-10-04 08:31:11

Re: [SOLVED] Struggling to optimize system startup times

#11 2023-10-04 16:00:52

Re: [SOLVED] Struggling to optimize system startup times

Board footer