You are not logged in.

#1 2012-05-16 09:40:10

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

[Solved] Systemd-analyze and optimizations ?

I've been running systemd since a month now and it works OK. Now it is time to look into it deeper and possibly make some optimizations. systemd-analyze is a great tool for this but for example; systemd-analyze blame seems to give different results on different boots. All hardware, services and network are the same for each boot. Like this;

Startup finished in 12946ms (kernel) + 16278ms (userspace) = 29225ms
  3833ms systemd-modules-load.service
  1796ms boot.mount
  1481ms rc-local.service
  1227ms console-kit-daemon.service
  1182ms console-kit-log-system-start.service
   752ms home.mount
   692ms media.mount
...

This was default without any tinkering. This is with comment=systemd.automount inserted into fstab for /boot and /home;

Startup finished in 12938ms (kernel) + 15339ms (userspace) = 28278ms
  4344ms console-kit-daemon.service
  3728ms systemd-modules-load.service
  2079ms mnt-SERVER_NYTT.mount
  1246ms console-kit-log-system-start.service
  1163ms rc-local.service
   781ms systemd-remount-api-vfs.service
   658ms udev-trigger.service
   649ms systemd-vconsole-setup.service
   470ms systemd-logind.service
   467ms syslog-ng.service
   419ms network.servicea8f39c515c882b6dd878b74800000721
   375ms home.mount
   173ms boot.mount
...

This is with the same setup ;

Startup finished in 12712ms (kernel) + 15743ms (userspace) = 28455ms
  4564ms boot.mount
  2748ms home.mount
  2172ms systemd-modules-load.service
  2028ms rc-local.service
  1760ms console-kit-daemon.service
  1583ms console-kit-log-system-start.service
  1272ms systemd-vconsole-setup.service
   433ms network.service
   348ms rpc-statd.service
   344ms systemd-logind.service
   255ms systemd-tmpfiles-setup.service
   180ms syslog-ng.service
   143ms mnt-SERVER_NYTT.mount
...

The two top listings, boot and home, are slow inspite of my fstab additions. And they vary. Why?

Further to look into are  systemd-modules-load.service and console-kit-daemon.service. I have no speical hardware, thus no special modules to load. I have three blacklisted modules, that's all. Concerning the CK listing, it seems from other threads that a login manager, I use slim, is still neccessary to get CK support.

Any hints on further systemd optimizations are very welcome!

More info; I've got the sysemd-arch-units installed but removed some daemons from rc.conf when installing systemd. rc.conf daemons; ( syslog-ng dbus network @crond @sshd )
systemd.conf:

[Manager]
#LogLevel=info
LogTarget=syslog
LogColor=yes
LogLocation=no
#DumpCore=yes
#CrashShell=no
ShowStatus=yes
SysVConsole=yes
#CrashChVT=1
#CPUAffinity=1 2
#MountAuto=yes
#SwapAuto=yes
#DefaultControllers=cpu
DefaultStandardOutput=syslog
DefaultStandardError=inherit
#JoinControllers=cpu,cpuacct

Last edited by swanson (2012-05-21 02:14:10)

Offline

#2 2012-05-16 12:04:18

65kid
Member
From: Germany
Registered: 2011-01-26
Posts: 663

Re: [Solved] Systemd-analyze and optimizations ?

have you also added noauto to fstab? This is neccessary together with comment=systemd.automount if you want things to speed up.

I wonder why systemd-modules-load.service takes so long if you are not loading any modules. check your /etc/modules-load.d/* . I only load acpi_cpufreq and systemd-modules-load.service takes 32ms on my system.

You may also disable syslog-ng.service if you don't need it. systemd-journald should be sufficient for most people.

Also check if you actually need rc-local.service. Whatever you are doing in /etc/rc.local, writing a separate service file for it may be faster.

rc.conf shouldn't have any effect on systemd as long as you don't use arch-daemons.target from initscripts-systemd.

Offline

#3 2012-05-16 12:54:57

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Thank you! Now I'll be busy all week end!

Offline

#4 2012-05-16 14:05:57

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

noauto in fstab and disabling rc.local.service really did some difference;
Startup finished in 12576ms (kernel) + 11161ms (userspace) = 23738ms
...
  2661ms home.mount
  2301ms systemd-modules-load.service
  1035ms console-kit-daemon.service
   970ms console-kit-log-system-start.service
   285ms systemd-vconsole-setup.service
...

In modules-load.d I had some uneccessary nfs module as well. Now I'll look into arch-daemons.target
and see if there's more to be done.

Offline

#5 2012-05-20 16:42:56

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Now I have only dbus in rc.conf which are started by arch-daemons.target. I tried to use the systemd native unit instead of network in rc.conf but boot time went from 25s to 76s.

systemd-analyze
Startup finished in 13785ms (kernel) + 62360ms (userspace) = 76146ms

systemd-analyze blame
  2113ms mnt-SERVER_NYTT.mount
  1772ms systemd-modules-load.service
  1768ms home.mount
   773ms console-kit-daemon.service
   679ms network.service
   546ms rpc-statd.service
   351ms systemd-vconsole-setup.service
   316ms systemd-logind.service
   295ms console-kit-log-system-start.service
   170ms sshdgenkeys.service
   156ms dev-hugepages.mount
   154ms udev.service
   141ms rpcbind.service
    91ms systemd-sysctl.service
    81ms sys-kernel-debug.mount
    74ms udev-trigger.service
    50ms crond.service
    31ms systemd-remount-api-vfs.service
    28ms media.mount
    26ms dbus.service
    18ms remount-rootfs.service
    18ms dev-mqueue.mount
    17ms sys-kernel-security.mount
    12ms systemd-user-sessions.service
     2ms systemd-tmpfiles-setup.service
     0ms sys-fs-fuse-connections.mount

The native unit looks like this;

cat /usr/lib/systemd/system/network.service
[Unit]
Description=Network Connectivity
Wants=network.target
Before=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/rc.d/network start
ExecStop=/etc/rc.d/network stop

[Install]
WantedBy=multi-user.target

This is with only dbus in rc.conf. I know that with empty daemons in rc.conf systemd boots in somehing like 110s, so the transition from rc.conf to native systemd isn't going to well for me. Any ideas?

Offline

#6 2012-05-21 02:13:37

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Fixed it! I were down to two things in c.conf, dbus and network. I daringly removed dbus from rc.conf without there being any dbus.service enabled by me .  And it worked! And then for network too!

2635ms systemd-modules-load.service
   787ms console-kit-daemon.service
   509ms systemd-vconsole-setup.service
   358ms dev-hugepages.mount
   340ms udev.service
   244ms systemd-logind.service
   212ms systemd-sysctl.service
   211ms sys-kernel-debug.mount
   196ms home.mount
   144ms mnt-SERVER_NYTT.mount
   142ms console-kit-log-system-start.service
   119ms crond.service
   105ms dbus.service
    98ms rpcbind.service
    96ms remount-rootfs.service
    79ms systemd-user-sessions.service
    33ms udev-trigger.service
    26ms dev-mqueue.mount
    25ms sys-kernel-security.mount
    23ms network.service
    22ms systemd-tmpfiles-setup.service
    21ms sshdgenkeys.service
    15ms systemd-remount-api-vfs.service
    12ms media.mount
     6ms rpc-statd.service

Good for now!

Last edited by swanson (2012-05-21 02:16:56)

Offline

#7 2012-05-21 10:51:34

65kid
Member
From: Germany
Registered: 2011-01-26
Posts: 663

Re: [Solved] Systemd-analyze and optimizations ?

you don't have to / shouldn't enable dbus.service, it will automatically be started by systemd when another service need it.
And do yourself a favor and get rid of arch-daemons.target / rc.conf altogether. netcfg has native systemd service files now and should work much better and faster. I don't even have an /etc/rc.conf on my systems anymore and everything works fine.

Offline

#8 2012-05-21 11:27:11

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Yes, I've finally got some things right. I weeded out the daemons in rc.conf one by one since the first try to get rid of them all at once failed. In the end there's was only dbus left and I saw a native unit file for systemd. That, however, wasn't possible to enable by systemctl, so I just winged a boot without it. It worked! The only unit I made myself was to copy code from falconindys crond.service at github. For sshd I now let systemd run it as socket - on demand.

Concerning the network.service ; I see that the native one uses the command rc.d/network start (and stop), Would it be beneficial to try netcfg unit instead, for my single NIC wired interface? It would get one step closer to get rid of rc.conf as well.

Offline

#9 2012-05-21 11:35:28

65kid
Member
From: Germany
Registered: 2011-01-26
Posts: 663

Re: [Solved] Systemd-analyze and optimizations ?

swanson wrote:

Concerning the network.service ; I see that the native one uses the command rc.d/network start (and stop), Would it be beneficial to try netcfg unit instead, for my single NIC wired interface? It would get one step closer to get rid of rc.conf as well.

well the fact that the network.service file uses the rc.d script IMHO doesn't really make it "native". wink Nevertheless, using netcfg may not neccessarily speed up boot but it's always better not to mix systemd with initscripts stuff like rc.d / rc.conf. The systemd-initscripts package will be dropped at some point, so why not get rid of it now? wink
If you only use DHCP on your wired interface you don't even need netcfg, see wiki: https://wiki.archlinux.org/index.php/Systemd#Network

see also: https://wiki.archlinux.org/index.php/Ne … md_support

Offline

#10 2012-05-23 02:27:57

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Startup finished in 8080ms (kernel) + 6562ms (userspace) = 14642ms
  2038ms mnt-SERVER_NYTT.mount
  1197ms home.mount
   894ms systemd-vconsole-setup.service
   260ms console-kit-daemon.service
   140ms systemd-tmpfiles-setup.service
   135ms systemd-logind.service
   104ms dev-hugepages.mount
    87ms network.service
    76ms systemd-modules-load.service
    73ms udev-trigger.service
    73ms sshdgenkeys.service
    70ms udev.service
    67ms systemd-remount-api-vfs.service
    67ms dev-mqueue.mount
    65ms sys-kernel-security.mount
    64ms rpcbind.service
    63ms systemd-sysctl.service
    63ms udisks2.service
    62ms sys-kernel-debug.mount
    52ms media.mount
    49ms rpc-statd.service
    36ms console-kit-log-system-start.service
    33ms systemd-user-sessions.service
    23ms crond.service
    19ms dbus.service
     2ms remount-rootfs.service
     0ms sys-fs-fuse-connections.mount

New record! Made loglevel down to 'emerg', reduced the settings for logfile size and reran e4rat.

Offline

#11 2012-05-25 09:06:16

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Further findings (I'd really like to learn systemd);

Startup finished in 7588ms (kernel) + 7314ms (userspace) = 14903ms
  4228ms systemd-vconsole-setup.service
  2571ms systemd-remount-api-vfs.service
  2010ms mnt-SERVER_NYTT.mount
  1150ms dev-hugepages.mount
   570ms udev.service
   563ms udev-trigger.service
   519ms systemd-sysctl.service
   517ms sys-kernel-debug.mount
   503ms systemd-modules-load.service
   497ms dev-mqueue.mount
   492ms sys-kernel-security.mount
   490ms media.mount
   476ms home.mount
   367ms console-kit-daemon.service
   326ms systemd-tmpfiles-setup.service
   145ms sshdgenkeys.service
   142ms systemd-logind.service
   134ms dev-sda3.swap
   110ms rpcbind.service
    75ms rpc-statd.service
    46ms eth0-network.service
    35ms console-kit-log-system-start.service
    30ms dbus.service
    25ms crond.service
    16ms remount-rootfs.service
    14ms systemd-user-sessions.service

The recent adjustments I've made is to the nfs-client mount and to the static wired service. I've tried netcfg, which worked, but prolonged the boot by 2s compared to the systemd/rc.conf combo of starting /etc/rc.d/network. I made this more simple unit from a blueprint found in the big systemd thread, thanks to ron9;

[Unit]
Description=Network Connectivity
Requires=sys-devices-pci0000:00-0000:00:04.0-0000:01:06.0-net-eth0.device
After=sys-devices-pci0000:00-0000:00:04.0-0000:01:06.0-net-eth0.device
Wants=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/ip link set eth0 up ; /sbin/ip addr add 192.168.1.251/24 dev eth0 ; /sbin/ip route add default via 192.168.1.250
ExecStop=/sbin/ip link set eth0 down

[Install]
WantedBy=multi-user.target

This brought down the start time from about 2-4s for my static wired network to 46ms (eth0-network.service). Great!

Then my attention went to my nfs client mount (nfs4) which could take from 2s to 70s (!) to start depending on the systemd network services I used. But I thought it must be simple; start the network, maybe some rpc*.service and then mount it. Thanks to our Wiki I found a blueprint for mounting nfs shares which I edited like this;

[Unit]
Description=/mnt/SERVER_NYTT
Wants=network.target rpc-statd.service
After=network.target rpc-statd.service 
 
[Mount]
What=192.168.1.250:/
Where=/mnt/SERVER_NYTT
Type=nfs4
Options=rw,hard,async,intr,rsize=49152,wsize=49152,proto=tcp 
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

After googling, trial and error, comparing boot times, I finally understood what the wiki said. One key for me was to name the unit correctly, corresponding to my mount point. The mountpoint is /mnt/SERVER_NYTT so the unit name had to be mnt-SERVER_NYTT.mount.

Two other findings; in some unit files, like systemd-vconsole-setup.service, there are references to Plymouth and readahead, which you might not use. Another finding is that systemd is under rapid development and results you get from google are often outdated, and often Fedora specific.
But lucky us, we have 65kid and others to guide and cheer us along!

Now there's only one annyoing thing left; the systemd-vconsole-setup.service ! Why does it take so long to start? I ironed out some problem with not finding lat0-16 font, but that did nothing and journalctl has nothing to offer. Here's my /etc/vconsole.conf;

KEYMAP="sv-latin1"
FONT="lat0-16"

What's up with this? It shouldn't take 4228ms to initialize tty1? Are there other ways?

Offline

#12 2012-05-25 10:12:45

65kid
Member
From: Germany
Registered: 2011-01-26
Posts: 663

Re: [Solved] Systemd-analyze and optimizations ?

I've tried netcfg, which worked, but prolonged the boot by 2s compared to the systemd/rc.conf combo of starting /etc/rc.d/network.

Just because systemd-analyze says that netcfg.service took 2 seconds longer than network.service, that doesn't neccessarily mean that boot actually took 2 seconds longer, because services are started in parallel. wink nevertheless I'm surprised that netcfg takes 2 seconds to set up a static ip, seems really a bit long for me.

And you can remove StandardError/StandardOutput from the NFS mount file. This doesn't make sense if you are not running syslog (may even cause problems, I don't know). Also remove the Wants=network.target. maybe that it what caused the long delay (70s). afaik network.target should only be pulled in by the service that actually sets up the network ( netcfg.service or your custom network service in this case).

Two other findings; in some unit files, like systemd-vconsole-setup.service, there are references to Plymouth and readahead, which you might not use.

if these references are just After/Before and readahead/plymouth are not actually enabled, this will not have any effect. So you don't have to worry that this might slow down things. wink

Not really sure what is going on with systemd-vconsole-setup, it takes 97ms on my system. have you some custom getty service files or just the single symlink in /etc/systemd/system/getty.target.wants/... ?

Offline

#13 2012-05-25 10:52:50

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

I was down to 15s with rc.conf/network combo, but as you did, I'd like to get rid of rc.conf and go native. Therefore I tried netcfg which had a native service file. I actually did some careful comparisons and;

Startup finished in 7493ms (kernel) + 9859ms (userspace) = 17353ms
  4606ms netcfg@eth0static.service
...
Startup finished in 7588ms (kernel) + 7314ms (userspace) = 14903ms
(snip)
46ms eth0-network.service

Concerning nfs; the long (70s) start time was with ... noauto,comment=systemd.automount ... in fstab. Thusly no *.mount file of my making (or Arch) was involved. I'll remove the 'syslog' references.

About console/tty1 ( ls /etc/systemd/system/getty.target.wants/
) ;

getty@tty1.service -> /usr/lib/systemd/system/getty@.service
->
[Unit]
Description=Getty on %I
BindTo=dev-%i.device
#After=dev-%i.device systemd-user-sessions.service plymouth-quit-wait.service
After=dev-%i.device systemd-user-sessions.service
#After=rc-local.service

# If additional gettys are spawned during boot then we should make
# sure that this is synchronized before getty.target, even though
# getty.target didn't actually pull it in.
Before=getty.target

[Service]
Environment=TERM=linux
ExecStart=-/sbin/agetty %I 38400
Restart=always
RestartSec=0
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
TTYVTDisallocate=yes
KillMode=process
IgnoreSIGPIPE=no

# Unset locale for the console getty since the console has problems
# displaying some internationalized messages.
Environment=LANG= LANGUAGE= LC_CTYPE= LC_NUMERIC= LC_TIME= LC_COLLATE= LC_MONETARY= LC_MESSAGES= LC_PAPER= LC_NAME= LC_ADDRESS= LC_TELEPHONE= LC_MEASUREMENT= LC_IDENTIFICATION=

# Some login implementations ignore SIGTERM, so we send SIGHUP
# instead, to ensure that login terminates cleanly.
KillSignal=SIGHUP

[Install]
Alias=getty.target.wants/getty@tty1.service

The contents are not mine except two '#'. I followed the wiki here: https://wiki.archlinux.org/index.php/Sy … default.3F

I really only need one other tty except tty7 of course, so might there be simplifications done? (Slim is set to start on tty7 from /etc/slim.conf)

Really appreciate your time and knowledge! Thanks!

Last edited by swanson (2012-05-25 10:55:14)

Offline

#14 2012-06-05 18:08:52

android_808
Member
Registered: 2011-10-17
Posts: 19

Re: [Solved] Systemd-analyze and optimizations ?

I'm sitting here thinking that there must be something seriously slowing down your boot times.

For example, Eee 1000, 8GB + 32GB original SSDs, 1.6Ghz Atom N270 with upgraded 2GB Crucial 667MHz DDR2.

Kernel Netbook:
Startup finished in 2319ms (kernel) + 2475ms (userspace) = 4795ms

Default Kernel is about 5700ms

   979ms systemd-udev-trigger.service
   923ms udisks2.service
   658ms NetworkManager.service
   645ms systemd-vconsole-setup.service
   423ms sys-kernel-debug.mount
   414ms dev-mqueue.mount
   408ms dev-hugepages.mount
   290ms systemd-logind.service
   252ms home.mount
   122ms systemd-udev.service
   117ms dbus.service
   111ms systemd-tmpfiles-setup.service
   106ms systemd-sysctl.service
    76ms console-kit-log-system-start.service
    65ms systemd-remount-fs.service
    48ms console-kit-daemon.service
    41ms systemd-readahead-replay.service
    37ms accounts-daemon.service
    36ms systemd-readahead-collect.service
    24ms systemd-user-sessions.service
    21ms upower.service
     7ms tmp.mount
     7ms rtkit-daemon.service
     5ms proc-sys-fs-binfmt_misc.mount
     4ms sys-fs-fuse-connections.mount

GRUB2 Default command line: "quiet add_efi_memmap init=/bin/systemd root=/dev/sda1 rootfstype=ext4 ro logo.nologo raid=noautodetect libahci.ignore_sss=1"

I have no daemons in rc.conf.  I don't need cron at the moment, syslog-ng isn't really required as systemd has journald.  See if you can specify more of the config at the grub command prompt to avoid some detection routines.  Module loading is taking about .5s which will delay anything requiring them, such as network drivers.  Might be worth trying to build them in a custom kernel.

initrd doesn't seem to have much of an effect on the reported time, but if you can do without, it should speed up overall boot time.

Offline

#15 2012-06-05 18:33:44

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Your SSD does account for some of the differences.

Offline

#16 2012-06-08 08:02:44

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

With updates and some edits to fstab i'm down to;

Startup finished in 8142ms (kernel) + 3425ms (userspace) = 11567ms

Which I'm happy with on my HDD. Since I'm using e4rat and autologin and starting some apps, which e4rat makes go into the kernel figure, it's really ok.

So, look out for unecessary lines in fstab, specifically tmpfiles.

Offline

#17 2012-06-12 18:22:46

teateawhy
Member
From: GER
Registered: 2012-03-05
Posts: 1,138
Website

Re: [Solved] Systemd-analyze and optimizations ?

@swanson:
There is no need to disable ttys with systemd as they are started on demand when you switch to them.
If you want to limit the amount of ttys anyway, because you are paranoid like me, you can do the following:
Edit      /etc/systemd/logind.conf
[Login]
NAutoVTs=3

You would get 3 ttys from this, the other ones are ignored. I do not now if it is safe to have less than that.
Try out 2 and tell me :-P

Offline

#18 2012-06-12 19:57:55

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: [Solved] Systemd-analyze and optimizations ?

Already have just 2 since a while and it's ok. (NAutoVTs=2)

Offline

#19 2012-06-29 09:48:30

jouke
Member
Registered: 2009-10-14
Posts: 72

Re: [Solved] Systemd-analyze and optimizations ?

65kid wrote:

I've tried netcfg, which worked, but prolonged the boot by 2s compared to the systemd/rc.conf combo of starting /etc/rc.d/network.

Just because systemd-analyze says that netcfg.service took 2 seconds longer than network.service, that doesn't neccessarily mean that boot actually took 2 seconds longer, because services are started in parallel. wink nevertheless I'm surprised that netcfg takes 2 seconds to set up a static ip, seems really a bit long for me.

In netcfg 2.8.5, the 2 second delay is turned into a polling timeout. For good hardware/drivers, this means the delay is gone. If your card is slow to come up, netcfg will calmly wait for it smile.

65kid was right about the parallel thing, though.

Offline

Board footer

Powered by FluxBB