[SOLVED][systemd-networkd] hostapd starts before wifi module loaded

epinephrine · 2015-11-21 22:06:50

SOLUTION:
a) Use a systemd timer hostapd.timer with OnBootSec=5sec to make sure all wifi hardware is initialized.
b) (more elegant to me) Add a dependency to the wifi device to the [Unit] section of hostapd.service:

BindsTo=sys-subsystem-net-devices-wlp4s0.device
After=sys-subsystem-net-devices-wlp4s0.device

I build an access point with systemd-networkd, which bridges two ethernet and two wifi interfaces (5GHz plus 2.4GHz AP) together.

The two wifi interfaces are wlp4s0 (mPCI express) and wlp0s18u1u1 (usb). However, I can see from journalctl that the hostapd systemd service starts before interface wlp4s0's module (ath10k_pci) is loaded. As such, the hostapd service fails to start:

# systemctl status hostapd -l

● hostapd.service - Hostapd IEEE 802.11 AP, IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
   Loaded: loaded (/usr/lib/systemd/system/hostapd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2015-11-21 19:33:59 CET; 3h 6min ago
  Process: 339 ExecStart=/usr/bin/hostapd /etc/hostapd/hostapd.conf (code=exited, status=1/FAILURE)
 Main PID: 339 (code=exited, status=1/FAILURE)

Nov 21 19:33:59 yoda hostapd[339]: nl80211: Driver does not support authentication/association or connect commands
Nov 21 19:33:59 yoda hostapd[339]: nl80211: deinit ifname=wlp4s0 disabled_11b_rates=0
Nov 21 19:33:59 yoda hostapd[339]: Could not read interface wlp4s0 flags: No such device
Nov 21 19:33:59 yoda hostapd[339]: nl80211 driver initialization failed.
Nov 21 19:33:59 yoda hostapd[339]: wlp4s0: interface state UNINITIALIZED->DISABLED
Nov 21 19:33:59 yoda hostapd[339]: wlp4s0: AP-DISABLED
Nov 21 19:33:59 yoda hostapd[339]: hostapd_free_hapd_data: Interface wlp4s0 wasn't started
Nov 21 19:33:59 yoda systemd[1]: hostapd.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 19:33:59 yoda systemd[1]: hostapd.service: Unit entered failed state.
Nov 21 19:33:59 yoda systemd[1]: hostapd.service: Failed with result 'exit-code'.

Why does systemd start the hostapd service before the wireless module is initialized? How can I make systemd wait for the interface wlp4s0 to be present before starting hostapd?

Another problem is that the bridge complains about the wifi interfaces not being ready

Nov 21 19:34:00 yoda systemd-networkd[333]: wlp4s0: wlp4s0          : could not join netdev: Operation not supported
Nov 21 19:34:00 yoda systemd-networkd[333]: wlp4s0: Failed
Nov 21 19:34:00 yoda systemd-networkd[333]: br0: Lost carrier

The same for wlp0s18u1u1.
My systemd config:
bridge.netdev

[NetDev]
Name=br0
Kind=bridge

br0.network

[Match]
Name=br0

[Network]
Address=192.168.0.3/24
Gateway=192.168.0.1
DNS=192.168.0.1

br0-slaves.network

[Match]
Name=enp* wlp*

[Network]
Bridge=br0

Should I remove the wlp* part from the Name assertion in the [Match] section and instead let hostapd add the interface to the bridge with the line

bridge=br0

in /etc/hostapd/hostapd.conf?

Cheers!

Last edited by epinephrine (2015-11-30 20:22:13)

Leonid.I · 2015-11-22 00:40:01

TL:DR: Start hostapd from a timer, configured with e.g. 15sec delay.

I have seen similar behavior with an intel card. Basically, AFAIU udev/kernel load a module and systemd proceeds booting the machine as usual. However, noone takes into accout the fact that it can take some time for the card CPU to actually process the firmware. Before systemd this was not a problem because the bootup was sequential and the card had enough time to finish the initialization before actual daemons are started. Now, the situation is fundamentally racy because you can't predict when hostapd is started.

About your bridge problem, I don't understand: why do you need networkd for that? The wireless interface can only be in the bridge when it is in Master mode, so networkd rightfully fails (hostapd is not around) -- it is hostapd's job to insert the wlan interface into bridge. But of course, the bridge should exist before hostapd is started.

epinephrine · 2015-11-22 20:53:00

Thanks! I also suspected a race condition caused by systemd. However, inserting a delay with, for example, ExecStart=sleep 5 in the [Service] section of hostapd.service should be a last resort. Seems kinda dowdy
I would have assumed that network.target is defined as reached when at least all modules have finished initialization... Is this worth a bug report to the systemd guys?

About the bridge, yeah, I removed the wlan interfaces from it in netword's config and let instead hostapd insert them.

loqs · 2015-11-22 22:21:47

epinephrine wrote:

I would have assumed that network.target is defined as reached when at least all modules have finished initialization... Is this worth a bug report to the systemd guys?

network.target

Leonid.I · 2015-11-22 23:40:59

epinephrine wrote:

Thanks! I also suspected a race condition caused by systemd. However, inserting a delay with, for example, ExecStart=sleep 5 in the [Service] section of hostapd.service should be a last resort. Seems kinda dowdy

Well, that's the price you pay for parallelizing stuff. I do this all the time for (i) delayed start of wpa_supplicant because the firmware in my Intel card loads and soft-blocks the radio, but rfkill-unblock starts _before_ the FW is loaded; (ii) proper mounting of BTRFS subvolumes, because otherwise the systemd breaks boot.

This may be a matter of taste, but I'd do this with an OnBoot=5sec timer, instead of ExecStart/ExecStartPre.

epinephrine wrote:

I would have assumed that network.target is defined as reached when at least all modules have finished initialization... Is this worth a bug report to the systemd guys?
About the bridge, yeah, I removed the wlan interfaces from it in netword's config and let instead hostapd insert them.

There is no magic here: network.target is simply an arbitrary ordering point with respect to starting of the services. It is customary to say that all network daemons (sshd, ntpd, etc) are After=network.target, and things like netctl/networkmanager are Before=network.target. But there is no quarantee (unless you use Type=oneshot services, like netctl) that network is set up in time for the daemon startup. This is as far you can go, because it is impossible to know in general what it means for network to be actually set up (an interface gets an IP? routing table is populated? FW is loaded?).

I would guess that systemd has no way of knowing that a FW is done loading, unless there is some kernel mechanism for notifying about this event, so the only reason it is systemd's fault is its way of parallelizing startup, but this is unlikely to change.

Last edited by Leonid.I (2015-11-22 23:41:59)

rsmarples · 2015-11-23 05:10:18

Leonid.I wrote:

Well, that's the price you pay for parallelizing stuff. I do this all the time for (i) delayed start of wpa_supplicant because the firmware in my Intel card loads and soft-blocks the radio, but rfkill-unblock starts _before_ the FW is loaded; (ii) proper mounting of BTRFS subvolumes, because otherwise the systemd breaks boot.

I don't have the hardware in question, but I tend to build my BSD and linux kernes monolithic just because some fancy init Sometimes Gets It Wrong.
I have yet to see any benefit from a modular system when you "hand roll" your own kernel and this approach is fine at runtime.

epinephrine · 2015-11-23 10:31:22

Leonid.I wrote:

This may be a matter of taste, but I'd do this with an OnBoot=5sec timer, instead of ExecStart/ExecStartPre.

I agree, that is much nicer. I haven't played a lot with systemd yet, didn't know the OnBootSec= timer yet. Will use that.

Leonid.I wrote:

epinephrine wrote:
I would have assumed that network.target is defined as reached when at least all modules have finished initialization... Is this worth a bug report to the systemd guys?
There is no magic here: network.target is simply an arbitrary ordering point with respect to starting of the services. It is customary to say that all network daemons (sshd, ntpd, etc) are After=network.target, and things like netctl/networkmanager are Before=network.target. But there is no guarantee (unless you use Type=oneshot services, like netctl) that network is set up in time for the daemon startup. This is as far you can go, because it is impossible to know in general what it means for network to be actually set up (an interface gets an IP? routing table is populated? FW is loaded?).
I would guess that systemd has no way of knowing that a FW is done loading, unless there is some kernel mechanism for notifying about this event, so the only reason it is systemd's fault is its way of parallelizing startup, but this is unlikely to change.

I now see that I didn't get the concept about network.target right. I saw it as too magical.
Maybe in the distant feature we will see a new target, network-hw-up.target that is triggered by all ethernet hardware being initialized

Will mark this as solved an use a timer with OnBootSec=5sec.

Slithery · 2015-11-23 16:55:59

epinephrine wrote:

Maybe in the distant feature we will see a new target, network-hw-up.target that is triggered by all ethernet hardware being initialized

Did you read the link in loqs earlier post?

Such a thing already exists.

epinephrine · 2015-11-23 17:00:13

slithery wrote:

epinephrine wrote:
Maybe in the distant feature we will see a new target, network-hw-up.target that is triggered by all ethernet hardware being initialized
Did you read the link in loqs earlier post?
Such a thing already exists.

I did read the link. But network-online.target is not what I mean. I am thinking about a target that is reached before network-online.target, but after all firmware is loaded etc., i.e., after all network modules reported back to the kernel/system that they have initialized themselves.

branch · 2015-11-23 23:03:50

epinephrine wrote:

I did read the link. But network-online.target is not what I mean. I am thinking about a target that is reached before network-online.target, but after all firmware is loaded etc., i.e., after all network modules reported back to the kernel/system that they have initialized themselves.

The init system has no way to know what interfaces to expect, nor what services need to wait for what interfaces unless you tell it.

Did you try specifying the dependency in the [Unit] section of hostapd.service like this?

BindsTo=sys-subsystem-net-devices-wlp4s0.device
After=sys-subsystem-net-devices-wlp4s0.device

Leonid.I · 2015-11-24 05:11:14

This will simply start hostapd after the device appears, not after it's ready... At least in my experience, such ordering didn't help but I can not speak for the OP.

epinephrine · 2015-11-30 20:20:05

branch wrote:

The init system has no way to know what interfaces to expect, nor what services need to wait for what interfaces unless you tell it.
Did you try specifying the dependency in the [Unit] section of hostapd.service like this?
BindsTo=sys-subsystem-net-devices-wlp4s0.device
After=sys-subsystem-net-devices-wlp4s0.device

Now this is what I was talking about! Great, this is even more elegant than using a timer and it works flawlessly. Thanks!

Arch Linux

#1 2015-11-21 22:06:50

[SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#2 2015-11-22 00:40:01

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#3 2015-11-22 20:53:00

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#4 2015-11-22 22:21:47

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#5 2015-11-22 23:40:59

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#6 2015-11-23 05:10:18

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#7 2015-11-23 10:31:22

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#8 2015-11-23 16:55:59

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#9 2015-11-23 17:00:13

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#10 2015-11-23 23:03:50

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#11 2015-11-24 05:11:14

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

#12 2015-11-30 20:20:05

Re: [SOLVED][systemd-networkd] hostapd starts before wifi module loaded

Board footer