Arch Linux

blackraen

Member

Registered: 2013-01-27

Posts: 3

[SOLVED] Network start issues in new build (with systemd v197)

As a note I'm new to Arch and Systemd startup method though I've been playing with an arch build on my desktop for a few weeks.  I just finished building up a media rack system and installed the latest Arch build on it and noticed what seemed to me to be a race condition flaw with the new interface renaming that seems to have been introduced in systemd v197.

Prefacing, this is a new build created using the Beginner's Guide wiki page: https://wiki.archlinux.org/index.php/Beginners'_Guide

My network choice was to use ifplugd with a static assignment (ethernet-static).  My original setting for interface was "eth0".

# cat /etc/network.d/ethernet-static 
CONNECTION='ethernet'
DESCRIPTION='raptor_media_rack A'
INTERFACE='eno1'
IP='static'
ADDR='10.0.1.5'
GATEWAY='10.0.1.1'
DNS=('10.0.1.1')

On first reboot I observed that my network was not plumbed:

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

Only my lo was visible.  I then identified that my interface name was wrong:

# dmesg | grep eth | grep -v ACPI
[    4.752715] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) c8:60:00:9a:ce:6a
[    4.752718] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[    4.752765] e1000e 0000:00:19.0 eth0: MAC: 10, PHY: 11, PBA No: FFFFFF-0FF
[    5.583550] systemd-udevd[182]: renamed network interface eth0 to eno1

I adjusted ethernet-static to point to eno1 but this failed as well.

...
INTERFACE=eno1
...

Each time though after boot was completed I was able to manually plumb my interface:

# ip link set dev eno1 up
# ip addr add 10.0.1.5/24 broadcast 10.0.1.255 dev eno1
# ip route add default via 10.0.1.1
#  ping -c 1 google.com
PING google.com (173.194.33.41) 56(84) bytes of data.
64 bytes from sea09s02-in-f9.1e100.net (173.194.33.41): icmp_seq=1 ttl=55 time=15.1 ms

At this point I switched to using the manual static method describe on the wiki page for Network Configure under manual static IP address:  https://wiki.archlinux.org/index.php/Ne … IP_address

# file: /etc/conf.d/network

interface=eno1
address=10.0.1.5
netmask=24
broadcast=10.0.1.255
gateway=10.0.1.1

file: /etc/systemd/system/network

[Unit]
Description=Wired Static IP Connectivity
Wants=network.target
Before=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/conf.d/network

ExecStart=/sbin/ip link set dev ${interface} up
ExecStart=/sbin/ip addr add ${address}/${netmask} broadcast ${broadcast} dev ${interface}
ExecStart=/sbin/ip route add default via ${gateway}

ExecStop=/sbin/ip addr flush dev ${interface}
ExecStop=/sbin/ip link set dev ${interface} down

[Install]
WantedBy=multi-user.target

I again found that using this network.service method failed to plumb my address on reboot (I also attempted using eth0).  I found that on one attempt though I did have successful plumb on eno1 but the interface still showed down:

# ip addr
...
2: eno1: <BROADCAST,MULTICAST,DOWBN,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether c8:60:00:9a:ce:6a brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.5/24 brd 10.0.1.255 scope global eno1

This is when I figured that the problem was probably that the network.target was not being held until udev finished executing, which was causing the interface name to change during network startup.

- Setting eth0 would fail because the name might change during or after network plumbing
- Setting eno1 would fail because the name wasn't changed when network was first starting

I created the following script and inserted into my network.service method:

#!/bin/bash
# waitforinterface - waits for set time/delay for udev to change if names
#    exists once the desired name is found, times out in TIMEOUT*DELAY seconds.

TIMEOUT=10
DELAY=3

. /etc/conf.d/network
for (( i=1; i<=$TIMEOUT; i++ ))
do
	RET="`iwconfig 2>&1 | grep ${interface}`"
	if [ "$RET" ]; then 
		exit 0
	fi
	echo "${interface} not found, sleeping for ${DELAY} on count ${i}"
	sleep $DELAY
done

file: /etc/systemd/system/network.service
....
EnvironmentFile=/etc/conf.d/network

ExecStart=/etc/conf.d/waitforinterface
ExecStart=/sbin/ip link set dev ${interface} up
ExecStart=/sbin/ip addr add ${address}/${netmask} broadcast ${broadcast} dev ${interface}
ExecStart=/sbin/ip route add default via ${gateway}
...

Network starts reliably (using the eno1 interface name) now that the network.service script waits for the if name to be changed.

This seemed to me like a bug that the interface name change process is introducing a a result of attempting to start network in parallel with udev's activities.  I'm not sure if this would be specific to my hardware, or if I just missed an obvious step that I'm fixing with a dirty hack.

I wanted to check before I suggested it was an actual bug though as I'm about 5 years out from actively messing with Linux and I'm coming from 2006 gentoo at that.

Thanks.

Last edited by blackraen (2013-01-28 03:30:17)

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: [SOLVED] Network start issues in new build (with systemd v197)

I had the same problem setting up a new server. If you check the log for the unit file (journalctl -x -u network) you should see a 'Cannot find device "eno1"', which is a slightly more direct way of confirming the nature of the problem.

As far as resolving it goes, I took a look at the dhcpcd@.service file, and it specifies:

BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

Where %i is the string between the @ character and the suffix for instantiated unit files like dhcpcd@ (instantiated unit files are used like 'systemctl enable dhcpcd@eno1').

The After line ensures that the specified unit file has finished starting up before this one begins running. You could use either BindsTo or Requires, with Requires ensuring that if this unit is started, so are the specified ones (and conversely that if the specified units are stopped, so is this one), and with BindsTo additionally ensuring that it will stop if the specified unit 'suddenly disappears'. To quote the manpage "Units can suddenly, unexpectedly disappear if a service terminates on its own choice, a device is unplugged or a mount point unmounted without involvement of systemd".
As noted here, the sys-subsystem-net-devices lines are meant to map to paths under /sys/subsystem, but this is not yet enabled in either mainline or Arch's binary kernel. udev supports the namespace internally, however, so this doesn't really matter too much.

Adapting that for the static network configuration service file example from the wiki, we get something along the lines of:

file: /etc/systemd/system/network@.service
[Unit]
Description=Static network configuration on %I
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/conf.d/network.%i

ExecStart=/sbin/ip link set dev %i up
ExecStart=/sbin/ip addr add ${address}/${netmask} broadcast ${broadcast} dev %i
ExecStart=/sbin/ip route add default via ${gateway}

ExecStop=/sbin/ip addr flush dev %i
ExecStop=/sbin/ip link set dev %i down

[Install]
WantedBy=multi-user.target

You can't use an EnvironmentFile in your [Unit] section, so it has to be an instantiated unit if you want the interface to be configurable without editing the service file itself. Your configuration file is basically the same, but with interface removed and now called network.%i. You could still use ${interface} instead of %i in the start/stop commands, but then you'd have the same information in two different locations. And of course, now you can properly use it with multiple separate interfaces.

This solved the issue for me, as far as I can tell (previously in around ~1/5 boots the network configuration would work, with this it's worked every time I've tried in - in this case, ~8 times), but I don't think this is really ideal. As far as changing unit files to deal with the problem, this is about as good as it gets, but at least to my mind this should be treated as a race condition in systemd rather than just the way things work. network.target should not be reached before udev has finished renaming the initial network interfaces. If this isn't treated as a bug, then every network-interface-configuring unit is going to need to do the same, which is a lot of duplicated configuration.

blackraen

Member

Registered: 2013-01-27

Posts: 3

Re: [SOLVED] Network start issues in new build (with systemd v197)

Thanks for the systemd information.  That is definitely a much more elegant way of handling it than my approach, I just wasn't sure how to go about it.  Also thanks for confirming the race condition.  I wasn't sure if I was making some kind of newbie arch mistake or if this was a legitimate problem I just happened to get to experience and identify in my first arch server build.

I added some additional debuggers to some of my existing scripts and found that on average udev was changing the device name between 50ms to 300ms after execution of my network.service script.  Definitely timed right to not have the right name in place for the first command, but easily changing the name while executing either the "ip addr" or "ip route" command.