You are not logged in.

#1 2020-12-21 22:47:56

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

[SOLVED] NetworkManager "manager: startup complete" with no conn...

Hello,

So, I have two computers at home which have NFS mounts setup via the "/etc/fstab" method as described in the Wiki. Let me put some names here:

  • ArchBox: Contains an outdated installation (1-2 months old)

  • KodiBox: Contains an updated installation (Updated on December, 17th)

Both of them were setup almost at the same time and they were working perfectly.
They are connecting to the same Wi-Fi network and the IP addresses are static.

Upon boot, the NFS mounts in the ArchBox machine work perfectly whereas the same mount points fail in the Kodibox machine (Network is unreachable).

As I was trying to find what may have changed I've pulled some information using the "journalctl" command.

For ArchBox:

-- Logs begin at Fri 2020-06-26 08:17:37 CEST, end at Mon 2020-12-21 22:26:08 CET. --
Dec 21 21:50:34 ArchBox systemd[1]: Starting Network Manager...
Dec 21 21:50:34 ArchBox NetworkManager[405]: <info>  [1608583834.6599] NetworkManager (version 1.26.4-1) is starting... (for the first time)
Dec 21 21:50:34 ArchBox NetworkManager[405]: <info>  [1608583834.6607] Read config: /etc/NetworkManager/NetworkManager.conf (etc: 20-connectivity.conf, wgpia.conf)
Dec 21 21:50:34 ArchBox systemd[1]: Started Network Manager.
...
Dec 21 21:50:48 ArchBox NetworkManager[405]: <info>  [1608583848.7993] device (wlp2s0): Activation: successful, device activated.
Dec 21 21:50:48 ArchBox NetworkManager[405]: <info>  [1608583848.8009] manager: NetworkManager state is now CONNECTED_GLOBAL
Dec 21 21:50:48 ArchBox NetworkManager[405]: <info>  [1608583848.8023] manager: startup complete

Nothing surprising here, 14 seconds to setup the network and the "NetworkManager-wait-onlne" service shows the same information:

Dec 21 21:50:34 ArchBox systemd[1]: Starting Network Manager Wait Online...
Dec 21 21:50:48 ArchBox systemd[1]: Finished Network Manager Wait Online.

And, of course, we can see that the mount process starts at 21:50:48:

Dec 21 21:50:48 ArchBox systemd[1]: Mounting /mnt/Games Backup...
Dec 21 21:50:50 ArchBox systemd[1]: Mounted /mnt/Games Backup.

For KodiBox:

-- Journal begins at Thu 2020-08-13 21:37:41 CEST, ends at Mon 2020-12-21 22:25:44 CET. --
Dec 21 21:54:39 KodiBox systemd[1]: Starting Network Manager...
Dec 21 21:54:39 KodiBox NetworkManager[459]: <info>  [1608584079.9736] NetworkManager (version 1.26.4-1) is starting... (for the first time)
Dec 21 21:54:39 KodiBox NetworkManager[459]: <info>  [1608584079.9737] Read config: /etc/NetworkManager/NetworkManager.conf (etc: 20-connectivity.conf, wgpia.conf)
Dec 21 21:54:39 KodiBox systemd[1]: Started Network Manager.
...
Dec 21 21:54:40 KodiBox NetworkManager[459]: <info>  [1608584080.0737] ovsdb: Non foi posíbel conectar: Non hai tal ficheiro ou directorio
Dec 21 21:54:40 KodiBox NetworkManager[459]: <info>  [1608584080.0741] manager: startup complete
...
Dec 21 21:54:47 KodiBox NetworkManager[459]: <info>  [1608584087.1419] device (wlp5s0): Activation: successful, device activated.
Dec 21 21:54:47 KodiBox NetworkManager[459]: <info>  [1608584087.1424] manager: NetworkManager state is now CONNECTED_GLOBAL

As you can see, in this case, the "startup complete" message is displayed 7 seconds before the device is completely activated and the network is online. Which, in turn, is the cause of the NFS failures... And, of course, the "wait-online" service shows the same timestamps:

Dec 21 21:54:39 KodiBox systemd[1]: Starting Network Manager Wait Online...
Dec 21 21:54:40 KodiBox systemd[1]: Finished Network Manager Wait Online.

And the mounting process fails at 21:54:40:

Dec 21 21:54:40 KodiBox systemd[1]: Mounting /mnt/Games Backup...
Dec 21 21:54:40 KodiBox mount[487]: mount.nfs: Network is unreachable
Dec 21 21:54:40 KodiBox systemd[1]: mnt-Games\x20Backup.mount: Mount process exited, code=exited, status=32/n/a
Dec 21 21:54:40 KodiBox systemd[1]: mnt-Games\x20Backup.mount: Failed with result 'exit-code'.
Dec 21 21:54:40 KodiBox systemd[1]: Failed to mount /mnt/Games Backup.

I'm at a loss here. Since I've not touched the configuration files for NetworkManager and /etc/fstab, I don't understand why it's suddenly failing to mount the remote file systems  properly.

Any hint or guide? I can provide the full journalctl boot logs if you find them useful smile

Cheers,
Condex

Edit:

  • Original title: NetworkManager "manager: startup complete" with no connection setup

  • See post #11 below for the solution.

Last edited by Condex (2021-01-10 12:17:14)


« Carpe diem quam minimum credula postero. »

Offline

#2 2020-12-22 08:33:10

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

Dec 21 21:54:40 KodiBox NetworkManager[459]: <info>  [1608584080.0737] ovsdb: Non foi posíbel conectar: Non hai tal ficheiro ou directorio

Thanks to me being fluent in Galic… google translate, I'm gonna say disable https://wiki.archlinux.org/index.php/Ne … domization

Also consider a lazy mount, eg. using systemd automounting (the device is mounted on first access)

Offline

#3 2020-12-22 21:53:10

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

Hi Seth, thanks for replying...

First and foremost, sorry about that message in Galician... I should've paid more attention to the message before posting wink

I've tested the first option, disabling "MAC address randomization" as stated in your link, but that did not do the trick:

condex@KodiBox ~ $ cat /etc/NetworkManager/conf.d/wifi_rand_mac.conf
[device]
wifi.scan-rand-mac-address=no

The "NetworkManager-wait-online" service still returns in less than a second:

condex@KodiBox ~ $ sudo journalctl -u NetworkManager-wait-online -b
-- Journal begins at Fri 2020-08-14 13:57:11 CEST, ends at Tue 2020-12-22 22:34:03 CET. --
Dec 22 22:17:50 KodiBox systemd[1]: Starting Network Manager Wait Online...
Dec 22 22:17:50 KodiBox systemd[1]: Finished Network Manager Wait Online.

And, in case it may help you, I've posted the full output of the "sudo journalctl -u NetworkManager -b" command here.

It's as if NetworkManager was considering that the network was online once the "localhost" link is up...


seth wrote:

Also consider a lazy mount, eg. using systemd automounting (the device is mounted on first access)

I've given this option a look, and it works for one of the mount points, as it does not need to be available on boot. Let me show you the lines I have in my "/etc/fstab", just in case:

NAS-DS218:/volume1/Games                        /mnt/Games\040Backup    nfs     _netdev,sync,noauto,x-systemd.automount  0 0
NAS-DS218:/volume1/Multimedia                   /mnt/Multimedia         nfs     _netdev,sync,noauto,x-systemd.automount  0 0

If the "Games" mount point is not available on boot, it doesn't really matter. As I use it once I'm logged in.

In the case of the "Multimedia" mount point, I have set it as "noauto", but as soon as the "Plex Media Server" starts, it will try to access this directory which, in turn, will activate the "automounter", that will fail several times because the network is not yet online. Of course, once the network is "really online", the remote file system gets properly mounted. Full journalctl details here.

This, in turn, has made me notice that the "Plex Media Server" service is being started as well before the network is really online:

-- Journal begins at Fri 2020-08-14 13:57:11 CEST, ends at Tue 2020-12-22 22:43:29 CET. --
Dec 22 22:17:50 KodiBox systemd[1]: Started Plex Media Server.
Dec 22 22:17:51 KodiBox Plex Media Server[453]: sendto: Network is unreachable
...

And last, but not least, I've updated the system, just in case an update in one of the packages solved the issue... No luck though sad

Hope this helps wink


« Carpe diem quam minimum credula postero. »

Offline

#4 2020-12-22 22:39:43

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

What's the output of "ip a"? Do you have any local switches? Why does NM try to contact openvswitch?

Offline

#5 2020-12-23 22:44:40

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

Hi again seth,

Thanks again for taking time trying to help me troubleshoot this poltergeist... sad

The ouput of "ip a" is:

condex@KodiBox ~ $ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: wlp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6c:6a:77:3c:27:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.6/24 brd 192.168.0.255 scope global noprefixroute wlp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::6e6a:77ff:fe3c:277e/64 scope link
       valid_lft forever preferred_lft forever

There is no Ethernet device, because I have disabled it in the BIOS, as I can't have a network cable laying around wink

Concerning the configuration of the network, I have:

  • The router/cable modem from the ISP

  • A NAS connected directly to the router/cable modem (Synology DS-218, static IP)

  • A switch connected to the router/cable modem (TP-Link, 5 port, Gigabit Network)

  • This computer (KodiBox) connected to the router using Wi-Fi (5GHz, static IP)

  • Another computer (ArchBox) connected to the router using Wi-Fi (5GHz, static IP). This is the computer I've used to check that the mount points work with an "old" ArchLinux install, and it's powered off as well.

  • A mobile phone and a tablet, both connected to the same Wi-Fi network but using dynamic IP addresses (DHCP)

seth wrote:

Why does NM try to contact openvswitch?

That's a good question and I don't have the slightest idea... sad

Your mention of the "switch" gave me an idea and I've enabled again the Ethernet port in the BIOS (with no cable attached) and, voilà, NetworkManager is behaving as expected again:

I have noticed that the Wi-Fi device name has changed, it is now "wlp6s0" instead of "wlp5s0". The message about "ovsdb" is still there, though. My guess is that the "ovsdb" message is not linked to the problem with NM.

And, of course, the last test, what would happen if I went to the BIOS and disabled the Ethernet port? You may have guessed it, the Wi-Fi device becomes "wlp5s0", and it doesn't work again sad

For the time being, I'm leaving the Ethernet card enabled to avoid the mounting issues smile

Cheers,
Condex cool


« Carpe diem quam minimum credula postero. »

Offline

#6 2020-12-26 15:18:05

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

Sorry for the delay, this ended in googles spam filter.

The name change is not really suprising (you get another pci device and everthing behind shifts by one)
Can you post a complete system journal for a bad boot ("sudo journalctl -b -1" for the previous boot)?
And check whether there's a stale entry for the lost ethernet NIC in your NM config.

Offline

#7 2020-12-31 12:57:13

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

seth wrote:

Sorry for the delay, this ended in googles spam filter.

No worries. As you can see, with Christmas and all, I didn't have much time to look into the issue wink

seth wrote:

The name change is not really suprising (you get another pci device and everthing behind shifts by one)
Can you post a complete system journal for a bad boot ("sudo journalctl -b -1" for the previous boot)?
And check whether there's a stale entry for the lost ethernet NIC in your NM config.

Here's what I did, if I understood your request:

  • Reboot the computer and disable the wired NIC in the BIOS.

  • Since I had setup the wired LAN in NM, there was a leftover 'Wired LAN.nmconnection' file, which I removed, then rebooted the computer again to start clean. The contents of /etc/NetworkManager are here.

  • The full journal from this "No wired NIC" log is here.

  • Since the problem with NM was there again, another reboot to enable the wired NIC, which led to the NM journal you can read here. This time around, in order to avoid issues, I had not changed the configuration in NM, so the configuration files are the same as the ones in the 2nd item.

Maybe if I disable the wired NIC and setup NM again from scratch: Reboot to disable NIC => Delete Wi-Fi configuration => Reboot => Configure WiFi again... Maybe that can be a possible solution?

Cheers!
Condex- cool


« Carpe diem quam minimum credula postero. »

Offline

#8 2020-12-31 15:25:21

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

It looks like the WiFi chip starts up rfkill'd and as a result, w/o the wired NIC the *only* interface is the loopback one and as soon as that's "connected", NM is like "mission accomplished" (Bush 43 style… ;-)

What if you pass "rfkill.default_state=1" or enable the rfkill-unblock service, https://bbs.archlinux.org/viewtopic.php … 8#p1784478 ?

Offline

#9 2021-01-05 22:49:25

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

Hi again,

Looks like that rfkill thingy is not the problem here. Here's what I did:

  • Reboot the Pc to disable the wired NIC, then remove the leftover "nmconnection" file for the wired LAN.

  • Add the kernel parameter to "/etc/default/grub" and rebuild the configuration file for GRUB using "grub-mkconfig". This leads to this boot log.

  • Since the previous method did not work, I went with the "rfkill-unblock service", which rendered similar results.

Another option tested: Delete the Wi-Fi connection => Reboot => Add the Wi-Fi connection again => Reboot... Same result sad

Looks like the only way I can get NM to behave properly is to leave the Wired NIC enabled hmm


« Carpe diem quam minimum credula postero. »

Offline

#10 2021-01-05 23:20:06

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

Still same problem.
nm-online sees the loopback device before the wifi-chip is active and goes like "job done".

This doesn't seem to be a new issue… https://lists.fedoraproject.org/archive … LFDA3TP7N/

How attached are you to networkmanager? (in relation to keeping the wired NIC active)

Offline

#11 2021-01-06 15:29:46

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

seth wrote:

How attached are you to networkmanager? (in relation to keeping the wired NIC active)

I don't mind replacing NM with another program. But I kinda like NM. It's easy to use and it integrates well in KDE via the plasma-nm applet wink

seth wrote:

Still same problem.
nm-online sees the loopback device before the wifi-chip is active and goes like "job done".

This doesn't seem to be a new issue… https://lists.fedoraproject.org/archive … LFDA3TP7N/

That's an interesting link you've found. It looks like the problem with the "wait-online" service is old big_smile

Nevertheless, having read the whole thread, I decided to give a try to the option of overriding the "NetworkManager-wait-online" service file to remove the "-s" argument from the "ExecStart" line. A quick search in the wiki to find how to create that "override" file, leaves me with the following contents there:

condex@KodiBox ~ $ sudo cat /etc/systemd/system/NetworkManager-wait-online.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/nm-online -q

A reboot to disable the NIC and it looks like the override file seems to have done the trick...

And here I am, eight reboots later, still no wired NIC and 0 failures:

We can still see the "startup complete" message at 16:14:18, but the "wait-online" service returns at 16:14:24, right when NM shows "manager: NetworkManager state is now CONNECTED_GLOBAL" in the log. And the attempt to mount the NFS mount point is executed as well at 16:14:25.

Since I was touching some "systemd" files, I've also overridden the "plexmediaserver" service file to force it to wait for the computer to be online, forcing it to start at 16:14:24 which, in turn, causes the "mount" command at 16:14:25. The wiki explains how to do it (just in case someone finds it useful) wink

So, unless something else breaks in the next reboots, I would say that the problem is solved. Don't you think?

Thanks a lot for your help, seth!

Cheers,
Condex- cool

Last edited by Condex (2021-01-06 15:32:29)


« Carpe diem quam minimum credula postero. »

Offline

#12 2021-01-06 16:10:16

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

The service sets that flag by default?
That's braindea… good to know that you fixed it and thanks for the reminder why I'm not a fan of wannabesmart network managers ;-)

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Edit: might be worth filing a bug to patch that away or edit the wiki. A "wait online" service that doesn't wait to be online. Awesome.

Last edited by seth (2021-01-06 16:11:43)

Offline

#13 2021-01-10 12:13:50

Condex
Member
From: Sol III - Sol System
Registered: 2008-02-04
Posts: 38

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

seth wrote:

The service sets that flag by default?
That's braindea… good to know that you fixed it and thanks for the reminder why I'm not a fan of wannabesmart network managers ;-)

Yep, It waits for the NM service to output "startup complete", so one might say that it works as intended. In my case, it's NM that it's saying that the startup is complete before connecting to the WiFi xD

seth wrote:

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Will do in a few, I was trying to find a way to link the post with the solution (#11), but can't seem to find a simple way to get the link. hmm

seth wrote:

Edit: might be worth filing a bug to patch that away or edit the wiki. A "wait online" service that doesn't wait to be online. Awesome.

I'll take a look at the wiki, maybe post a message there for other users. It's strange that no one else has found the same issue. Other than the thread in the link you sent a few posts ago.

Thanks again for your help!

Cheers,
Condex- cool

Last edited by Condex (2021-01-10 12:14:13)


« Carpe diem quam minimum credula postero. »

Offline

#14 2021-01-10 12:16:35

seth
Member
Registered: 2012-09-03
Posts: 21,122

Re: [SOLVED] NetworkManager "manager: startup complete" with no conn...

https://bbs.archlinux.org/viewtopic.php … 2#p1948182

It's behind the date in the header.

Offline

Board footer

Powered by FluxBB