Boot stalls at "A start job is running for Network Name Resolution"

EndUserOnly · 2024-05-26 22:14:51

There is no official package using icu. I am updating AUR first, just released a new qt5-webkit (the offender). Then I will do a proper update on official repository.

Helping myself to a slice of humble pie - Seth was correct. Like I said, from now on I will try a different approach.

To answer your question the programs impacted by icu in the main repository are gspell, harfbuzz-icu, and raptor - so says pacman. One of them apparently cause a major failure. But that is above my level of expertise - no need to explain.

Last edited by EndUserOnly (2024-05-26 22:29:49)

ilf0 · 2024-05-28 07:58:10

Can confirm this bug.

% lsmod | grep r816
r8169                 126976  0
mdio_devres            12288  1 r8169
libphy                241664  3 r8169,mdio_devres,realtek

What's the correct solution for this?

There is r8168-lts in Extra and r8168 in AUR, but the latter is orphaned.

V1del · 2024-05-28 08:24:10

https://bbs.archlinux.org/viewtopic.php … 8#p2173588

Try adding r8169 to the MODULES=() array in your /etc/mkinitcpio.conf and regenerating the initramfs. Which networking daemon are you using?

ilf0 · 2024-05-28 08:49:14

V1del wrote:

Which networking daemon are you using?

systemd-networkd.

I'd rather not implement workarounds until the exact cause is known and the proper solution available.
Afterall, it's not reproducable at every boot, it only happens at some boots.

perseus · 2024-05-28 10:14:44

seth wrote:

Different chip from #7 but the common pattern is that systemd-networkd doesn't get an r8169 driven NIC up.
For both the r8169 module load late, what if you add it to the initramfs?

Thank you seth.
This solves the problem for me.
In case anyone else with a venerable Gigabyte H170-HD3-CF motherboard comes along, the manual does not tell me precisely which variant I have, but inxi reports
Device-1: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet driver: r8169
The box in question has been up pretty much 24/7 since I assembled it 8 years ago, running every arch kernel since then, and has had no problems of this kind until I installed 6.9.2-arch1-1.

seth · 2024-05-28 14:33:39

ilf0 wrote:

I'd rather not implement workarounds until the exact cause is known and the proper solution available.
Afterall, it's not reproducable at every boot, it only happens at some boots.

This is called a "race condition", the exact cause is that systemd on all levels just optimistically does stuff, making an ass out of u and me and then fails hard when its assumptions don't pan out.
This is also why https://wiki.archlinux.org/title/Kernel … _KMS_start is suggested, basically just start the entire system in the initramfs so systemd won't shit itself

The proper solutions are to either pander to its shortcomings or avoid it (ie. use NM, netctl or just dhcpcd)
You can also (probably waste your time and) file an upstream bug.

@perseus, did you also update systemd at the time?
Do you get a different outcome w/ the LTS kernel?
Do you also have an encrypted system?
A lot of the boot (and r8169 loading) delay for @libertepourmoi comes from

Mai 25 17:58:22 archlinux systemd-tty-ask-password-agent[251]: Password query on /dev/tty1 finished successfully.
Mai 25 17:58:23 archlinux systemd-cryptsetup[248]: Asking FIDO2 token for authentication.

perseus · 2024-05-28 17:15:38

Hi @seth,

Short answers:
1. "Did you also update systemd at the time?"
Yes. systemd 255.6-2 and Kernel 6.9.2-arch1-1 arrived on the same day. As it happens, the pacman -Syu that delivered systemd came an hour or two before the kernel.
2. I am afraid that I don't keep an LTS kernel.
3. No. The system is not encrypted. I have one encrypted directory, but that is in a user directory and encryption/decryption is manual after I start X.

A bit more context if anyone is interested:
As I said, this box runs 24/7. I have never reinstalled, so I guess that it is the product of at least 3,000 pacman -Syu updates.

I reboot only for kernel updates, and power down once every month or two to clean fans and filters.

This bug hit as I rebooted from the systemd / kernel updates. It twice went into an endless "A start job is running for network resolution" and on the third hard reset I got lucky.

I added the relevant entry to mkinitcpio.conf today, and rebooted cleanly a few times to test.

Last edited by perseus (2024-05-28 18:07:23)

Steven Loving · 2024-05-29 17:21:20

Just to give a report from the field, in hopes that it helps someone:

I was experiencing this bug for the past few days. I would get the "A start job..." thing about 50% of the time. I do have a Gigabyte motherboard which uses an ethernet chipset which uses the r8169 driver. I also use systemd-networkd.

At first, I switched back to NetworkManager (which I had used until recently, so I was familiar with it). The bug seemed to go away.

And then, yesterday morning, I switched back to systemd-networkd and I put 'r8169' in the MODULES field of mkinitcpio.conf and rebuilt the initramfs. The bug also went away in this situation.

So yeah, this bug seems to afflict people with a Gigabyte motherboard (or rather, people who have an NIC chipset which uses that driver) and use systemd-networkd. The solution seems to be early loading the r8169 module.

Disclaimer: I didn't do any rigorous scientific testing here; after each solution, I only rebooted like 3 or 4 times. And knowing my luck, the bug will come back tomorrow and i'll look like an idiot.

seth · 2024-05-29 19:29:45

Since it would also make me look like I'd not understand what's going on, the bug better don't dare.

If anyone here has older journals around, did the network driver always load this late-ish?

libertepourmoi · 2024-05-29 19:54:52

I would say so.

Just did a random 'journalctl -b -15', and turns out that was the boot right before the update of systemd came in. See for yourself, http://0x0.st/XN24.txt .

V1del · 2024-05-30 19:54:31

FWIW I ran into not being able to start my session and anything trying to access the network/network devices (e.g. a simple ip addr) stalling up. REISUBing out of that situation gives some traces in the log Where

[  246.743655] INFO: task NetworkManager:912 blocked for more than 122 seconds.
[  246.743657]       Not tainted 6.9.2-zen1-1-zen #1
[  246.743658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  246.743659] task:NetworkManager  state:D stack:0     pid:912   tgid:912   ppid:1      flags:0x00004002
[  246.743661] Call Trace:
[  246.743662]  <TASK>
[  246.743664]  __schedule+0x3e5/0x19f0
[  246.743666]  ? __get_random_u32_below+0x16/0x70
[  246.743670]  ? srso_return_thunk+0x5/0x5f
[  246.743674]  schedule_preempt_disabled+0x2e/0x100
[  246.743676]  rwsem_down_read_slowpath+0x17d/0x510
[  246.743681]  ? srso_return_thunk+0x5/0x5f
[  246.743684]  down_read+0x48/0xa0
[  246.743686]  led_trigger_register+0xe6/0x1a0
[  246.743689]  phy_led_triggers_register+0xee/0x25c [libphy 1b5d4b493f77693a904b79f142e947d8fad79821]
[  246.743706]  phy_attach_direct+0x3a5/0x3b0 [libphy 1b5d4b493f77693a904b79f142e947d8fad79821]
[  246.743720]  ? __pfx_r8169_phylink_handler+0x10/0x10 [r8169 fc47a7bdf640aa522bf5f88e4f48eaf00dd898d3]
[  246.743727]  phy_connect_direct+0x25/0x70 [libphy 1b5d4b493f77693a904b79f142e947d8fad79821]
[  246.743739]  rtl_open+0x34b/0x5f0 [r8169 fc47a7bdf640aa522bf5f88e4f48eaf00dd898d3]
[  246.743747]  __dev_open+0x136/0x200
[  246.743751]  __dev_change_flags+0x21e/0x2c0
[  246.743754]  dev_change_flags+0x43/0x270
[  246.743757]  do_setlink+0x802/0x1380
[  246.743760]  ? srso_return_thunk+0x5/0x5f
[  246.743762]  ? __kmalloc_node_track_caller+0x23e/0x620
[  246.743764]  ? srso_return_thunk+0x5/0x5f
[  246.743766]  ? __nla_reserve+0x3c/0x50
[  246.743769]  ? pskb_expand_head+0x95/0x420
[  246.743773]  ? kmalloc_reserve+0x62/0xf0
[  246.743775]  ? srso_return_thunk+0x5/0x5f
[  246.743777]  ? srso_return_thunk+0x5/0x5f
[  246.743779]  ? security_sock_rcv_skb+0x38/0x50
[  246.743782]  ? srso_return_thunk+0x5/0x5f
[  246.743783]  ? sk_filter_trim_cap+0x13b/0x260
[  246.743786]  ? srso_return_thunk+0x5/0x5f
[  246.743788]  ? pskb_expand_head+0x1f0/0x420
[  246.743790]  ? srso_return_thunk+0x5/0x5f
[  246.743792]  ? kmalloc_trace+0x124/0x480
[  246.743794]  ? srso_return_thunk+0x5/0x5f
[  246.743796]  ? __nla_parse+0x43/0x280
[  246.743799]  rtnl_newlink+0x641/0xc40
[  246.743806]  ? srso_return_thunk+0x5/0x5f
[  246.743808]  ? srso_return_thunk+0x5/0x5f
[  246.743810]  ? security_capable+0x44/0x70
[  246.743813]  rtnetlink_rcv_msg+0x14b/0x500
[  246.743819]  ? srso_return_thunk+0x5/0x5f
[  246.743821]  ? __rtnl_unlock+0x37/0x60
[  246.743823]  ? srso_return_thunk+0x5/0x5f
[  246.743825]  ? netdev_run_todo+0x64/0x5a0
[  246.743827]  ? srso_return_thunk+0x5/0x5f
[  246.743829]  ? __rtnl_unlock+0x37/0x60
[  246.743830]  ? srso_return_thunk+0x5/0x5f
[  246.743833]  ? netdev_run_todo+0x64/0x5a0
[  246.743835]  ? srso_return_thunk+0x5/0x5f
[  246.743837]  ? security_capable+0x44/0x70
[  246.743840]  ? srso_return_thunk+0x5/0x5f
[  246.743842]  ? rtnetlink_rcv_msg+0x15b/0x500
[  246.743844]  ? xa_load+0x5d/0x150
[  246.743847]  ? srso_return_thunk+0x5/0x5f
[  246.743849]  ? memcg_list_lru_alloc+0xab/0x3d0
[  246.743853]  ? srso_return_thunk+0x5/0x5f
[  246.743855]  ? try_charge_memcg+0x93/0xa90
[  246.743857]  ? srso_return_thunk+0x5/0x5f
[  246.743859]  ? srso_return_thunk+0x5/0x5f
[  246.743861]  ? _copy_to_iter+0x8b/0x560
[  246.743864]  ? srso_return_thunk+0x5/0x5f
[  246.743866]  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
[  246.743869]  netlink_rcv_skb+0x53/0x100
[  246.743875]  netlink_unicast+0x306/0x4a0
[  246.743879]  netlink_sendmsg+0x1fe/0x430
[  246.743883]  ____sys_sendmsg+0x398/0x3d0
[  246.743887]  __sys_sendmsg+0x1de/0x2c0
[  246.743898]  do_syscall_64+0x83/0x190
[  246.743900]  ? srso_return_thunk+0x5/0x5f
[  246.743902]  ? proc_sys_call_handler+0x132/0x350
[  246.743905]  ? srso_return_thunk+0x5/0x5f
[  246.743907]  ? vfs_write+0x2c6/0x4a0
[  246.743911]  ? srso_return_thunk+0x5/0x5f
[  246.743914]  ? srso_return_thunk+0x5/0x5f
[  246.743916]  ? syscall_exit_to_user_mode+0x75/0x1f0
[  246.743918]  ? srso_return_thunk+0x5/0x5f
[  246.743920]  ? do_syscall_64+0x8f/0x190
[  246.743924]  ? srso_return_thunk+0x5/0x5f
[  246.743926]  ? __x64_sys_openat+0x1f5/0x230
[  246.743928]  ? srso_return_thunk+0x5/0x5f
[  246.743930]  ? syscall_exit_to_user_mode+0x75/0x1f0
[  246.743932]  ? srso_return_thunk+0x5/0x5f
[  246.743934]  ? do_syscall_64+0x8f/0x190
[  246.743937]  ? srso_return_thunk+0x5/0x5f
[  246.743939]  ? mntput+0x65/0x3d0
[  246.743943]  ? srso_return_thunk+0x5/0x5f
[  246.743945]  ? kmem_cache_free+0x43f/0x490
[  246.743948]  ? srso_return_thunk+0x5/0x5f
[  246.743949]  ? syscall_exit_to_user_mode+0x75/0x1f0
[  246.743952]  ? srso_return_thunk+0x5/0x5f
[  246.743954]  ? do_syscall_64+0x8f/0x190
[  246.743956]  ? srso_return_thunk+0x5/0x5f
[  246.743958]  ? do_syscall_64+0x8f/0x190
[  246.743959]  ? irq_exit_rcu+0x53/0xc0
[  246.743962]  ? srso_return_thunk+0x5/0x5f
[  246.743964]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  246.743966] RIP: 0033:0x7312cbb3647b
[  246.743969] RSP: 002b:00007ffe35e2c620 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
[  246.743971] RAX: ffffffffffffffda RBX: 000000000000000c RCX: 00007312cbb3647b
[  246.743973] RDX: 0000000000000000 RSI: 00007ffe35e2c660 RDI: 000000000000000d
[  246.743974] RBP: 00007ffe35e2c640 R08: 0000000000000000 R09: 0000000000000000
[  246.743976] R10: 0000000000000000 R11: 0000000000000293 R12: 00005899e5342500
[  246.743977] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
[  246.743981]  </TASK>

reads somewhat interesting... There are also two bugfixes in r8169 in 6.9.3 so things might already be fixed... or all of the LED enablement code leading to some issue... I don't use the module and don't have time to debug further, have just blacklisted it for now since I don't actively use that port.

sinsplitter · 2024-06-03 17:37:07

Hi, I am also experiencing this bug.

As others say, the startup stalls at "A start job is running for Network Name Resolution" seemingly at random. Sometimes it boots up fine, other times it gets stuck in an infinite loop and I am forced to power off with the physical power button.

It is not particularly common. It has happened to me four times. Twice in a row on 5/20/24 and twice more in a row again on 6/3/24 (today).

I am only running Arch. No dualbooting.
My internet connection is via wifi.

Motherboard: Gigabyte B650 Gaming X AX V2
Wireless Communication Module: Realtek® Wi-Fi 6E RTL8852CE (PCB rev. 1.0)

(seems there are two possible wireless modules for my mobo according to the manual, but I checked my box and it says PCB rev. 1.0 so it must be the Realtek one)

Steven Loving · 2024-06-03 18:11:37

sinsplitter,

Have you tried early-adding the network driver to the initramfs?

It involves putting the name of the driver in the "MODULES=()" field in /etc/mkinitcpio.conf, and rebuilding the inintramfs with "mkinitcpio -p linux"

I don't know which module to recommend to you, because in my case, it was the driver for my wired ethernet card, namely the 'r8169' driver

I think I also experienced the same bug as you are: machine randomly won't boot, sometimes it didn't, sometimes it did; the solution was to early-load the network driver via mkinitcpio.conf

Now, to which module you want to early-load? I'm not sure what to tell you. If you are using the running machine, you can run 'lsmod' and look for the relevant kernel module. Alternatively, you can run 'inxi -F' and look for your wireless device. It should display the relevant driver.

For instance, when I run 'inxi -F', I get:

Network:
  Device-1: Intel Wireless 7265 driver: iwlwifi

So if this applies to you, try adding 'iwlwifi' to your mkinitcpio.conf and rebuilding the initramfs

seth · 2024-06-03 20:23:39

Wireless Communication Module: Realtek® Wi-Fi 6E RTL8852CE (PCB rev. 1.0)

lspci -k

But the triggered (known) deadlock seems to only affect the r8169 driver, it does not necessarily matter which NIC ultimately provides a connection. systemd only has to bring the nic up to cause this.

Archttila · 2024-06-04 14:48:02

same problem here

Deskmini X300

seth · 2024-06-04 14:49:39

Same solutions/workarounds.
https://wiki.archlinux.org/title/Genera … mpty_posts

Archttila · 2024-06-04 15:22:56

Unfortunately I don't speak the language very well, but I understand the problem, sorry!

seth · 2024-06-04 15:34:12

https://www.deepl.com/translator#en/hu/ … 0initramfs.

Archttila · 2024-06-04 15:52:43

I've already added it, it looks good so far, it works! Thank you!

loqs · 2024-06-04 16:29:52

V1del wrote:

There are also two bugfixes in r8169 in 6.9.3 so things might already be fixed... or all of the LED enablement code leading to some issue... I don't use the module and don't have time to debug further, have just blacklisted it for now since I don't actively use that port.

I believe the issue is still outstanding at least from my reading of https://lore.kernel.org/regressions/d83 … edhat.com/ and the current suggested fix for 6.9 is to revert 66601a29bb23b61f9676f51c0b4b78b38c601536.
Edit:
Upstream revert https://git.kernel.org/pub/scm/linux/ke … 115a9108aa

Last edited by loqs (2024-06-16 19:27:26)

bvtthead · 2024-06-29 13:43:29

6.9.7 has the revert, and is now out of testing and in core.

Location of the revert.

Last edited by bvtthead (2024-06-29 13:44:40)

Tuxfan81 · 2025-04-18 23:25:03

I encountered this problem in 2025 on my second attempt to install Arch, waiting six minutes every time (= the global timeout). Fixed when I discovered systemd-neworkd and NetworkManager were both enabled and disabled one of them.

Lone_Wolf · 2025-04-19 11:46:22

Conflicting network services cause issues 99.9% of the time, but this thread is almost 10 months old.
Also the thread starter hasn't visited the forum since june last year.

Please read https://wiki.archlinux.org/title/Genera … bumping%22

Closing this old thread.

Arch Linux

#26 2024-05-26 22:14:51

Re: Boot stalls at "A start job is running for Network Name Resolution"

#27 2024-05-28 07:58:10

Re: Boot stalls at "A start job is running for Network Name Resolution"

#28 2024-05-28 08:24:10

Re: Boot stalls at "A start job is running for Network Name Resolution"

#29 2024-05-28 08:49:14

Re: Boot stalls at "A start job is running for Network Name Resolution"

#30 2024-05-28 10:14:44

Re: Boot stalls at "A start job is running for Network Name Resolution"

#31 2024-05-28 14:33:39

Re: Boot stalls at "A start job is running for Network Name Resolution"

#32 2024-05-28 17:15:38

Re: Boot stalls at "A start job is running for Network Name Resolution"

#33 2024-05-29 17:21:20

Re: Boot stalls at "A start job is running for Network Name Resolution"

#34 2024-05-29 19:29:45

Re: Boot stalls at "A start job is running for Network Name Resolution"

#35 2024-05-29 19:54:52

Re: Boot stalls at "A start job is running for Network Name Resolution"

#36 2024-05-30 19:54:31

Re: Boot stalls at "A start job is running for Network Name Resolution"

#37 2024-06-03 17:37:07

Re: Boot stalls at "A start job is running for Network Name Resolution"

#38 2024-06-03 18:11:37

Re: Boot stalls at "A start job is running for Network Name Resolution"

#39 2024-06-03 20:23:39

Re: Boot stalls at "A start job is running for Network Name Resolution"

#40 2024-06-04 14:48:02

Re: Boot stalls at "A start job is running for Network Name Resolution"

#41 2024-06-04 14:49:39

Re: Boot stalls at "A start job is running for Network Name Resolution"

#42 2024-06-04 15:22:56

Re: Boot stalls at "A start job is running for Network Name Resolution"

#43 2024-06-04 15:34:12

Re: Boot stalls at "A start job is running for Network Name Resolution"

#44 2024-06-04 15:52:43

Re: Boot stalls at "A start job is running for Network Name Resolution"

#45 2024-06-04 16:29:52

Re: Boot stalls at "A start job is running for Network Name Resolution"

#46 2024-06-29 13:43:29

Re: Boot stalls at "A start job is running for Network Name Resolution"

#47 2025-04-18 23:25:03

Re: Boot stalls at "A start job is running for Network Name Resolution"

#48 2025-04-19 11:46:22

Re: Boot stalls at "A start job is running for Network Name Resolution"

Board footer