Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

Toolybird · 2022-01-28 21:50:13

> The status looks like this:

Hmm, looks basically fine. In my case I'm using systemd-networkd and systemd-resolved so my output is slightly different. It would be interesting to see a few other logs for comparison. Here's my "journalctl -b -u libvirtd" output immediately after boot:

Jan 29 06:55:03 ryzen systemd[1]: Starting Virtualization daemon...
Jan 29 06:55:03 ryzen systemd[1]: Started Virtualization daemon.
Jan 29 06:55:04 ryzen dnsmasq[1259]: started, version 2.86 cachesize 150
Jan 29 06:55:04 ryzen dnsmasq[1259]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cr>
Jan 29 06:55:04 ryzen dnsmasq-dhcp[1259]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Jan 29 06:55:04 ryzen dnsmasq-dhcp[1259]: DHCP, sockets bound exclusively to interface virbr0
Jan 29 06:55:04 ryzen dnsmasq[1259]: reading /etc/resolv.conf
Jan 29 06:55:04 ryzen dnsmasq[1259]: using nameserver 127.0.0.53#53
Jan 29 06:55:04 ryzen dnsmasq[1259]: read /etc/hosts - 3 addresses
Jan 29 06:55:04 ryzen dnsmasq[1259]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 29 06:55:04 ryzen dnsmasq-dhcp[1259]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jan 29 06:57:03 ryzen systemd[1]: libvirtd.service: Deactivated successfully.
Jan 29 06:57:03 ryzen systemd[1]: libvirtd.service: Unit process 1259 (dnsmasq) remains running after unit stopped.
Jan 29 06:57:03 ryzen systemd[1]: libvirtd.service: Unit process 1260 (dnsmasq) remains running after unit stopped.

Here's the same but after booting a VM:

Jan 29 07:41:09 ryzen systemd[1]: libvirtd.service: Found left-over process 1259 (dnsmasq) in control group while starting unit. Ignoring.
Jan 29 07:41:09 ryzen systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 29 07:41:09 ryzen systemd[1]: libvirtd.service: Found left-over process 1260 (dnsmasq) in control group while starting unit. Ignoring.
Jan 29 07:41:09 ryzen systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 29 07:41:09 ryzen systemd[1]: Starting Virtualization daemon...
Jan 29 07:41:09 ryzen systemd[1]: Started Virtualization daemon.
Jan 29 07:41:09 ryzen dnsmasq[1259]: read /etc/hosts - 3 addresses
Jan 29 07:41:09 ryzen dnsmasq[1259]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 29 07:41:09 ryzen dnsmasq-dhcp[1259]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPDISCOVER(virbr0) 52:54:00:22:e2:25
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPOFFER(virbr0) 192.168.122.87 52:54:00:22:e2:25
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPREQUEST(virbr0) 192.168.122.87 52:54:00:22:e2:25
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPACK(virbr0) 192.168.122.87 52:54:00:22:e2:25 archiso
Jan 29 07:46:27 ryzen systemd[1]: libvirtd.service: Deactivated successfully.
Jan 29 07:46:27 ryzen systemd[1]: libvirtd.service: Unit process 1259 (dnsmasq) remains running after unit stopped.
Jan 29 07:46:27 ryzen systemd[1]: libvirtd.service: Unit process 1260 (dnsmasq) remains running after unit stopped.
Jan 29 07:46:27 ryzen systemd[1]: libvirtd.service: Consumed 1.157s CPU time.

Here's "journalctl -b -g virbr0"

Jan 29 06:55:04 ryzen systemd-networkd[835]: virbr0: Link UP
Jan 29 06:55:04 ryzen dnsmasq-dhcp[1259]: DHCP, sockets bound exclusively to interface virbr0
Jan 29 07:42:23 ryzen kernel: virbr0: port 1(vnet0) entered blocking state
Jan 29 07:42:23 ryzen kernel: virbr0: port 1(vnet0) entered disabled state
Jan 29 07:42:23 ryzen kernel: virbr0: port 1(vnet0) entered blocking state
Jan 29 07:42:23 ryzen kernel: virbr0: port 1(vnet0) entered listening state
Jan 29 07:42:25 ryzen kernel: virbr0: port 1(vnet0) entered learning state
Jan 29 07:42:27 ryzen kernel: virbr0: port 1(vnet0) entered forwarding state
Jan 29 07:42:27 ryzen kernel: virbr0: topology change detected, propagating
Jan 29 07:42:27 ryzen systemd-networkd[835]: virbr0: Gained carrier
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPDISCOVER(virbr0) 52:54:00:22:e2:25
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPOFFER(virbr0) 192.168.122.87 52:54:00:22:e2:25
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPREQUEST(virbr0) 192.168.122.87 52:54:00:22:e2:25
Jan 29 07:43:23 ryzen dnsmasq-dhcp[1259]: DHCPACK(virbr0) 192.168.122.87 52:54:00:22:e2:25 archiso
Jan 29 07:44:11 ryzen kernel: virbr0: port 1(vnet0) entered disabled state
Jan 29 07:44:11 ryzen kernel: virbr0: port 1(vnet0) entered disabled state
Jan 29 07:44:11 ryzen systemd-networkd[835]: virbr0: Lost carrier

Here's "iptables -nL"

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_FWX  all  --  0.0.0.0/0            0.0.0.0/0           
LIBVIRT_FWI  all  --  0.0.0.0/0            0.0.0.0/0           
LIBVIRT_FWO  all  --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_OUT  all  --  0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:68

Toolybird · 2022-01-29 01:32:09

Just found another report on reddit:

https://www.reddit.com/r/archlinux/comm … t_problem/

It seems there is a definite problem somewhere...

I can only assume some recent pkg updates in combination with certain conditions (new setups?) have introduced the problem.

Total speculation, but my top suspects would be systemd and/or networkmanager.

Arrowhead · 2022-01-29 04:15:17

That's interesting!

I have similar logs, but the DHCP part is completely missing:

Jan 29 06:56:04 astoria systemd[1]: Started Virtualization daemon.
Jan 29 06:56:04 astoria libvirtd[5193]: 2022-01-29 03:56:04.812+0000: 5193: info : libvirt version: 8.0.0
Jan 29 06:56:04 astoria libvirtd[5193]: 2022-01-29 03:56:04.812+0000: 5193: info : hostname: astoria
Jan 29 06:56:04 astoria libvirtd[5193]: 2022-01-29 03:56:04.812+0000: 5193: debug : virLogParseOutputs:1635 : outputs=3:file:/var/log/libvirt/libvirtd.log
Jan 29 06:56:04 astoria libvirtd[5193]: 2022-01-29 03:56:04.812+0000: 5193: debug : virLogParseOutput:1482 : output=3:file:/var/log/libvirt/libvirtd.log
Jan 29 06:56:04 astoria dnsmasq[1344]: read /etc/hosts - 9 addresses
Jan 29 06:56:04 astoria dnsmasq[1344]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 29 06:56:04 astoria dnsmasq-dhcp[1344]: read /var/lib/libvirt/dnsmasq/default.hostsfile

It never gets to `DHCPDISCOVER`, `DHCPOFFER` etc, so I don't see these records. The output from `iptables -nL` is identical.

I'll try replacing my version of NetworkManager with something else.

Arrowhead · 2022-01-29 06:06:45

OK, it's working now!

Toolybird wrote:

Total speculation, but my top suspects would be systemd and/or networkmanager.

You were exactly right.

My problem was that I had both `NetworkManager` and `systemd-networkd` enabled and running at the same time. How this happened I have no idea, but I blame the GUI installer. Once I disabled `systemd-networkd`, the problem disappeared.

Thanks!

wildbb · 2022-02-09 14:23:08

For those who choose systemd-networkd (I have several setup files in /etc/systemd/network/ to rename interfaces and all so I had a kind of incentive on my side).

First thanks @Lone_Wolf, you lead me to the solution on my side (about primarily looking at the bridge setup). I also found this link which helped me understand some other points: Custom routed network

I didn't strictly applied the setup given in the link but from it I understand that starting the virtual network with virsh trigger systemd to start another instance of dnsmasq which is incompatible by default to the dnsmasq setup for my host as a router. So the solution on my side is:

manually setup a bridge
abandoned the idea of starting a virtual network with virsh
adapt the iptable/nftable to add bridge to interfaces on which you forward to internet
make bridge interface another interface listen by dnsmasq for dhcp and dns services
choose the newly created bridge with network bridge option in virt-manager for each image instead of virtual network 'default' option

From this moment everything worked fine be it a Windows or Linux image.

Hope it can help, if not sorry for the spam.

PS: part of the solution I found is based on my lack of understanding on how to setup correctly two dnsmasq instances (among other things I must admit). There might certainly be a simpler solution with systemd-networkd.

sdamaye · 2024-07-21 07:18:16

I was struggling with the same issue as you guys (DHCP not assigning IP addresses to my VM while virbr0 interface was up) and eventually fixed my issue by appending

firewall_backend = "iptables"

to

/etc/libvirt/network.conf

, as recommended in the doc (https://wiki.archlinux.org/title/Libvirt#Server). Hopefully this will be helpful.

Arch Linux

#26 2022-01-28 21:50:13

Re: Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

#27 2022-01-29 01:32:09

Re: Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

#28 2022-01-29 04:15:17

Re: Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

#29 2022-01-29 06:06:45

Re: Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

#30 2022-02-09 14:23:08

Re: Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

#31 2024-07-21 07:18:16

Re: Libvirt, dnsmasq, QUEMU/KVM : default network dhcp not working

Board footer