Arch Linux

squatched

Member

Registered: 2014-12-08

Posts: 12

[SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

I can't say for certain what precipitated this issue but my searches online have turned up with nothing (or, at least nothing helpful, only thread I actually found to tug on was someone disabled apparmor which I am using, so I stopped it but it had no effect on the behavior).

The issue is that when I start a virtual network or create one, I receive the following error:

# virsh net-info default
Name:           default
UUID:           36fa1556-247c-40f8-867d-fba3374710d9
Active:         no
Persistent:     yes
Autostart:      yes
Bridge:         virbr0

# virsh net-start default
error: Failed to start network default
error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set

I don't know if it means something but I can manually run that failing dnsmasq command without any issue. Unfortunately, it has no effect on things as far as libvirt is concerned though...

# VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

# virsh net-info default
Name:           default
UUID:           36fa1556-247c-40f8-867d-fba3374710d9
Active:         no
Persistent:     yes
Autostart:      yes
Bridge:         virbr0

# virsh start <my-vm>
error: Failed to start domain '<my-vm>'
error: Requested operation is not valid: network 'default' is not active

# virsh net-start default
error: Failed to start network default
error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set

I'm sorry if pertinent info is missing, I'll be happy to provide additional configuration information if requested.

Thanks in advance for any light you can shed on this or any direction you could give for where to look/ask for more info.

Last edited by squatched (2022-03-22 18:28:55)

blufinney

Member

Registered: 2014-08-26

Posts: 59

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

I have the same exact issue (after recent upgrade).  Haven't had time to troubleshoot yet - will post whatever I learn over the next couple days.

blufinney

Member

Registered: 2014-08-26

Posts: 59

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

FWIW I've tried a bunch of things and read through a handful of "KVM network issue" type threads - none of them related to this and a PATH envvar not being found.  Since I do indeed have a PATH envvar on my system - maybe it's something to do with a python jail or something?  I don't know python very well, so not sure.  I'm going to try rolling back to see if that helps temporarily - really depend on vm's.

blufinney

Member

Registered: 2014-08-26

Posts: 59

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

FYI - I downgraded back to 2022-03-01 and VM's (and the network) work again.

Maybe it's a bug, idk.  Searches didn't find anything for me...

squatched

Member

Registered: 2014-12-08

Posts: 12

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

FYI - I downgraded back to 2022-03-01 and VM's (and the network) work again.

Maybe it's a bug, idk.  Searches didn't find anything for me...

Thanks for looking into it, your data point is appreciated. Guess it's time to figure out how to report this to upstream. Maybe I can get around to that this weekend. If I do, I'll respond here with details.

Would you happen to have the package(s) and version(s) you downgraded to? I had libvirt-1:8.1.0-4 installed and I just downgraded libvirt, libvirt-storage-gluster, libvirt-storage-iscsi-direct, and libvirt-storage-rbd all to v1:8.0.0-2, a quick restart of libvirtd and I'm in business. Do you have a newer version than 1:8.0.0-2 you were able to get working? (Just trying to narrow down as much as possible which version has a problem.)

To help anyone else that ran into this issue...

# pacman -U /var/cache/pacman/pkg/libvirt-1:8.0.0-2-x86_64.pkg.tar.zst /var/cache/pacman/pkg/libvirt-storage-gluster-1:8.0.0-2-x86_64.pkg.tar.zst /var/cache/pacman/pkg/libvirt-storage-iscsi-direct-1:8.0.0-2-x86_64.pkg.tar.zst /var/cache/pacman/pkg/libvirt-storage-rbd-1:8.0.0-2-x86_64.pkg.tar.zst
loading packages...
warning: downgrading package libvirt (1:8.1.0-4 => 1:8.0.0-2)
warning: downgrading package libvirt-storage-gluster (1:8.1.0-4 => 1:8.0.0-2)
warning: downgrading package libvirt-storage-iscsi-direct (1:8.1.0-4 => 1:8.0.0-2)
warning: downgrading package libvirt-storage-rbd (1:8.1.0-4 => 1:8.0.0-2)
resolving dependencies...
looking for conflicting packages...

Packages (4) libvirt-1:8.0.0-2  libvirt-storage-gluster-1:8.0.0-2  libvirt-storage-iscsi-direct-1:8.0.0-2  libvirt-storage-rbd-1:8.0.0-2

Total Installed Size:  43.01 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n]
[snip]

# systemctl restart libvirtd.service

Toolybird

Member

Registered: 2017-09-30

Posts: 76

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

> report this to upstream

There is already an upstream bug report [1] but no response from upstream dev's so far. There was definitely a big refactoring of code surrounding dnsmasq in the 8.1.0 release so a latent bug seems possible.

> VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq...

Where is it getting /usr/local from ? Do you have a dnsmasq in /usr/local?

[1] https://gitlab.com/libvirt/libvirt/-/issues/282

squatched

Member

Registered: 2014-12-08

Posts: 12

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

Do you have a dnsmasq in /usr/local?

Yes because firejail.

$ which dnsmasq
/usr/local/bin/dnsmasq
$ readlink /usr/local/bin/dnsmasq
/usr/bin/firejail

Also, and more importantly, thank you for the info about upstream, I'll keep an eye on that.

blufinney

Member

Registered: 2014-08-26

Posts: 59

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

Would you happen to have the package(s) and version(s) you downgraded to? I had libvirt-1:8.1.0-4 installed and I just downgraded libvirt, libvirt-storage-gluster, libvirt-storage-iscsi-direct, and libvirt-storage-rbd all to v1:8.0.0-2, a quick restart of libvirtd and I'm in business. Do you have a newer version than 1:8.0.0-2 you were able to get working? (Just trying to narrow down as much as possible which version has a problem.)

I downgraded my entire system using the Arch Linux Archive by updating "/etc/pacman.d/mirrorlist" with

Server=https://archive.archlinux.org/repos/2022/03/01/$repo/os/$arch

and then running:

# pacman -Syyuu

blufinney

Member

Registered: 2014-08-26

Posts: 59

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

Yes because firejail.

FWIW I also use Firejail.  It's possible Firejail is causing interference with something new in dnsmasq.  Hopefully this weekend I'll have time to re-upgrade and attempt to deactivate Firejail on dnsmasq to see if there is any effect.

squatched

Member

Registered: 2014-12-08

Posts: 12

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

It's possible Firejail is causing interference with something new in dnsmasq.  Hopefully this weekend I'll have time to re-upgrade and attempt to deactivate Firejail on dnsmasq to see if there is any effect.

Maybe, but it would have to be causing a difference with how dnsmasq is used by libvirt considering that my downgrading of JUST libvirt and its dependents got my system working again. I'm going to poke through my firejail dnsmasq profiles and see if anything jumps out at me, regardless...

Edit: Just did a quick 'sudo firejail --list' and dnsmasq is not among the listed sandboxes so, firejail isn't at play here at all.

Last edited by squatched (2022-03-11 16:55:34)

Maniaxx

Member

Registered: 2014-05-14

Posts: 757

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set

The official 'extra/dnsmasq 2.86-1' package doesn't provide a '/usr/local/bin/dnsmasq' file.
Check for pacnew config files. I think there was a major update recently for libvirt (or was it just virt-manager?).
Network commands work properly here.

---

sys2064

Toolybird

Member

Registered: 2017-09-30

Posts: 76

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

So, the difference in the latest libvirt release is that dnsmasq is now searched for in $PATH. Previously, it was found at build time and effectively hardwired to /usr/bin/dnsmasq .

It appears libvirt's use of dnsmasq is incompatible with firejail. I'd guess the quick fix is to simply remove the dnsmasq symlink in /usr/local/bin then restart libvirtd (or reboot).

michelesr

Member

Registered: 2016-02-04

Posts: 67

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

I'd guess the quick fix is to simply remove the dnsmasq symlink in /usr/local/bin then restart libvirtd (or reboot).

Yes, that did the trick for me. Thanks!

squatched

Member

Registered: 2014-12-08

Posts: 12

Re: [SOLVED]Libvirt Virtual Network Start/Create Fails w/ PATH envvar...

It appears libvirt's use of dnsmasq is incompatible with firejail. I'd guess the quick fix is to simply remove the dnsmasq symlink in /usr/local/bin then restart libvirtd (or reboot).

This worked for me as well, thank you. To persist this change though through subsequent runs of firecfg, I also commented out "dnsmasq" in /etc/firejail/firecfg.config with a comment indicating it can be added back in once the upstream issue is resolved.

Thanks for the help on this, I'll mark this thread as resolved.

Also, thanks for not listening to me when I thought that firejail wasn't the issue due to my erroneous logic =P.