You are not logged in.

#1 2016-09-10 01:12:10

foustone
Member
Registered: 2013-02-19
Posts: 6

libvirt - after update to 2.2.0, it does not start up

Hi,

I am posting this to the boards because I have no clue where to look after a possible configuration error from my side. I think it is a configuration error from my side, so maybe you can help me to find it.

To my setup:

I have libvirt installed with only one single domain called "mail", attached to a single network called "default6" (hint: This is not the default network which comes installed with libvirt).

I have this in the default6 network configuration:

<network ipv6='yes'>
  <name>default6</name>
  <uuid>40ac8ed3-XXXX-XXXX-XXXX-5b4481357e8e</uuid>
  <forward mode='nat'/>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:XX:XX:5f:b4:dd'/>
  <ip address='10.0.2.1' netmask='255.255.255.0'>
  </ip>
  <ip family='ipv6' address='XXXX:XXXX:daaa:XXXX::1' prefix='64'>
  </ip>
</network>

(XXXX replaced due to privacy)

So, to my problem:

After updating to libvirt 2.2.0, my virtual machine just does not start up. It seems that libvirt is hanging somewhere during its startup-process.
The problem seems to be completely unrelated to my domain, it seems to be only related to my network "default6", because even if I mark my domain "mail" as not autostarted (only the network "default6" autostarted), libvirt does not start up.

"default6" is the only network from libvirt that is autostarted. I have no other networks configured to autostart.

This is the log output from systemd unit libvirtd when the error happens:

Sep 10 02:29:37 XXX systemd[1]: Started Virtualization daemon.

But it should be something like this, as I see after downgrading to 2.1.0:

Sep 10 02:37:43 XXX systemd[1]: Started Virtualization daemon.
Sep 10 02:37:48 XXX dnsmasq[716]: started, version 2.76 cachesize 150
Sep 10 02:37:48 XXX dnsmasq[716]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
... (many other lines)
Sep 10 02:37:54 XXX libvirtd[441]: libvirt version: 2.1.0
... (and even more lines)

As you see, after 11 seconds, libvirt got started, all is fine. This is not the case when I have 2.2.0 installed. It just hangs.

I looked into the changelog for libvirt 2.2.0 but did not find any backwards-incompatible stuff related to networks.

When I manually try to restart libvirt after my system booted, I found the following in the log (note that here "mail" was marked as autostarted):

Sep 10 02:17:25 XXX systemd[1]: Stopping Virtualization daemon...
Sep 10 02:17:25 XXX systemd[1]: Stopped Virtualization daemon.
Sep 10 02:17:25 XXX systemd[1]: Started Virtualization daemon.
Sep 10 02:17:25 XXX libvirtd[880]: libvirt version: 2.2.0
Sep 10 02:17:25 XXX libvirtd[880]: hostname: XXX
Sep 10 02:17:25 XXX libvirtd[880]: internal error: Network is already in use by interface virbr1
Sep 10 02:17:25 XXX libvirtd[880]: internal error: Failed to autostart VM 'mail': Requested operation is not valid: network 'default6' is not active
Sep 10 02:18:00 XXX libvirtd[880]: internal error: Network is already in use by interface virbr1

Which also doesnt give me any hints where my problem is.

Can you maybe help me to solve this? Which additional info do you need to do this?

Thanks in advance for your help
Regards
foustone

Offline

#2 2016-09-25 09:19:05

nesk
Member
Registered: 2011-03-31
Posts: 181

Re: libvirt - after update to 2.2.0, it does not start up

It seems you have a conflict with another bridge. Check the network definitions (.xml files) that you have in /etc/libvirt/qemu/networks/

Sep 10 02:17:25 XXX libvirtd[880]: internal error: Network is already in use by interface virbr1

Offline

#3 2016-09-25 14:33:50

foustone
Member
Registered: 2013-02-19
Posts: 6

Re: libvirt - after update to 2.2.0, it does not start up

First of all: Thanks for your reply! smile

But no, I have no conflict with another network. I only have the network "default6" configured, the other network which exists is the "default"-network, which uses virbr0. Additionally, the error message also only appears after manually restarting libvirtd - I think it is caused by network default6 not successfully shutted down on restart.

By the way, just to make it clear: The exact same configuration is working without problems with libvirt 2.1.0-1.

Any other ideas? smile

Offline

#4 2016-09-25 14:49:41

nesk
Member
Registered: 2011-03-31
Posts: 181

Re: libvirt - after update to 2.2.0, it does not start up

Did you edit .xml config file by hand or with virsh? Try letting libvirt 2.2.0 create its default network, then edit it with virsh (add ipv6 and so on) and see what happens. Virsh did help me to catch some syntax gotchas in the past.
Worth a try before getting into deeper stuff.

Offline

#5 2016-09-25 18:54:04

foustone
Member
Registered: 2013-02-19
Posts: 6

Re: libvirt - after update to 2.2.0, it does not start up

I edited the XML configuration with virsh, yes.

I already completely removed the default6 network (prior updating to 2.2.0), stopped libvirtd, updated to 2.2.0, copied the default.xml file to default6.xml and changed some basic stuff (name, uuid), started libvirtd and changed the rest of the settings via virsh. Virsh is not complaining about the syntax of the XML file.

Maybe of interest, dont know: I have a qemu-hook /etc/libvirt/hooks/qemu configured, but as I said, the error also happens when only starting the network, not a domain, so this should have no impact. But just to say it: The hook is working in 2.1.0 and has no syntax errors or something (it is a bash script), and it only does something when that one specific domain is being started/stopped.

Offline

#6 2016-10-10 13:17:33

foustone
Member
Registered: 2013-02-19
Posts: 6

Re: libvirt - after update to 2.2.0, it does not start up

Problem still exists with version 2.3.0
Any ideas? sad

Offline

#7 2016-10-10 14:48:26

nesk
Member
Registered: 2011-03-31
Posts: 181

Re: libvirt - after update to 2.2.0, it does not start up

I'm planning to do some IPv6 networking myself in the near future, so I'll try your config on my Arch in the coming days (libvirt 2.3.0).
Meanwhile you might want to ask libvirt-users mailing list as well.

Offline

#8 2016-10-14 17:27:40

nesk
Member
Registered: 2011-03-31
Posts: 181

Re: libvirt - after update to 2.2.0, it does not start up

Ok, so I took example network configuration from here https://libvirt.org/formatnetwork.html#examples and put it in a file default6.xml.
virsh just hangs indefinitely on creating it (net-create default6.xml), and this is what I got in journal:

Oct 14 20:16:22 arch NetworkManager[342]: <error> [1476465382.4349] device (virbr1): Generated connection does not verify: bridge.forward-delay: value '1' is out of range <2-30>

... even if I add delay="5" to xml. So my guess is that NM interferes with bridge setup because it thinks it's invalid. Take it from here smile

Offline

#9 2016-10-15 01:24:39

foustone
Member
Registered: 2013-02-19
Posts: 6

Re: libvirt - after update to 2.2.0, it does not start up

Thanks for your reply. After I read this, I thought "what the fuck".
So I tried again several things.

But, just to keep this clear from the beginning: I do not use NetworkManager nor do I have it installed!

What I did (short version):
1) I stopped all services, including libvirtd, virtlogd.service, virtlogd.socket
2) I did a system update (again) to update to libvirt 2.3.0
3) Deleted all autostart symlinks (the default6 network, my "mail" domain, etc.). libvirt wasn't starting anything except itself after this.
4) Reboot
5) Manually "net-start default6" --> same "behaviour" as before
6) Took a look at the journal after that net-start command. Some error messages about ntpd and about br_netfilter not being loaded.
7) Deinstalled ntpd and replaced it with systemd-timesyncd and loaded br_netfilter at boot
8) Reboot
9) Manually "net-start default6" --> same "behaviour" as before
10) Took a look at the journal after that net-start command. All clear. No error message. Everything seems straight.

This is the short version. I additionally tried different things, among them opening my iptables/ip6tables firewall to everyone, restricting dnsmasq (/etc/dnsmasq.conf) to the lo interface (and other things I probably already have forgotten), but none of those things helped. The "behaviour" is still the same. "net-start default6" just keeps hanging indefinitely.

I have the output of `journalctl -f` after point 9:

Oct 15 02:42:32 XXX systemd-timesyncd[371]: Network configuration changed, trying to establish connection.
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Synchronized to time server [2a01:XXX:XXX:XXX::2]:123 (2.arch.pool.ntp.org).
Oct 15 02:42:32 XXX kernel: tun: Universal TUN/TAP device driver, 1.6
Oct 15 02:42:32 XXX kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Network configuration changed, trying to establish connection.
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Synchronized to time server [2a01:XXX:XXX:XXX::2]:123 (2.arch.pool.ntp.org).
Oct 15 02:42:32 XXX kernel: virbr1: port 1(virbr1-nic) entered blocking state
Oct 15 02:42:32 XXX kernel: virbr1: port 1(virbr1-nic) entered disabled state
Oct 15 02:42:32 XXX kernel: device virbr1-nic entered promiscuous mode
Oct 15 02:42:32 XXX systemd-networkd[430]: virbr1-nic: Gained carrier
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Network configuration changed, trying to establish connection.
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Synchronized to time server [2a01:XXX:XXX:XXXX::2]:123 (2.arch.pool.ntp.org).
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Network configuration changed, trying to establish connection.
Oct 15 02:42:32 XXX systemd-timesyncd[371]: Synchronized to time server [2a01:XXX:XXX:XXXX::2]:123 (2.arch.pool.ntp.org).
Oct 15 02:42:32 XXX kernel: virbr1: port 1(virbr1-nic) entered blocking state
Oct 15 02:42:32 XXX kernel: virbr1: port 1(virbr1-nic) entered listening state
Oct 15 02:42:32 XXX kernel: IPv6: ADDRCONF(NETDEV_UP): virbr1: link is not ready
Oct 15 02:42:34 XXX systemd-networkd[430]: virbr1-nic: Gained IPv6LL
Oct 15 02:42:34 XXX systemd-timesyncd[371]: Network configuration changed, trying to establish connection.
Oct 15 02:42:34 XXX systemd-timesyncd[371]: Synchronized to time server [2a01:XXX:XXX:XXXX::2]:123 (2.arch.pool.ntp.org).
Oct 15 02:42:34 XXX kernel: virbr1: port 1(virbr1-nic) entered learning state
Oct 15 02:42:36 XXX systemd-networkd[430]: virbr1: Gained carrier
Oct 15 02:42:36 XXX systemd-timesyncd[371]: Network configuration changed, trying to establish connection.
Oct 15 02:42:36 XXX systemd-timesyncd[371]: Synchronized to time server [2a01:XXX:XXX:XXXX::2]:123 (2.arch.pool.ntp.org).
Oct 15 02:42:36 XXX kernel: virbr1: port 1(virbr1-nic) entered forwarding state
Oct 15 02:42:36 XXX kernel: virbr1: topology change detected, propagating
Oct 15 02:42:36 XXX kernel: IPv6: ADDRCONF(NETDEV_CHANGE): virbr1: link becomes ready
Oct 15 02:42:38 XXX systemd-networkd[430]: virbr1: Gained IPv6LL

As I said. No error message. Everything looks good (or am I wrong?). It just keeps hanging. Nothing happens afterwards this (ok, I "only" waited 5 minutes).

Any ideas? sad

Offline

#10 2016-10-15 06:49:57

nesk
Member
Registered: 2011-03-31
Posts: 181

Re: libvirt - after update to 2.2.0, it does not start up

Wrong assumption from my side about NetworkManager.
I don't have any more ideas.
I'm pretty sure you will have more luck on libvirt-users mailing list.

Offline

#11 2016-11-04 00:50:27

foustone
Member
Registered: 2013-02-19
Posts: 6

Re: libvirt - after update to 2.2.0, it does not start up

Short update to this thread:
I got the problem fixed! It was a bug at upstream libvirt.

If anyone wants further information, in those thread it can be found:
My bug report https://www.redhat.com/archives/libvirt … 00032.html
The upstream bugfix: https://www.redhat.com/archives/libvir- … 01281.html

The bugfix ended up in release 2.4.0, so the problem is fixed there.

Thanks anyway for anyone who helped, especially to you, nesk smile

Offline

Board footer

Powered by FluxBB