You are not logged in.

#1 2017-11-10 22:24:39

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

[Solved]virt-manager hangs on QEMU/KVM - Connecting...

Suddenly started having an issue with virt-manager not being able to connect to libvirtd.  The connection doesn't fail, it just doesn't complete.  virsh hangs as well.  Any help is appreciated.

Configurations and output:

libvirtd appears to run fine(I'm not using virtualbox currently, so I'm not worried about that warning).

[root@<DEVICE>]# systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-11-10 14:48:18 CST; 13min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 3415 (libvirtd)
    Tasks: 18 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─3415 /usr/bin/libvirtd
           └─3438 /sbin/ebtables --concurrent -L

Nov 10 14:48:18 <USER> systemd[1]: Started Virtualization daemon.
Nov 10 14:48:18 <USER> libvirtd[3415]: 2017-11-10 20:48:18.482+0000: 3415: info : libvirt version: 3.8.0
Nov 10 14:48:18 <USER> libvirtd[3415]: 2017-11-10 20:48:18.482+0000: 3415: info : hostname: <USER>
Nov 10 14:48:18 <USER> libvirtd[3415]: 2017-11-10 20:48:18.482+0000: 3415: warning : vboxGetHypervisorDriver:7756 : Libvirt doesn't support VirtualBox API version 5002000

kvm modules loaded

[root@<DEVICE> ~]# lsmod | grep kvm
kvm_intel             192512  0
kvm                   516096  1 kvm_intel
irqbypass              16384  2 kvm,vfio_pci

USER is in kvm and libvirt groups

[root@<DEVICE>]# cat /etc/group | grep -e libvirt -e kvm
kvm:x:999:<USER>
libvirt:x:991:<USER>

Debug info when launching virt-manager.  I imagine server and user should be null as I'm not connecting to a remote guest?

[<USER>@<DEVICE> ~]$ LIBVIRT_DEBUG=1 LIBVIRT_LOG_OUTPUTS="1:file:client.log" virt-manager --no-fork
2017-11-10 21:16:37.388+0000: 3775: info : libvirt version: 3.8.0
2017-11-10 21:16:37.388+0000: 3775: info : hostname: <USER>
2017-11-10 21:16:37.388+0000: 3775: debug : virLogParseOutputs:1731 : outputs=1:file:client.log
2017-11-10 21:16:37.454+0000: 3775: debug : virLogParseOutput:1559 : output=1:file:client.log
2017-11-10 21:16:37.455+0000: 3775: debug : virGlobalInit:386 : register drivers
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:684 : driver=0x7f90f1f7d500 name=Test
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:695 : registering Test as driver 0
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:684 : driver=0x7f90f1f7fde0 name=OPENVZ
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:695 : registering OPENVZ as driver 1
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:684 : driver=0x7f90f1f805c0 name=VMWARE
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:695 : registering VMWARE as driver 2
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:684 : driver=0x7f90f1f7f400 name=PHYP
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:695 : registering PHYP as driver 3
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:684 : driver=0x7f90f1f80d80 name=ESX
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:695 : registering ESX as driver 4
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:684 : driver=0x7f90f1f7e000 name=remote
2017-11-10 21:16:37.455+0000: 3775: debug : virRegisterConnectDriver:695 : registering remote as driver 5
2017-11-10 21:16:37.562+0000: 3775: debug : virGetVersion:886 : libVir=0x7fff7a4be2e8, type=(null), typeVer=(nil)
2017-11-10 21:16:37.596+0000: 3775: debug : virEventRegisterImpl:241 : addHandle=0x7f90da138d10 updateHandle=0x7f90da138c10 removeHandle=0x7f90da138b00 addTimeout=0x7f90da138a30 updateTimeout=0x7f90da138f80 removeTimeout=0x7f90da138ec0
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenAuth:1245 : name=qemu:///system, auth=0x7f90d889bf30, flags=0x0
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c8001b10 classname=virConnect
2017-11-10 21:16:37.652+0000: 3784: debug : virConfGetValueStringList:986 : Get value string list (nil) 0
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1033 : Split "qemu:///system" to URI components:
  scheme qemu
  server <null>
  user <null>
  port -1
  path /system
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1083 : trying driver 0 (Test) ...
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1098 : driver 0 Test returned DECLINED
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1083 : trying driver 1 (OPENVZ) ...
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1098 : driver 1 OPENVZ returned DECLINED
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1083 : trying driver 2 (VMWARE) ...
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1098 : driver 2 VMWARE returned DECLINED
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1083 : trying driver 3 (PHYP) ...
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1098 : driver 3 PHYP returned DECLINED
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1083 : trying driver 4 (ESX) ...
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1098 : driver 4 ESX returned DECLINED
2017-11-10 21:16:37.652+0000: 3784: debug : virConnectOpenInternal:1083 : trying driver 5 (remote) ...
2017-11-10 21:16:37.652+0000: 3784: debug : doRemoteOpen:916 : proceeding with name = qemu:///system
2017-11-10 21:16:37.652+0000: 3784: debug : doRemoteOpen:925 : Connecting with transport 1
2017-11-10 21:16:37.652+0000: 3784: debug : doRemoteOpen:1060 : Proceeding with sockname /var/run/libvirt/libvirt-sock
2017-11-10 21:16:37.652+0000: 3784: debug : virNetSocketNewConnectUNIX:640 : path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null>
2017-11-10 21:16:37.652+0000: 3784: debug : virNetSocketNewConnectUNIX:704 : connect() succeeded
2017-11-10 21:16:37.652+0000: 3784: debug : virNetSocketNew:236 : localAddr=0x7f90d889ba20 remoteAddr=0x7f90d889bab0 fd=4 errfd=-1 pid=0
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c8004030 classname=virNetSocket
2017-11-10 21:16:37.652+0000: 3784: info : virNetSocketNew:292 : RPC_SOCKET_NEW: sock=0x7f90c8004030 fd=4 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c8004320 classname=virNetClient
2017-11-10 21:16:37.652+0000: 3784: info : virNetClientNew:328 : RPC_CLIENT_NEW: client=0x7f90c8004320 sock=0x7f90c8004030
2017-11-10 21:16:37.652+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c8004320
2017-11-10 21:16:37.652+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c8004030
2017-11-10 21:16:37.652+0000: 3784: debug : virKeepAliveNew:196 : client=0x7f90c8004320, interval=-1, count=0
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c80050a0 classname=virKeepAlive
2017-11-10 21:16:37.652+0000: 3784: info : virKeepAliveNew:215 : RPC_KEEPALIVE_NEW: ka=0x7f90c80050a0 client=0x7f90c8004320
2017-11-10 21:16:37.652+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c8004320
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c8005210 classname=virConnectCloseCallbackData
2017-11-10 21:16:37.652+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c8005210
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c8004280 classname=virNetClientProgram
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c8003f20 classname=virNetClientProgram
2017-11-10 21:16:37.652+0000: 3784: info : virObjectNew:254 : OBJECT_NEW: obj=0x7f90c80054a0 classname=virNetClientProgram
2017-11-10 21:16:37.652+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c8004280
2017-11-10 21:16:37.652+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c8003f20
2017-11-10 21:16:37.653+0000: 3784: info : virObjectRef:388 : OBJECT_REF: obj=0x7f90c80054a0
2017-11-10 21:16:37.653+0000: 3784: debug : doRemoteOpen:1179 : Trying authentication
2017-11-10 21:16:37.653+0000: 3784: debug : virNetMessageNew:46 : msg=0x7f90c80056a0 tracked=0
2017-11-10 21:16:37.653+0000: 3784: debug : virNetMessageEncodePayload:389 : Encode length as 28
2017-11-10 21:16:37.653+0000: 3784: info : virNetClientSendInternal:2119 : RPC_CLIENT_MSG_TX_QUEUE: client=0x7f90c8004320 len=28 prog=536903814 vers=1 proc=66 type=0 status=0 serial=0
2017-11-10 21:16:37.653+0000: 3784: debug : virNetClientCallNew:2072 : New call 0x7f90c8005430: msg=0x7f90c80056a0, expectReply=1, nonBlock=0
2017-11-10 21:16:37.653+0000: 3784: debug : virNetClientIO:1878 : Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 dispatch=(nil)
2017-11-10 21:16:37.653+0000: 3784: debug : virNetClientIO:1937 : We have the buck head=0x7f90c8005430 call=0x7f90c8005430

I have a generated /etc/libvirt/libvirtd.conf, but everything is currently commented out.

Thanks again,

Sir_Kibble

Last edited by Sir_Kibble (2018-03-02 20:21:07)

Offline

#2 2017-11-12 04:17:30

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Does anyone have any idea what's going on?  Is this posted in the right section?

Offline

#3 2017-11-12 13:54:25

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

What is the output of the virsh commands listed in wiki ?
https://wiki.archlinux.org/index.php/Libvirt#Test


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#4 2017-11-13 00:51:52

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

As root, neither commands give output.  Both hang.  As USER, virsh -c qemu:///system hangs, while virsh -c qemu:///session start a virsh CLI.  'list' doesn't show any of my old VMs.  I'll look more over the link you posted and see if there's anything else in there.

Offline

#5 2017-11-14 03:50:40

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

I removed qemu, libvirt, ovmf, and virt-manager, ran pacman -Syu, followed https://wiki.archlinux.org/index.php/PC … h_via_OVMF and reinstalled qemu, libvirt, ovmf, and virt-manager and made it to setting up the guest OS before encountering the original issue.  Is there anything else anyone recommends?

Offline

#6 2017-11-14 19:53:25

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Has libvirtd worked successfully on that system?  As groups contains kvm as assigned by systemd with the first available gid below 1000 rather than 78 as assigned by qemu

kvm:x:999:<USER>

So did you fix the ownership of /var/lib/libvirt/qemu and change group="78" to group="kvm" in etc/libvirt/qemu.conf?
https://bugs.archlinux.org/task/54943

Offline

#7 2017-11-15 01:21:53

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

I'm away from my system, but I'll check that when I'm back home.  I'll post back with results when I can.

Offline

#8 2017-11-15 10:21:14

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

It had been previously been working.  I had not edited the .conf files(reset from the uninstall/reinstall?  Seem to remember setting them before...), so I updated the group and user lines appropriately and changed /var/lib/virtlib/qemu and contents ownership to root:kvm, but with no change.  Tried uid/gid as well in qemu.conf.  Changed the gid of kvm to 78 to see if that would change anything with

groupmod -g 78 kvm

and updating old gids with

find / -gid 999 ! -type l -exec chgrp 78 {} \;

with no change.  I looked at /usr/lib/sysusers.d/basic.conf and saw that kvm is listed under the hardware access group, for what that's worth.

Are there other .conf files I would need to update the gid/group name in?  The bug report appears to still be open, so should I look at applying the latest patch mentioned by loqs?

Last edited by Sir_Kibble (2017-11-15 10:23:19)

Offline

#9 2017-11-19 00:05:25

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Issue still unresolved.  How long does it typically take for patches to be merged into the official package?  I'm looking into how to apply the patches myself, but have been quite busy lately.

Offline

#10 2017-11-19 00:37:35

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

I suspect the issue has another source as you have changed the gid of kvm back to 78 and changed the ownership of all files to match.

Offline

#11 2017-11-20 02:04:54

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

I'm thinking I'll just reinstall Arch and see what happens, but is there any less extreme solution you'd recommend?

Offline

#12 2017-11-20 10:28:57

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Possibly check the logs in /var/log/libvirt/qemu/

Offline

#13 2017-11-22 11:53:15

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Sofar we've focused on libvirt, maybe the problem is in qemu setup.

Could you try creating / running a VM manually ?


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#14 2017-11-24 04:09:56

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

@loqs: The machine logs don't show anything out of the ordinary.  Is there something I should be looking for?  greping for 'error' gave me 2 lines in regards to USB configuration, and 'warning' gives me nothing actionable. 

@Lone_Wolf: I can create VMs connecting to the qemu:///session URI, but I can't connect and create on qemu:///system.  I was thinking it might be the group permissions, but changing those hasn't solved the issue.

Offline

#15 2017-11-30 03:10:30

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Made a backup and have started reinstalling.  Work's picking up, so it'll be a few days, but I'll post back with my reinstall results when I can.

Offline

#16 2017-12-02 23:04:16

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Got done quicker than expected.  Not sure what I did to screw it up before, but the reinstall has fixed it.  Even though it may not be very helpful to future users, I'm marking this as solved.

Offline

#17 2018-01-26 16:30:21

sleeping
Member
Registered: 2016-01-09
Posts: 53

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

What do you mean by 'reinstall'? Did you reinstall virt-manager?

I am having exactly the same issue. It was working yesterday, but all of the sudden it doesn't connect anymore.

Edit: I reinstalled virt-manager and libvirt, this did not solve this issue.

Output virt-manager --debug:

[vr, 26 jan 2018 18:11:46 virt-manager 1562] DEBUG (connection:602) conn=qemu:///system changed to state=Connecting
[vr, 26 jan 2018 18:11:46 virt-manager 1562] DEBUG (connection:1019) Scheduling background open thread for qemu:///system

Running with sudo -E virt-manager does not solve anything.

Edit2: You reinstalled Arch... My God. Please unmark it as solved.

Last edited by sleeping (2018-01-26 17:16:00)

Offline

#18 2018-01-26 23:31:43

sleeping
Member
Registered: 2016-01-09
Posts: 53

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Offline

#19 2018-01-28 21:55:36

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Removed [SOLVED] in title.

Offline

#20 2018-02-16 11:36:35

sleeping
Member
Registered: 2016-01-09
Posts: 53

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

The solution is to remove /var/lib/ebtables/lock if anyone else comes across this!

Offline

#21 2018-03-02 20:17:22

Sir_Kibble
Member
Registered: 2017-11-10
Posts: 13

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Nice!  I'll update the title.  Thanks everyone.

Offline

#22 2018-09-01 09:15:36

arifur
Member
Registered: 2017-04-08
Posts: 3

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

sleeping wrote:

The solution is to remove /var/lib/ebtables/lock if anyone else comes across this!

This did not seem to fix my issue.

Offline

#23 2018-09-01 09:50:23

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,784
Website

Re: [Solved]virt-manager hangs on QEMU/KVM - Connecting...

Then you have a different issue.

https://wiki.archlinux.org/index.php/Co … ow_to_post

Closing.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

Board footer

Powered by FluxBB