You are not logged in.

#1 2016-02-29 16:28:59

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

System freezing when turning on virtual machines

I'm using KVM/qemu with virt-manager (everything on top of   4.4.1-2-ARCH #1 x86_64 fully updated except the to packages below ) as easy way to manage the virtual machines, however after the last kernel upgrade the system is getting some serious issues when starting the virtual machines.

It randomly just "freezes" and the only way is to force poweroff and than try another time until I can start the virtual machine. It's true that I kept back :
warning: libvirt: ignoring package upgrade (1.2.21-1 => 1.3.1-2)
warning: qemu: ignoring package upgrade (2.4.1-2 => 2.5.0-1)
Due to https://bbs.archlinux.org/viewtopic.php?id=206206 , but even with the latest the same happens.

When this happens:
1) I can't get shell anymore
2) nothing works
3) even switching to f2, f3 etc doesn't work
4) There is nothing on the log, no kernel panic trace no nothing.

The only serious messages I found after I powerup the system is:

Feb 29 17:07:16 ig kernel: kvm [1627]: vcpu0 unhandled rdmsr: 0x611
Feb 29 17:07:16 ig kernel: kvm [1627]: vcpu0 unhandled rdmsr: 0x639
Feb 29 17:07:16 ig kernel: kvm [1627]: vcpu0 unhandled rdmsr: 0x641
Feb 29 17:07:16 ig kernel: kvm [1627]: vcpu0 unhandled rdmsr: 0x619
Feb 29 17:07:35 ig kernel: kvm [1642]: vcpu0 unhandled rdmsr: 0x611
Feb 29 17:07:35 ig kernel: kvm [1642]: vcpu0 unhandled rdmsr: 0x639
Feb 29 17:07:35 ig kernel: kvm [1642]: vcpu0 unhandled rdmsr: 0x641
Feb 29 17:07:35 ig kernel: kvm [1642]: vcpu0 unhandled rdmsr: 0x619
Feb 29 17:07:43 ig kernel: kvm [1665]: vcpu0 unhandled rdmsr: 0x611
Feb 29 17:07:43 ig kernel: kvm [1665]: vcpu0 unhandled rdmsr: 0x639
Feb 29 17:07:43 ig kernel: kvm [1665]: vcpu0 unhandled rdmsr: 0x641
Feb 29 17:07:43 ig kernel: kvm [1665]: vcpu0 unhandled rdmsr: 0x619

However when researching the error it looks "harmless" and I didn't find anything to relate it with the issue except remotely https://forum.proxmox.com/threads/error … dmsr.5046/ but there the main system is still usable although the virtual isn't anymore reachable for a period of time.

This is becoming a pane in the buttocks!

Any help is appreciated.

Last edited by r0b0t (2016-02-29 16:29:36)

Offline

#2 2016-03-01 07:48:36

aliena
Member
Registered: 2015-10-09
Posts: 29

Re: System freezing when turning on virtual machines

Similar problem here, except for the "unhandled rdmsr" lines in the log. When opening a shell with

dmesg -w

before starting VM, then start the VM (getting system to freeze) and wait 120s the kernel log will tell that 8 processes hang, each of them providing a callstack in the log. Unfortunately I cannot post it, because after reboot it's not in the journal.. Maybe this is helpful

Offline

#3 2016-03-01 12:35:51

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: System freezing when turning on virtual machines

Yes it is actually I saw the trace to on dmesg, however didn't prepare myself to "save" it.
I set echo "1">/proc/sys/kernel/hung_task_timeout_secs
So the waiting wouldn't be that long, and I'v logged in via SSH remotely to save the log.
It looks pretty bad neutral

Did you try any kernel downgrades if that fixes it?

Offline

#4 2016-03-01 14:02:58

aliena
Member
Registered: 2015-10-09
Posts: 29

Re: System freezing when turning on virtual machines

All I can tell that downgrading libvirt to 1.3.1-1 did not solve the problem, I haven't done further research yet.

Offline

#5 2016-03-01 15:23:02

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: System freezing when turning on virtual machines

If that's the case I'm using 1.2.21-1 and it still happens, I will probably try downgrading the kernel

Offline

#6 2016-03-01 19:55:23

Observer
Member
Registered: 2016-03-01
Posts: 16

Re: System freezing when turning on virtual machines

Same problem here, system up to date.

If I want to start a VM the machine freezes instantly. But "freezes" is not correct. I can move the mouse, nothing else. Problem exists with every attempt.

Offline

#7 2016-03-01 21:11:23

kdall99
Member
Registered: 2015-06-12
Posts: 17

Re: System freezing when turning on virtual machines

Me too™

But with the update to libvirt-1.3.2-1-x86_64 libvirtd itself doesn't seem to really start anymore. Every 90 seconds I see this in the journal:

Mar 01 22:06:59 legolas systemd[1]: Starting Virtualization daemon...
-- Subject: Unit libvirtd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit libvirtd.service has begun starting up.
Mar 01 22:07:00 legolas dnsmasq[1147]: read /etc/hosts - 2 addresses
Mar 01 22:07:00 legolas dnsmasq[1147]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 01 22:07:00 legolas dnsmasq-dhcp[1147]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Mar 01 22:07:00 legolas libvirtd[4390]: libvirt version: 1.3.2
Mar 01 22:07:00 legolas libvirtd[4390]: hostname: legolas.home.ddt.intern
Mar 01 22:07:00 legolas libvirtd[4390]: Interner Fehler: missing machine type
Mar 01 22:07:00 legolas libvirtd[4390]: Interner Fehler: missing machine type
Mar 01 22:07:00 legolas libvirtd[4390]: Interner Fehler: missing machine type
Mar 01 22:07:00 legolas libvirtd[4390]: Interner Fehler: missing machine type
Mar 01 22:08:30 legolas systemd[1]: libvirtd.service: Start operation timed out. Terminating.
Mar 01 22:08:30 legolas systemd[1]: Failed to start Virtualization daemon.
-- Subject: Unit libvirtd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit libvirtd.service has failed.
-- 
-- The result is failed.
Mar 01 22:08:30 legolas systemd[1]: libvirtd.service: Unit entered failed state.
Mar 01 22:08:30 legolas systemd[1]: libvirtd.service: Failed with result 'timeout'.
Mar 01 22:08:30 legolas systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart.
Mar 01 22:08:30 legolas systemd[1]: Stopped Virtualization daemon.
-- Subject: Unit libvirtd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit libvirtd.service has finished shutting down.
Mar 01 22:08:30 legolas systemd[1]: Starting Virtualization daemon...

Technically that fixes the problem with the system hanging, since I can't start a VM anymore. Nevertheless, that's nothing I'd call a real improvement.

Last edited by kdall99 (2016-03-01 21:13:48)

Offline

#8 2016-03-02 01:35:57

gg
Member
Registered: 2015-03-06
Posts: 8

Re: System freezing when turning on virtual machines

Offline

#9 2016-03-03 12:33:57

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: System freezing when turning on virtual machines

Observer wrote:

Same problem here, system up to date.

If I want to start a VM the machine freezes instantly. But "freezes" is not correct. I can move the mouse, nothing else. Problem exists with every attempt.


Yes, thanks for the correction, I'm also able to move the mouse, and strangely enough the script which runs the cpu monitoring via Conky works, but not the RAM or the network widget.

Offline

#10 2016-03-03 12:37:41

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: System freezing when turning on virtual machines

@kdall99
are you sure we are talking about the same issue?

● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor pre
   Active: active (running) since Thu 2016-03-03 10:08:03 CET; 3h 26min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 675 (libvirtd)
    Tasks: 30 (limit: 512)
   CGroup: /system.slice/libvirtd.service
           ├─ 675 /usr/bin/libvirtd
           ├─ 885 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/......
           ├─ 904 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/.....
           ├─ 952 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/...
           ├─ 953 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/.....
           ├─ 972 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/....
           ├─ 991 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/....
           ├─1019 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/....
           ├─1020 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/....
           ├─1041 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/......
           ├─1061 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/,,,,
           ├─1080 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/.....
           ├─1099 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/....
           ├─1118 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/,,,,
           └─1138 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/...

In our case there is a major kernel panic which I'v seen but haven't had the chance to grab the traceback and paste it here yet, as for libvirtd it runs fine.
At least with my outdated version.

Offline

#11 2016-03-03 14:11:16

schlehmil
Member
Registered: 2010-10-01
Posts: 19

Re: System freezing when turning on virtual machines

Similar or same issue already described on KVM mailing list:
http://marc.info/?l=kvm&m=145279173925135&w=4

Offline

#12 2016-03-03 19:56:36

kdall99
Member
Registered: 2015-06-12
Posts: 17

Re: System freezing when turning on virtual machines

r0b0t wrote:

@kdall99
are you sure we are talking about the same issue?

Maybe I haven't made myself clear, sorry. There are actually 2 seperate issues I've been experiencing.

First, I had the same problem you describe: when trying to start a VM the system hangs ("hangs" meaning that while it's still possible to do things like moving the mouse, no new process can be started or stopped, rendering the machine pretty much useless).

Second, after installing an libvirt update, I couldn't reproduce the issue any more. But not because the problem was solved but because libvirtd wouldn't really start and I couldn't spawn new VMs - which prevented me from triggering the first bug.

BTW: I have downgraded to libvirt 1.2.21-1 and the LTS kernel and haven't seen either problem since then.

Offline

#13 2016-03-04 04:23:10

keegano
Member
Registered: 2016-03-04
Posts: 4

Re: System freezing when turning on virtual machines

I am having the same issue, or at least I believe so. I have a Windows guest, and when I attempt to start a process within the guest (such as Chrome) the VM "freezes" - I can move the mouse but do nothing else.

Interestingly, I can get the guest to run if I only give it one CPU core.

I'm running the guest using a script that starts qemu directly (no libvirt) and I've tried using both the standard and LTS kernel, to no avail.

Any other suggestions?

Offline

#14 2016-03-04 07:44:57

Ropid
Member
Registered: 2015-03-09
Posts: 1,069

Re: System freezing when turning on virtual machines

keegano wrote:

[...]

Any other suggestions?

Perhaps try something with the "-cpu" flag? See output of "qemu-system-x86_64 -cpu help".

Last edited by Ropid (2016-03-04 07:45:19)

Offline

#15 2016-03-04 17:52:24

rtollert
Member
Registered: 2013-05-22
Posts: 1

Re: System freezing when turning on virtual machines

Note that after fixing the hang issue (which I wound up doing by downgrading to linux-lts for the time being), those of us using libvirt may also bump into https://bugs.archlinux.org/task/48440. The symptom will be that libvirtd will be in a kill+restart loop.

Offline

#16 2016-03-06 19:41:41

deusstultus
Member
Registered: 2016-03-04
Posts: 4

Re: System freezing when turning on virtual machines

I had observed this fault as well a couple times, but haven't gone troubleshooting yet.  Curiously, haven't seen again since dealing with #48440.

I observed during the hang that processing load went through the roof preventing new process execution without affecting running processes, e.g. followed log files still displaying and running videos continuing.

Offline

#17 2016-03-10 13:42:27

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: System freezing when turning on virtual machines

I'm beginning to hate this issue, it never happens while I'm debugging it sad
I can also confirm however that the old processes keep running but any new process would fail , even just clicking on applications tabs to switch from one application to another.

Offline

#18 2016-03-15 14:09:47

schlehmil
Member
Registered: 2010-10-01
Posts: 19

Re: System freezing when turning on virtual machines

Someone already tried to reproduce with kernel 4.5 from testing? As I stated before some changes to cgroups may solve the issue.

Offline

#19 2016-03-28 00:56:09

r0b0t
Member
From: /tmp
Registered: 2009-05-24
Posts: 505

Re: System freezing when turning on virtual machines

After 4.4.5-1-ARCH update I'm not being able to reproduce this anymore.
Anyone else share the same experience?

Offline

Board footer

Powered by FluxBB