libvirt: Best way to allocate cores to host and guest at runtime?

woah · 2017-03-20 17:43:41

Hello, I use arch as my main and then run various OSs as guests with libvirt. I only have a 4 core CPU (i5 6600K) so when I'm running CPU intensive tasks on the guest systems I like to exclusively assign 3 of the 4 cores to the guest OS. When I'm not running a VM, I like to assign all of them to the host.

I will describe the way I do this below but I would also like to know if there is a better way to go about this.

I'm currently doing this by:
(1) appending isolcpus=1,2,3 to the kernel command line. Thus by default the host system only uses core 0 (core 0 in particular because the kernel always uses it).
(2) configuring the libvirt XML for my guest OSs to use these cores (with the <vcpupin> tags)
(3) running a couple of simple scripts on the host system to either let host processes use cores 1,2,3 or limit it to core 0. The scripts are listed below. I loop over all processes and threads because the real-time attributes of a pre-existing child process are not changed by changing the parents's. Only new processes/threads inherit the changes (AFAIK)

cpuaffinityallhost.sh

#!/bin/bash 

sudo ps -e h| awk '{print $1}' | while read pid
do
	sudo chrt -r -a -p 50 "${pid}"
	sudo taskset -p -a -c 1,2,3 "${pid}"
done

cpuaffinity0host.sh

#!/bin/bash 

sudo ps -e h| awk '{print $1}' | while read pid
do
	sudo chrt -o -a -p 0 "${pid}"
	sudo taskset -p -a -c 0 "${pid}"
done

Few notes:
(1) Although the cpuaffinityallhost.sh script is named "all", the command only assigns cores 1,2,3 to the host system processes. This is because I'm unable to group the isolated cores (1-3) with core 0. Attempting to run the command with "-c 0,1,2,3" results in only core 0 being utilized. Not sure if this is the intended behavior of isolcpus but from the name that would make sense. Note that the host kernel still utilizes core 0 a little bit anyway (it always will from what I've read).

(2) When assigning all cores to the host, the chrt command is used to set the scheduling policy to the round-robin real-time scheduler policy (SCHED_RR). This is because the "other" scheduling policy (SCHED_OTHER/CFQ) doesn't function with tasket as one might expect--it only assigns the first of the cores listed to the provided PID. From a quick google search, this is either the expected behavior of SCHED_OTHER or a bug with taskset/the kernel

So again, I therefore first set scheduling policy to SCHED_RR with chrt's "-r" parameter. It would be nice if I could use SCHED_OTHER/CFQ since it makes the system way more responsive (try running cpuburn with SCHED_RR--your system will be unusable, but with SCHED_OTHER it's completly fine) but I haven't found a solution for that yet.

(3) The above commands spit out various error messages on certain PIDs (e.g. "taskset: failed to set pid 31461's affinity: Invalid argument" or "chrt: cannot obtain the list of tasks: No such file or directory"). I'm guessing these are just limitations enforced by the kernel but overall cpu intensive processes (e.g. firefox) correctly use cores 1-3 with the "allhost" script and cores 1-3 are completely silent with the "0host" script.

So this is what I'm currently doing but I'm just wondering if there's a better way to go about it.

Last edited by woah (2017-03-20 17:45:21)

Lone_Wolf · 2017-03-21 14:30:53

Your description sounds like a good match for using Cgroups.

Disclaimer : i have never used Cgroups myself

woah · 2017-03-22 15:45:13

Lone_Wolf wrote:

Your description sounds like a good match for using Cgroups.
Disclaimer : i have never used Cgroups myself

Thanks, I actually looked into Cgroups at one point myself but after going through many guides I could never actually get it to work. If I remember correctly, the cpuset functionality still isn't working very well with systemd.

I know you personally don't have any experience with cgroups, but if anyone has found a tutorial/guide that has worked for them I'd be happy to hear about it.

Arch Linux

#1 2017-03-20 17:43:41

libvirt: Best way to allocate cores to host and guest at runtime?

#2 2017-03-21 14:30:53

Re: libvirt: Best way to allocate cores to host and guest at runtime?

#3 2017-03-22 15:45:13

Re: libvirt: Best way to allocate cores to host and guest at runtime?

Board footer