Arch Linux

Zemtriz

Member

Registered: 2012-07-25

Posts: 6

libvirt can't use 1G hugepages

Hello everyone. I'm stuck when I want to run a VM using the hugepages.

Here in libvirt xml file is definition to use hugepages:

<memoryBacking>
    <hugepages/>
</memoryBacking>

Here is visible that I have hugepages setup on my system:

$ cat /proc/meminfo | grep Huge
AnonHugePages:  31684608 kB
ShmemHugePages:        0 kB
FileHugePages:      4096 kB
HugePages_Total:      16
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        16777216 kB

I allowed hugepages by passing kernel parameters like this:

... default_hugepagesz=1G hugepagesz=1G hugepages=16 ...

When I try to run the VM, I receive the following error:

error: internal error: hugetlbfs filesystem is not mounted or disabled by administrator config

Hugepages are shown as mounted in /proc/mounts:

...
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,gid=992,mode=1770,pagesize=1024M 0 0
...

In /etc/libvirt/qemu.conf I've tried to uncomment the following line:

...
hugetlbfs_mount = "/dev/hugepages"
...

In fstab I defined to change owner group of /dev/hugepages file to kvm:

...
hugetlbfs               /dev/hugepages          hugetlbfs               mode=01770,gid=kvm      0 0
...

My user groups:

$ groups myusername
wheel input kvm libvirt myusername

Also qemu and libvirt-qemu users belong to kvm group too.
/dev/hugepages ownership:

drwxrwx--T  2 root kvm     0 Jun 27 12:59 .
drwxr-xr-x 18 root root 3.9K Jun 27 13:03 .. 

I'm out of ideas. Hugepages exists, it is mounted, ownership is kvm, it's just there. Why is libvirt complaining that it is disabled or not mounted? What am I missing?

seth

Member

Registered: 2012-09-03

Posts: 56,256

Re: libvirt can't use 1G hugepages

Race condition?
Have yo utried to restart libvirtd?
https://www.mail-archive.com/libvirt-us … 09408.html

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Zemtriz

Member

Registered: 2012-07-25

Posts: 6

Re: libvirt can't use 1G hugepages

Race condition?
Have yo utried to restart libvirtd?
https://www.mail-archive.com/libvirt-us … 09408.html

I've tried to restart libvirtd process just now but it didn't help. I also tried to umount and mount /dev/hugepages and also when I created hugepages as described above, I restarted the whole system just to be sure. But nothing is working
But in my situation the /dev/hugepages should be mounted by default before even libvirtd process starts, right? Because it is passed right as kernel parameter in boot process

Last edited by Zemtriz (2024-06-28 07:27:44)

Zemtriz

Member

Registered: 2012-07-25

Posts: 6

Re: libvirt can't use 1G hugepages

OK, so I can use hugepages, when I don't use virtlib manager.
I was inspired by this gentoo forum: https://forums.gentoo.org/viewtopic-t-1 … art-0.html
So I setup the following command to run my VM for testings. I didn't even setup network interface, only basic things.

/usr/bin/qemu-system-x86_64 \
 -name guest=GamingVM,debug-threads=on \
 -machine type=q35,usb=off,vmport=off,smm=on,hpet=off,acpi=on \
 -accel kvm \
 -cpu host,migratable=off,topoext=on,hypervisor=off,kvmclock=off,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-vendor-id=Origina790ia,hv-frequencies=on,hv-tlbflush=on,hv-ipi=on,hv-avic=on,kvm=off \
 -smp 8,sockets=1,cores=4,threads=2 \
 -m size=12582912k -mem-prealloc -mem-path /dev/hugepages \
 -drive file=/home/myusername/.local/share/libvirt/images/GamingVM.img,format=raw,if=virtio \
 -drive file=/mnt/nvme1n1p1/myusername/qemu/images/Windows.iso,index=1,media=cdrom \
 -vnc 0.0.0.0:3 \
 -monitor stdio \
 -device qemu-xhci \
 -usbdevice tablet \
 -boot menu=on \
 -vga qxl

and now it runs and cat /proc/meminfo | grep Huge looks like this:

AnonHugePages:  25579520 kB
ShmemHugePages:        0 kB
FileHugePages:     14336 kB
HugePages_Total:      16
HugePages_Free:        4
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        16777216 kB

It seems like libvirt is still looking into wrong path and doesn't find hugepages location. But why? It should look into /proc/mounts and there it is if it doesn't find. Can I somehow force it to read /dev/hugepages?

Last edited by Zemtriz (2024-06-28 18:03:42)

seth

Member

Registered: 2012-09-03

Posts: 56,256

Re: libvirt can't use 1G hugepages

The gentoo thread speculates that this is a permission issue - what if you make the entire hugetlbfs ugo+rw ?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Zemtriz

Member

Registered: 2012-07-25

Posts: 6

Re: libvirt can't use 1G hugepages

It didn't help. I updated my /etc/fstab and changed mode to 01777 so everyone has read/write/execute access. Rebooted the server and tried to start the VM using virsh start command. It didn't work. So it seems that libvirt is not looking into correct hugepages folder.