Abysmal I/O with KVM guest

Pres · 2012-07-13 03:00:21

Setting up my home server with KVM so I can replace several machines with it. All has been going well. I have three guests with root partitions set up on qcow2 image files. One machine I want to run a 500GB samba share on. I would rather the files be stored on a partition directly on the hard drive rather than in a qcow2 image. I've set up a partition as a linear raid array as described here.

This works well and I'm able to mount it correctly. The problem comes when I try to transfer files on the guest. The transfer rate is terrible. I have this set up on an encrypted disk with RAID 1 (hardware), so I'm not expecting it to be blazingly fast, but it's still terrible even by those standards.

Here's rsyncing a 1GB file inside the guest on the 500GB partition:

sending incremental file list                                                                                                           
test.img                                                                                                                                
  1048576000 100%    3.33MB/s    0:05:00 (xfer#1, to-check=0/1)                                                                         
                                                                                                                                        
sent 1048704084 bytes  received 31 bytes  3478288.94 bytes/sec                                                                          
total size is 1048576000  speedup is 1.00

This is transferring a 1GB file on the qcow2 image (root) of the guest:

qcow2
sending incremental file list                                                                                                           
test.img                                                                                                                                
  1048576000 100%   30.50MB/s    0:00:32 (xfer#1, to-check=0/1)
sent 1048704084 bytes  received 31 bytes  32267818.92 bytes/sec
total size is 1048576000  speedup is 1.00

5GB file on the host (unencrypted):

Host on root partition
sending incremental file list
test.img
  5242880000 100%   29.62MB/s    0:02:48 (xfer#1, to-check=0/1)

sent 5243520085 bytes  received 31 bytes  30935221.92 bytes/sec
total size is 5242880000  speedup is 1.00

Mounted the same 500GB partition on the host:

sending incremental file list
test.img
  1048576000 100%   32.87MB/s    0:00:30 (xfer#1, to-check=0/1)

sent 1048704084 bytes  received 31 bytes  33292194.13 bytes/sec
total size is 1048576000  speedup is 1.00

Here's how I start the guest:

qemu-kvm -drive file=guest1.img,if=virtio,cache=none -drive file=/dev/md0,if=virtio -drive file=/dev/md1,if=virtio -net nic,model=virtio,macaddr=00:00:00:00:00:00 -net tap,ifname=tap0,script=no,downscript=no -m 2048 -smp 4 -cpu host -vnc :50000 &

How the RAID is created (used the defaults from the guide):

mdadm --build --verbose /dev/md0 --chunk=16 --level=linear --raid-devices=2 /dev/loop0 /dev/mapper/vg-network

ebal · 2012-07-13 11:24:01

I am missing the point of using a linear software raid partition (i didnt say thats its wrong - i dont get it).

I prefer nfs mount from host to guest.

If i want to access filesystem from any guest, i use libguestfs
or kpartx to map the partition and then mount the virtual partition to a mount point.
(second method preferred with offline guest)

Pres · 2012-07-13 12:48:14

The point of this was to keep the partition on the disk intact, while letting the guest access it. I first tried sharing the whole disk with the guest (`-drive file=/dev/sda,if=virtio`) and mounting the partition, but this was less than ideal as the disk is encrypted, so I'd have to type in my passphrase twice. Once on the host and once on the guest. With a software RAID, a small file is created to act as the MBR, then the RAID is aligned so that the partition on the RAID matches with the partition on the disk (so you can deactivated the RAID and still mount the partition).

I haven't used libguestfs and kpartx yet, but from what I've read, those are for accessing disk images, correct? What I'm trying to avoid is using a 500GB .img file.

I guess I could set up an NFS mount on the host, but then I might as well use that for my network storage. The idea here was to strip down the host so it's only doing one thing: KVM. Then the guests would replace the machines I have running now. One of those is running a samba share I use as network storage.

Leonid.I · 2012-07-13 16:04:56

Samba server inside KVM -- that's neat Have you guys tried -virtfs option (http://wiki.qemu.org/Documentation/9psetup)? This uses a plan9 filesystem to share a folder (any size) between guest and host. E.g.

qemu-kvm -hda $1 -net nic,model=virtio \
	-net user,hostfwd=tcp:127.0.0.1:15000-:22 -vga std \
	-usbdevice tablet -device intel-hda -device hda-duplex -m 512 \
	-virtfs local,id=exp1,path=/home/quarantine/shared_dir,security_model=mapped,mount_tag=virtfs

It's fast enough for me even w/o virtio modules inside guest...

Last edited by Leonid.I (2012-07-13 16:09:30)

Pres · 2012-07-13 17:32:11

I played around with virtfs, but wasn't too amazed with it. The guest can't see the size of the shared folder, which was causing problems with samba. I also couldn't get it to mount from /etc/fstab. To get it to mount with the mount command, I needed to issue it twice (the first one would always fail with an error, forget what it was). Rather than play around with it, I decided that probably wasn't the best way to go.

Arch Linux

#1 2012-07-13 03:00:21

Abysmal I/O with KVM guest

#2 2012-07-13 11:24:01

Re: Abysmal I/O with KVM guest

#3 2012-07-13 12:48:14

Re: Abysmal I/O with KVM guest

#4 2012-07-13 16:04:56

Re: Abysmal I/O with KVM guest

#5 2012-07-13 17:32:11

Re: Abysmal I/O with KVM guest

Board footer