qemu-kvm, virtual network stalling under load

cosmonate · 03-08-2010, 01:04 PM

Hi, i have this problem with every network model and also using the newest release 0.12.3, which states that this problem is fixed for virtio-net :-/

host system is slackware64 13.0, guest too! kernel 2.6.29.6 on qemu-kvm-0.12.3
network is bridged using tap and virtio (copy performance around 60MB/s until stall :-(

copying large files via scp from the host to vm's /dev/null works! but copying e.g. the slackware-installiso from a mounted nfs share stalls the network of the virtual machine completely!!
also transfering a few hundret megs via ftp/ftps stalls the network completely!
no network response from the vm (ssh, icmp, etc..)
i can only shutdown and restart the vm using qemu monitor since i am running headless servers!

when using another NIC model the amount of transferable data decreases to only a few megs untill the network stalls :-/

error logs simply say nothing that indcate the problem :-/

anyone some hints or a possible solution for this defect?

regards
bernd

HasC · 03-09-2010, 06:22 PM

How did you configure your VMs? I mean, how's your kvm command/script built?
Something like:

Code:

$ kvm -hda hda.img -net nic -net tap

perhaps?

Chuck56 · 03-09-2010, 10:36 PM

Quote:

Originally Posted by cosmonate

anyone some hints or a possible solution for this defect?

Does it just lock up on high loads? I used to have the same issue but it's been so long that I forget the exact fix. Have you thought about moving the disk drivers to virtio too? I don't run 64 bit guests on 64 bit hosts in production so I can't comment on that combination. Here's how I start Slackware32-13 guests on Slackware64-13 hosts.

Code:

        su - kvm -c "qemu-system-x86_64 \
        -daemonize \
        -localtime \
        -nographic \
        -m 512 \
        -monitor telnet::xxxx,server,nowait \
        -serial telnet::yyyy,server,nowait \
        -net nic,macaddr=52:54:00:xx:yy:zz,model=virtio \
        -net tap,ifname=tap1,script=no \
        -drive file=/home/guest1.raw,if=virtio,boot=on,cache=none \
        -drive file=/home/guest1_home.ext3,if=virtio,cache=none"

xx, yy, and zz uses to hide actual values.

I use raw files for my VMs and separate loopback files for /home partitions. Keeps files sizes more manageable. All the drives are ext3 for stability and recovery.

cosmonate · 03-10-2010, 01:46 AM

qemu command in a script of course:

qemu-system-x86_64 -cpu qemu64,+ssse3,+ht,+monitor,+vmx,+est,+cx16,+xtpr
-smp 2
-boot c
-drive file=$IMAGE,if=virtio,cache=none,boot=on
-m $MEM
-net nic,vlan=$VLAN,model=virtio,macaddr=$MACADDR
-net tap,vlan=$VLAN,ifname=$TAP,script=$NETUP,downscript=$NETDOWN
-pidfile $PIDFILE
-usbdevice tablet
-nographic
-monitor $MONITOR 1>$LOGFILE 2>$ERRORFILE &

MEM is between 1 and 4 GB depending on the server (there are actually 6 of them running, more to come)

it just locks up under heavy load resp. after some datatransfer oder the virtual network via nfs... (this happens regardless if there where 1 or more servers running)

case study 100% reproducable:
- using virtio i can -> update slackware with all current patches, but i cannot dump or copy the slackware iso from the nfs mount. i CAN copy the iso via scp although!

- using virtio and changing the rsize/wsize to 1MB increases virtual network speed from 60MB/s to 124MB/s and lets me dump/copy the iso sometimes...

- using another nic model (e.g. one of the intel or rtl) i cannot even upgrade slackware! network stalls when transferring the gimp package!

- i changed the kernel to 2.6.30.5. now i can dump/copy the iso some more times, but not always! but the network still stalls when i try to dump/copy a urandom created 50gig image! so switching the kernel did not fix the bug...

i'm running out of ideas, and i don't want to switch my hypervisor to virtualbox or vmware, simply because of their limitations and performance, i really like kvm :-( PLZ HELP!! :-)

btw: what exactly is the difference of kvm88 and qemu-kvm-devel-88 and qemu-kvm-0.12.3 (the first two state to be development branch, the last states to be the stable but is the newest!?) i don't understand their release management!?

i am using qemu-kvm-0.12.3 on a tyan server mobo with 2x quadcore xeon w. hyperthreading and 24GB ram... i run 64bit host and guests to get the most perfmormance out of my virtual server environment! all machines are powered by a raid5 + lvm using xfs... (this is just to give you a impression of the environment i deal with)

i hope there is a solution out there somewhere, but the replys on qemu and kvm forums are NULL... :-(

regards
bernd

Chuck56 · 03-10-2010, 05:37 AM

Quote:

Originally Posted by cosmonate

-cpu qemu64,+ssse3,+ht,+monitor,+vmx,+est,+cx16,+xtpr

I recommend you remove the -cpu options and let KVM determine the best config. This may be contributing to your locking issues.

Quote:

Originally Posted by cosmonate

-smp 2

Try running without this option as a test. The cpu functions add emulation load to the guest and host which may impact the locking issues.

Quote:

Originally Posted by cosmonate

btw: what exactly is the difference of kvm88 and qemu-kvm-devel-88 and qemu-kvm-0.12.3 (the first two state to be development branch, the last states to be the stable but is the newest!?) i don't understand their release management!?

The old format combined a modified version of qemu and kvm into a single release. The folks at KVM have been merging their QEMU changes upstream to avoid forking a modified version of QEMU. They currently release in 2 parts and no longer support the kvm-xx release management format. In addition to installing qemu-kvm you might want to consider installing kvm-kmod which compiles newer version of the KVM modules (without impacting the Slackware shipped versions). You can find a build script at SlackBuilds.