LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 09-20-2017, 07:59 AM   #1
Chris@Work
LQ Newbie
 
Registered: Sep 2017
Posts: 4

Rep: Reputation: Disabled
Understanding of network limitation when trying to maximize pps in VM


Hello All,

I have been struggling for days trying to find out how to optimize packet sending on my virtual server. I have made a small c++ script that does something like : while (1) { send-empty-udp-packet; }.
When specifying my own eth0 IP, I reach about 400k pps. When I use another IP (hosted on another hypervisor), I get down to 100k pps. I made sure that trafic is one way only.

I am far to use the bandwidth (100M on a 10Gb nic) or the vcpu (10%) nor the pcpu (10% of one core). I don't see any drop, it is just slower.
I tried boosting the local queue (wmem_max, netdev_max_backlog) but without success (which is, IMHO, normal as it is not a buffer full issue that would lead to packet drops)

The VM (Centos 7 with kernel 3.10) uses virtio which does not offer as much capabilities as a pnic. qemu is used (but the VM is alone on the HV), and the hardware is a Cisco UCS. There's a Linux bridge along with an openvswitch on the path too.

What I don't understand is, why my program does not send packets at full speed ? It should send at max speed all the time, the system would just drop packets that does not fit the buffer. But that's clearly not the behavior.

My guess would be some sort of latency, but I don't have anything that proves that.

What am I missing ?
 
Old 09-21-2017, 03:13 PM   #2
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,839

Rep: Reputation: 160Reputation: 160
What's pps of your script on host machine?
 
Old 09-22-2017, 04:45 AM   #3
Chris@Work
LQ Newbie
 
Registered: Sep 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
it actually depends on the interface I test it on. But I got at least 200k pps on one of them.

There's something I didn't try that I'll do as soon as I can is to have 2 VM on the same host and send packets from on to another. That way it will use only the TAP and openvswitch. No linux bridge nor physical interface involved.
 
Old 09-22-2017, 06:01 PM   #4
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,839

Rep: Reputation: 160Reputation: 160
If your packet size is 1500 bytes, 400k pps is equal to 400*1024*1500*8, 4.9G bps. One thread might be normal.
 
Old 09-22-2017, 07:47 PM   #5
Chris@Work
LQ Newbie
 
Registered: Sep 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
it is empty udp packets (so less than 50B with the header). Bandwidth is not an issue.
I did try without going to the linux bridge but only open vswitch and I achieved something better (260k pps) but still not what I expect anyway.
 
Old 09-25-2017, 03:25 PM   #6
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,839

Rep: Reputation: 160Reputation: 160
If empty UDP packet go to physical interface on host machine and pps is low, it might be issue. But if it go through like open vswitch, it is a switching stack for hardware virtualization. Software is involved, it is hardly to say that it is problem or not.
 
Old 10-13-2017, 10:51 AM   #7
Chris@Work
LQ Newbie
 
Registered: Sep 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
Just for information, I found Something.
Qemu is configured with vhost=on. Which mean a kernel process is created to manage the tap interface. This process is switching cpu during my test, which is BAD.
I use taskset to pin the process to a cpu and achieved +80% of throughput (470kpps for two VMs on the same ovs). I still have insufficient throughput when going through the full chain (vm, ovs, linux bridge, phys interface and phy switch and same up), but it's a nice improvment.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
PPS TOOLS with PPS via DCD RS232 with a sure GPS BOARD - PPSTEST problem ofrappier Linux - Software 0 01-09-2013 11:44 AM
understanding my network siva balan Linux - Networking 3 11-16-2011 09:01 AM
Understanding this network Hyakutake Linux - Networking 1 09-30-2011 06:58 PM
understanding the network manager nkd Ubuntu 6 04-04-2011 02:57 PM
local network understanding bloodyscript Linux - Networking 3 07-17-2007 04:18 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 10:04 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration