Sorry I don't think I can help much since I'm not a network expert, nor am I familiar with vmserver.
However I do have some experience in these areas (vm, networking, etc.) so here it goes.
This is what I would try and do.. either together or separately.
- strip everything down as much as possible and add one new feature at a time, until you find the culprit.
- since it's deb based.. you should have a plethora of programs to trace your systems actions.
- what fs are you using.. if ext4 it needs to be set up right or it will actually be sloooower than 3.
- I am supposing vmserver is more appropriate for your task than kvm? All I know is, KVM on single system/servers is actually FASTER than on the *regular* OS from the partition!
- kernel config.. is it selected and *tuned* for your project(s)?
- extranious programs, daemons etc. that are using resources, or NOT using them when they should.. again, this is where there should be some programs available in apt to help.. sorry I can not suggest any, since I have not had the opportunity to evaluate any yet significantly.
Either way, lettuce know