Debian Lenny, Xen 3.0, pciback - Network speed
Hello everybody at linuxquestions.org.
I got a little question and also a small "how-to" for pciback and how I got it working.
First off, the question:
I've now used pciback to hide my network card from my Dom0 and gotten my DomU to access it. And it works! I've got maybe transfer speed ~15MB/sec faster, from ~20MB/sec to about ~35MB/sec. This is kinda great but not as fast as I want it. My LAN uses CAT 6 only and my switches are Gbit. I do also use a dedicated striped 500GB pair on my DomU, and I get speeds over 120MB/sec on the raid from the DomU. I've also dedicated 8 cores on about 2.6GHz each and 4GB memory. I've tried both ISCASI and samba, and they get about the same speed. ISCASI is a "little" more stable and locks at 36MB/sec when samba jumps between 33MB/sec - 36MB/sec. I've also used both ethtool and mii-tool and both show 1000Mb/sec (1000baseT-FD).
And my question is: Is there some way to get an even greater speed? Any suggestions are welcome!
How I got this working: (all of this is done on the Dom0)
First off, I had to find what code my NIC had. In this case I had 07:01.0. You'll have to enter lspci and find your NIC there and remeber it's pci-code or whatever it is.
In my case it looked like this:
07:01.0 Ethernet controller : Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet [14e4:16a7] (rev 02)
Now when I know what pci-code it got (I will call it pci-code from now on, even if it's not correct) I'll have to get pciback working, so it'll hide the NIC from the Dom0, and make it possible for the DomU to use it.
I went to /etc/initramfs-tools/modules and entered pciback at the bottom. (echo pciback >> /etc/initramfs-tools/modules)
When this was done, I had to create a new initrd.img which with the new module. In my case, I created one with the ending -pciback, in case something went wrong.
mkinitramfs -o /boot/initrd.img-`uname -r`-pciback `uname -r`
Now I edited my grub menu and changed it from:
module /boot/vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda1 ro console=tty0
module /boot/vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda1 ro console=tty0 pciback.hide=(07:01.0)
Now you should change pciback.hide=(07:01.0) to your own pci-code. And correct me if I'm wrong. To hide two pci-codes just do it like this: pciback.hide=(07:01.0)(08:01.0)
Now, if you reboot, the NIC (or whatever pci-hardware you're using) will be hidden from the Dom0 and you'll be able to give your DomU access to it.
But I'll tell you what I did before rebooting.
Because I use three nics, I got a small multiple-network-bridges script in /etc/xen/scripts/ which let's the DomU's access all my NICs. But now, only one DomU will be able to use it. So I had to remove it from my script.
I also had to remove it from /etc/network/interfaces (I use Debian) and also all DomU configurations in /etc/xen (.cfg's).
When this was done, I went to my DomU configuration, removed the vif = [ '<some stuff here>' ] and added pci = [ '07:01.0' ].
But a tips before rebooting or anything. Install all modules on your DomU or else the NIC won't work. In my case (on the DomU, before reboot):
apt-get install linux-modules-`uname -r`
Now reboot your Dom0! (I really hope it works for you too!)
Hopefully this can be to some kind of help to other people. Since I had kind of a hard time figuring it out.
Please tell me if I've done some mistake when writing this and I'll correct it. And I'll try to help as much as possible on the forums.