Router simulating Network Lag, for developing network code for games
 

Hi !

I'm a hobbiest game developer. I've got a problem where I'm searching for a solution for some time now. I'ld like to simulate the lag created by the
internet locally between (at least) two of my pcs. What I have in mind is
something I could plug in between the network adapters of the pcs and that's it.

The effects I'ld like to have simulated are:

- delayed transfer of packets. The time delta min and max should be
configurable so that the actually time a packet is hold back is like
random(mindelta,maxdelta)
- packet reordering (what basically happens automatically if packets get
delayed by different amounts of time)
- packet loss: a probability of a packet to get lost should be configurable
- is there something else that happens on the internet and might be good to test locally?

I'ld like to affect icmp, udp and tcp by that thing in the middle. As well,
the parameters should be configurable for both directions seperatly.
Maybe a plugin for linux's network bridging capabilites is the solution? Any
ideas/hints how to do this? Maybe there is already some software that does this and I didn't find it?

Because this targets for game development and the networking footprint of
games is generally quite low, there should be a lot of free cpu power to
accomplish this task.

The perfect solution would also allow to have more than 2 PCs. Also a fast
way to change the parameters without disconnecting the bridge would be cool. E.g. a small textmode frontend that can be accessed via telnet or ssh.

I hope you can help me. I guess such a thing will be helpful for all time
critical network related developments.

I don't know much about it myself, but I think the tc (traffic control) system on Linux should do what you want. I did a quick Google search which returned lots of good information.

Good luck, I'd be interested to know if that helps or not. :)

~sind

I had a look at the docu and it looks to me as I can shape the traffic in a way that I can define what package comes before which, but I can't find anything about introducing delaying times to packages. It seems as if tc always creates a behaviour that, if there is free bandwidth, it sends queued packages. Only if there isn't enough bandwidth, tc decides what packages should be sent first. I have plenty of bandwidth in my lan and still want to slow things down ... :Pengy:

I found

NIST
http://snad.ncsl.nist.gov/itg/nistnet/

It does exactly what I want. With nist, you can specify a delay and a sigma, as well as the chance for packet loss and even duplication per host<->host connection. The pc running nist has to be the router between those hosts. It is really easy to use, once you got it compiled. I used the 2.6.13-kanotix-10, even though 2.6.11 was the last mentioned on the nist page.

I strongly suggest such a router for everyone developing time critical networking code, e.g. for games. You can change the values within seconds via a cmdline tool as well as the included x-based dialog.

A sad thing that it is rather complicated to install (at least for me it was, I'm not fiddeling with packages etc. that much). A debian package would be awesome. ;)

This is fantastic. I never thought about using a linux box to create these types of issues.

I'll be using this in my network troubleshooting classes from now on.

GREAT IDEA!

Thanks!

MrKnisely

Oh nice to see that my question is usefull for others too.

I tried to change the thread's topic from Bridge to Router because actually NIST Net doesn't support layer 2 yet. But it only changed the heading of my initial post. Maybe some mod could change it?

According to its page, bridging is planed for the future though. For now NIST routes, it doesn't bridge.

A lag bridge would be fantastic... I can think of some fun things to do with that in my office.

You can add latency using Cisco routers too... but it adds the same latency to each packet. This is used with VoIP to smooth out jitter. This is as close as I've found to this functionality.

If you have any notes on your build, I'd greatly appreciate it. I'll be using a debain based distro too.

MrKnisely

How does it smooth out jitter by adding a constant delay to every packet? Or do you mean, it adds such an delay to each packet that from the point of view of the voipphone the packets arrive in a constant frequency? However, I'ld assume that voipphones have such a buffer interally themself. :confused:

Bridging would be easier to set up if you have 2 physical independent nets. If you have just one physical net and just set up 2 independent nets on it, you need a router to connect them. However, you have to add routs to all the clients etc.

Did you already try to install it? I'm not what details you need. Ask me if you run into some problems.

Depending on your distribution, you have to make sure that you have the srces of your kernel isntalled as well as the version of the compiler that compiled the kernel. If you compiled your kernel yourself, then you probably have fitting versions installed. If you use a distro with a precompiled kernel and updated gcc to 4.0.x, like I had, then it won't work. Well that's a general additional modules compilation problem though. But for that was the point I took longest to figure it out.

When you checked those versions, just follow http://snad.ncsl.nist.gov/itg/nistnet/install.html step by step and ignore the "up to 2.6.11" comment, I have got it working with 2.6.13.

When you call make, it probably complaines about some missing files. You can run konquerer apt:/ and search for files. It will display you the package where the missing file is in. Do an apt-get install {packetname} on that. Repeat until make completes. I had to do it 3x: libxmu-dev, libxaw7-dev and xaw3dg-dev were missing on my kanotix (however, I installed some x-stuff bevor manually aswell and maybe nist neads that too. In that case, a few more interations are needed). You also have to look exactly for what is missing. Basically, you can get a message for a missing file from gcc or from the linker. The last one is a bit trickier to read ;)

That's what I remember just now. You have to setup at least to subnets like you do it generally in linux too. Afterwards load that module with the Load.Nistnet-script and set parameters with the grafical or cmdline tool.

Maybe that was a bit too much in detail. Or maybe even not. I don't know what you know. :cool:

No, I haven't tried installing yet. I'll build a system late next week. I'm taking two of the 4 CCNP test (Tuesday and Wednesday) so my spare time is spent studying for those for now. I'll definitely post back when I'm building.

Regarding jitter... Yes, that is the exact reson for the jitter buffer. Our ears can deal with lag in a conversation best when that lag is constant. It is easier for us to deal with 300ms of lag than a range of 100-250ms. Our brains just get calibrated to deal, and it isn't an issue anymore. No, the phones I've worked with don't buffer in that way. The jitter buffer needs to be done at the switch/router level. I don't know why this is the case. It does seem smart to smooth out the jitter at the phone itself. It's the only way to ensure a constant lag in the conversation... but when has Cisco done anything that made 100% sense?

Your description was right on for detail. I just wanted to find our what you ran into so I could just have a heads-up. Missing modules, dependency problems, kernel compilation and such was what I was looking for.

Regarding the lag bridge... you wouldn't need two subnets. Everything would be in one network, the lag would be in layer-2 frame switching. You would use two switches/hubs and place the lag-bridge inbetween. When the ethernet frame entered the first switch the switch would look for the MAC of the destination IP. Since it would be on the other side of he lag bridge the bridge would respond, via ARP proxy if setup, or the frame would be broadcasted out all ports. Either way, it hits the lag-bridge. The bridge would then hold the frame for the random ammount of time applying all the rules of the lag/drop. Once the rules have completed the frame would then be released to the second switch/hub to be forwarded to the destination. There would be no layer-3 routing process at all. All you really save is not needing to add a route statement to the participating workstations and you keep everything in one address space.

As it stands, I think this solution will work better for me. I'll just setup two VLANS and have this box route between the two. I'd like to have this thing understand 802.1q fram tagging so I can just pass a trunk port to the device and have sub-interfaces on the one network card. That way I wouldn't be loosing two switchports to the lag-router.... but that may be too much for this project right now.

Do you have any experinence working with VLANS in Linux? I know it can be done, I've just not tried it.

MrKnisely

That's what I meant with 2 independent nets == 2 swichtes/hubs. :)

What exactly is a VLAN? VPN + something that is exactly like VPN but without encryption? If you do your things locally, you can just add some routes here and there. That's what I did. I don't know if that is already a VLAN then.

VLAN is a "Virtual Local Area Network"

This is done using a "frame tagging" method if you use the 802.1q (dot1q). This is the open standard that most vendors use. There is also a Cisco proprietary call ISL (I think) that completely encapsulates the ethernet frame. This is seldom used outside the Cisco world, and even Cisco has moved to dot1q on most of their equipment now.

What this method of frame tagging does is allow you to tell a switch, port 1 is in logical network 1; Port 2 is lin logical network 2; Port3 is in logical network 1; etc. What then happens is that the ports in the same VLAN can talk to one another, and the ports in different VLANS can not talk unless they go through a Layer3 routing process. This is an easy way to logically segragate a network.

Even if a machine addresses himself with an IP of another VLAN, his traffic will not move over, since it lacks the proper fram tag.

There is another idea, trunking. This is an interface where all VLANS can traverse. These ports are used to move all vlan traffic from one switch to another switch. So ports on VLAN1 on Swtch1 can talk to ports on VLAN1 on Switch2. It is a wonderfull way to break up a network. You can also pass one of the Trunk ports over to a router and setup subinterfaces on that router to route between VLANs. The great part about this is that you only need to use up one of your switch ports to connect multiple networks. Think about it, if you had 5 networks you wanted to connect, without this you would need a router with 5 ports and to use up 5 switch ports. What a resurce waste.

Newer multi-layer switches are capable of providing this routing function within themselves. With a Cisco device you create a virtual interface called a Switched Virtual Interface (SVI). Once you have done that, you put the proper IP address with that vlan and you enable routing. The traffic coming in on VLAN1 destined for VLAN2 would come in via the SVI for VLAN1 it would then hit the routing process and be handed to the SVI for VLAN2 and then traverse the network through that VLAN and trunking ports that carry that VLAN until it got to its destination. It is truely and elegent solution. Cisco has done even better than this too... They provide for something called Cisco Express Forwarding (CEF). This allows for an area of memory called Terniary Content Addressable Memory (TCAM) to buid a tabe of devices and associated information like VLAN, Port, MAC, Filtering Rules to get there, etc and put it all into this table. Once a frame comes in it is evaluated against the table, all done in hardware, and the frame is then forwarded. This allows for frames to be forwarded from VLAN to VLAN at near wire speed. This takes the latency out of the routing process. This is why you want your jitter buffer done at the access switch where the phone device is connected. Once it hits one of these layer 3 devices, the frame is moved VERY quickly, and any ammount of buffering would not only slow the one packet, but it would force software evaluation of every packe thereby not allowing CEF to do its job.

Well, I know I went well beyond your question; I just LOVE infrastructure.

Infrastructure is an amazing world that programmers seldom look at. I am impressed that you are diving as deep as your are. Maybe one day, I'll look into the programming world... right now I'm blissfully ignorant of programming of any sort.

MrKnisely

Ah so its basically a way to split switches up in virtual subswitches in a transparent way. Interesting, my cheat 8 port switch can't do such things. ;)

Network topology is something you need to handle if you want to write software for it. I don't like it that much, especially if I have to change some wires and all the physical work related to it. But something like NIST Net is pretty usefull for my needs and I didn't had to rewire anything so I gave it a try :)

Will do.

You can get some cheap switches that can do this. I've got a few Cisco switches 1900, 2900, 2950 and all of them can do this. You can get these devices on Ebay starting at $20 and shipping.

Linksys, Netgear, and D-Link all make switches that can do VLANs. In all, they're somewhat inexpensive but not cheap. Expect to pay aroud $100 to start.

MrKnisely

Just thought I would mention something I noticed whilst (re)setting-up iptables rules today. There is a match module (called 'random') that will match a given percentage of packets. So to lose say 10% of all incoming packets you could use something like -m random 10 -j DROP. You could add more criteria to only drop from a certain host etc.

Hope that is some use.

Interesting. Does it also help you somehow to produce lag?