What Traffic Shaping Approach for Interactivity?

tvynr · 10-13-2005, 05:08 PM

Hi, there! I've been retuning my router for a new home network lately and the demands of the other users are really pushing my traffic shaper to its limits. While we're still able to get fairly reasonable access to web pages and the like, the connection is behaving like a modem for things like SSH and games, not like a 1.5/768 cable line. This script worked fine when I was the only one beating it up, but it seems that it needs a bit more work to handle the new requirements.

The problem is that we are all bandwidth fiends, BitTorrent being the primary culprit. The original approach was to split the connection into three queuing disciplines. Handles 1:1 and 1:2 were SFQ and 1:3 was a TBF with a cap upstream speed at about 80% of our maximum upload capacity. This worked fine until the data really started coming down.

After all, there's not much I can do to regulate the downstream, right? I tried to do so anyway; I started policing traffic leaving my router's internal interface as well, hoping that the remote TCP/IP stack would get the idea when it stopped receiving ACKs and back off a bit. This was less than successful; there was little change.

Then it occurred to me that the ACK packets for the batch connections (BitTorrent, FTP data, etc.) were higher priority than the interactive connection data (data for SSH, Diablo II, etc.). So I created a new, much more complex heirarchy for both interfaces:

Code:

                         +-------+
                         | prio  |
                         |   1   |
                         +-------+
                           | | |
              +---1:1------+ | +---1:3-- (unused)
              |             1:2
              |              |
          +-------+      +-------+          +-------+
          | prio  |      |  tbf  |---12:1---| prio  |
          |  11   |      |  12   |          |  121  |
          +-------+      +-------+          +-------+
            | | |                             | | |
    +-------+ | +---11:3-- (unused)   +-------+ | +---121:3-- (unused)
  11:1      11:2                      |         |
    |         |                       |         |
+-------+ +-------+               +-------+ +-------+
|  sfq  | |  sfq  |               |  sfq  | |  sfq  |
|  111  | |  112  |               | 1211  | | 1212  |
+-------+ +-------+               +-------+ +-------+

Interactive ACK as well as SSH were routed to 111. Interactive connections of other kinds were routed to 112. Batch ACK and all SYN were routed to 1211. All other batch was routed to 1212.

If anything, this was worse, but I don't understand why. My understanding of the "prio" qdisc suggests to me it's a strict priority queue; if interactive data is waiting to be sent, it will go next, regardless of how much batch data there is or how long its been waiting. But no dice.

Has anyone dealt with this before? Does anyone have any suggestions? Moderating the downstream of each individual machine is inefficient as well as impractical, so I'm trying to determine some way to handle this all from the router.

Any advice will be appreciated quite a lot. I've kind of exhausted my list of ideas.

Cheers!

SirGertrude · 10-13-2005, 07:38 PM

From what I understand your problems come from a saturated downstream connection. Using policing and/or shaping you should be able to keep your upstream under control.

Here is my input on your issue:

Policing downstream traffic works if you have a 1:1 TCP connection.

Example: You are downloading a file from a sourceforge.net mirror. You have a single connection between you and that mirror. If you are policing the inbound data you essentially drop excess packets causing the TCP protocol to adjust window sizes and slow down the transfer. This is effective because windowing is done between you and a single host.

The problem I see is policing traffic from multiple sources as in torrent downloads.

Example: If you have 20 people sending you torrent data that means you have 20 open connections each with different window sizes, etc. Obviously you are limiting bandwidth based on destination in this case� whether it be IP address or port (port being the best option). When the policer drops packets, it is not done in any uniform manor� they are just dropped when the speed is in excess of its limit. If you drop 300 packets they may have all come from a single host. This would mean that the other hosts would increase their windows sizes and send more data. It would be a never ending battle b/c from the sending hosts perspective there is no consistent packet loss� only random packet loss. Some hosts would send faster while others would send slower for a short period of time. This seems very inefficient as your users will only see a small percentage of their overall traffic while a huge percentage is dropped at your expense.

The solution:

I don�t know, I could only assume that the bandwidth limiter in your torrent program is more efficient in its limiting abilities.

Now keep in mind that this is all theory, I just brewed this up in my head. This may not be how things happen, and if I am way off on my thinking let me know. If this is correct then I would be interested in knowing how to address the problem, or at least mitigate it. My goal here is to shed some light on the topic so people smarter than me can offer a solution.

I hope this gets us some answers

tvynr · 10-13-2005, 08:28 PM

Actually, your description of the problem seems to be what I witnessed. I monitored the download speed of one of the BitTorrent clients and observed it decreasing when I implemented the download policing solution. However, the overall traffic on my interfaces did not change significantly and interactivity was still low... actually, it seemed to get a bit worse.

I appreciate your thoughts on this matter. It's nice to have independent confirmation of my own suspicions, at the very least. You are correct in that the upstream is controllable (it is presently controlled quite well).

The thing that confuses me about this is that there must be a solution. After all, I've seen a number of large networks (such as the one at my old college) with thousands of users; they have successfully screened downstream traffic so as to allow interactivity despite a great deal of network beating by the users themselves. So there is a solution... we just need to find it.

Thanks again for your response. If I discover anything I will, of course, post it here.

Cheers!

rlh989 · 10-14-2005, 08:21 AM

http://l7-filter.sourceforge.net/

This works very well when setup right, i leave BT client on full time and browsing and games are just fine latency wise. The only problem is now BT is starting to encrypt headers which makes this approach fail for now.... Very few BT clients can do this thou atm...

tvynr · 10-14-2005, 02:18 PM

Thank you for your response. I am a bit confused, however. Presently, I am not having any trouble identifying the traffic (since my users are willing to stay on a set of predetermined ports). The problem is that the traffic is saturating my downstream and I'm not sure how to regulate this. Are there any specific parameters of setup that you use? Or have I misunderstood the purpose of the filter?

Thank you for the link regardless. If I start having trouble with misidentified traffic (people setting their BitTorrent server port to port 80 or something) then this will be very handy.

Cheers!

s_c_braithwaite · 10-15-2005, 08:28 AM

A Mice and Elephants queueing discipline should be most suitable for interactive traffic.

Unfortunately Bit Torrent creates so many flows. with different port and ip numbers, so it would be a most difficult case. However, you may want to look at the following link:-

http://www.sci.usq.edu.au/staff/brai...roj/index.html