Hi folks -- I've been teaching my self about traffic control and want to take advantage of the queueing built into the kernel. Powerful stuff, but, I'd like to see if anyone has comments about the setup I've arrived at and perhaps any optimizations yet to be had.

My setup: Unlike most examples, I'm NOT using my box for a gateway, but, I do want to smartly throttle all egress traffic destined for the gateway router. Specifically, I want:
1) Local enet traffic to have full (~100mb) bandwidth
2) A high priority egress band for interactive and ACK traffic going out the DSL line
3) A low priority bulk egress band for all non-interactive traffic.
4) All combined upload DSL traffic, (sum of high & low bands) to be capped at ~33kBps to prevent queueing backup at the router.

So, here's the setup I arrived at. It seams to work fairly well. Please feel free to comment if there's a better way.

(BTW: I'm intentionally avoiding CBQ because it just seams far more comlex than needed)

Thanks for time!

~foobert


# Clear all queues before starting
tc qdisc del dev eth0 root

PEAK="33kbps"
#Define root level token bucket
# any traffic that misses every filter automatically ends up in low
# priority class (12) with this default
tc qdisc add dev eth0 root handle 1: htb default 12

# Setup the first node to limit all traffic to ~100 mb
tc class add dev eth0 parent 1: classid 1:1 htb rate 12mbps

# define a sub class that is capped at DSL upload rate
tc class add dev eth0 parent 1:1 classid 1:2 htb rate $PEAK

# devine a sub class from the combined DSL class for high priority traffics
tc class add dev eth0 parent 1:2 classid 1:11 htb rate 32kbps \
ceil $PEAK prio 1

#Define a sub class frin combined DSL for low priority bulk traffics
# This is throttled heavily if high priority traffic is present.
# Also it is never allowed to borrow the full upload bandwidth
# from hi-priority (probably overkill)
tc class add dev eth0 parent 1:2 classid 1:12 htb rate 1kbps \
ceil 28kbps prio 2

# Define a sub class for all local LAN traffic. It gets slightly
# less than max LAN bandwidth, insuring that DSL upload is always
# left with something.
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 11.8mbps

# Do sfq (round robin) queuing on all leaf nodes
tc qdisc add dev eth0 parent 1:10 handle 100: sfq perturb 10
tc qdisc add dev eth0 parent 1:11 handle 110: sfq perturb 10
tc qdisc add dev eth0 parent 1:12 handle 120: sfq perturb 10

#Filter all local lan traffic and put it in the 1:10 class
# This is given highest filter priority so it is triggered first, preventing
# LAN traffic from being snagged by the other filters.
tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32
match ip dst 192.168.0.0/16 flowid 1:10

#Filter SSH traffic into high priority DSL class (1:11)
# (technically not needed if you rely on TOS filter below, but, I'm paranoid)
tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 \
match ip dport 22 0XFFFF \
match ip sport 22 0xFFFF flowid 1:11

#Filter ICMP traffic into 1:11 to get good ping times.
tc filter add dev eth0 parent 1:0 protocol ip prio 12 u32 \
match ip protocol 1 0xff flowid 1:11

#filter ACK only packets into 1:11 to maintain high download throughput
tc filter add dev eth0 parent 1:0 protocol ip prio 13 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 flowid 1:11

#Filter "low latency" TOS traffic into 1:11
tc filter add dev eth0 parent 1:0 protocol ip prio 11 u32 \
match ip tos 0x10 0xff flowid 1:11

How many clients are on your network?

How many network devices are there on the network?

The network devices are the key to bandwidth question.

What type of network are you using? Wired or wireless?

Please keep in mind that network speeds are effected by many variables. One of the most overlooked variable types is enviroment. Line noise, static and defects in cabling can effect the speed of your network.

If you can account for all of these physical variables then you can find out how effective your code is.

Thorn

Originally posted by thorn168
How many clients are on your network? How many network devices are there on the network?
--> Generally 3 at the most. All of them surf the web through a squid proxy w/ adzapper running on this box. Generally, none of the other clients access the gateway directly (and if they do, I can deal with the router backing up in that case).

What type of network are you using? Wired or wireless?
--> 100mb wired.

Please keep in mind that network speeds are effected by many variables. One of the most overlooked variable types is enviroment. Line noise, static and defects in cabling can effect the speed of your network.

Thanks for the comment, but, I think that's splitting hairs. I'm not really asking how to eak out every last byte of throughput from anything. I am being conservative in my BW assignments (with the exception, perhaps, of the lan queue, which, shouldn't cause much issue if it does back-up -- a very rare event if it ever does happen). I'm simply trying to create an environment where I can have relatively snappy interactive response to the outside word through my DSL line while maintaining a fairly high BW "best effort" queue for the rest of the upload traffic.

I was more interested in the overall structure and setup of the queues and filters and wondered if there were any obvious "gotchas" that I wasn't aware of.

Cheers,

~john

"I was more interested in the overall structure and setup of the queues and filters and wondered if there were any obvious "gotchas" that I wasn't aware of."

I don't see any.

As for my comments on the physical environment...Sometimes it is a factor. You could have perfect code running the way it should and there could still be poor network performance because of issues with the network cables.

It is just something keep in mind when you measure the performance of networks.

Thorn

Looks good to me. I would add an extra line for all UDP traffic (protocol 17) so it goes smoothly like los TOS, ICMP, SSH. Why? Because many services require low latency connection using UDP which is faster than TCP. These are usually all kinds of chat, speech(VOIP), video chats, computer games. They do not use much bandwidth but require to go smoothly. Go ahead and test it - start downloading with maximum speed and try to play any internet game - it will lag unless you create a specific rule for UDP.

Good point Pave, thanks. This is especially true considering the change I implemented...


One thing I've noticed after putting it through more testing -- using SFQ on the bulk upload queue (120) causes HUGE queue lengths. It will run about 100-200 packets deep when I have heavy torrent traffic . Figure roughly 1300 byte average packet size (measured) and a drain rate of 25KB/s, that's a 5 - 10 second queue time. However, if some high-priority traffic comes along, the drain rate will drop substantially, and the queue time will easily go into minutes worth of time! This is not good at all.

Imagine this scenario -- a TCP connection is happily working along and dealing with a 5 second round-trip time. Thus it's retransmission timer is ohh, say ~15 seconds. It just put a packet into the queue. The queue time then goes to 2 minutes because of some heavy high-priority traffic. Thus, TCP will timeout, retransmit the packet, and double the retrans timer to 30 seconds. 30 seconds later it does the same thing. So, now we have the same packet (data) queued 3 times and the original packet hasn't even gone out yet (no dropping is happening with the SFQ, so it all must be sitting on the queue). Thus, this causes inefficient usage of the small upload bandwidth because it will unnecessarily send all these retransmissions over the wire.

I think a better approach is to have a relatively shallow bfifo queue of, say 9kB. This WILL cause packet drops when a connection tries to enqueue while the FIFO is full. At first glance, one might think this is a worse condition than having the very long latency. However, since the packet drop is happening before hitting the wire, the dropped packets are not wasting "precious" upload bandwidth. In the end, the same net result is achieved -- TCP sees one or more packet drops (via timeouts), thus, it enters slow-start mode with a congestion window of 1 SMSS (MTU) until it successfully gets a packet through. It will then have a congestion window that is more representative of the new (limited) bandwidth due to the high priority traffic being present. Additionally, the round trip times will be much more consistent and generally quite small all very Good Things for TCP.

Testing shows that the small bfifo queue is never empty (as expected) and the undisturbed upload rates are unchanged. When a high-priority disturbance shows up, the upload rates don't fall off as sharply as when using SFQ. I haven't done any formal testing, but, I'm not really THAT anal (contrary to what it may appear).

Cheers,

~john

Hi to everyone, i have been looking at your posts, and i had a really big problem.

The thing is i have a a net with 75 machines wired in a 100Mb LAN with some connected and some not to the web via 2 ADSL lines of small capacity. (2mb, and 3mb ).

I use Linux as gateway for all the net, so i was wondering if the use of QoS service from Linux can give me some help to manage all problems related to bulk traffic, games, and so on.

Last year i tried to set up HTB queques with not too much result, due to my inexperience, so i control the bandwitdh from each client by software applications.

I really think it can be done, but it takes quite a heavy work and knowledge that may be i don't have yet, so i wonder if anyone can give me a hint.

I still have a lot of work to do to get it right, but step by step y can improve each little thing, but time is what i have less.

So if anyone, has the will to think it over, not in deep, but superficially, i will really thanks for any idea.

I just want to have no delays to fresh connections, while downloads reach a decent rate, without conflicting other connections.

Thanks to all, you seem to be way more capacitated compared to me.
So HELP !!.