[SOLVED] tc cbq classes not obeying rates

kslen · 06-30-2010, 04:28 PM

Problem:
I had managed to get traffic shaping through tc with cbq up and running. When all classes were 'bounded'; traffic obeyed rate limits as 'bounded' classes must obey their rates. When classes weren't 'bounded' I were puzzled by the fact that 100% of my connections capacity was in use instead of 75% as the total sum of all the classes rates represent.

Solution:
I was missing was a class which limits all of the priority classes to 75% of the connections capacity...

Current tc script:

Code:

#!/bin/bash

# iptables:
#
# Shape all traffic outbound on the external interface according to classid 1:13
#  iptables -t mangle -A FORWARD -p ALL -s 192.168.1.0/24 ! -d 192.168.1.0/24 -j MARK --set-mark 13
#  iptables -t mangle -A OUTPUT -p ALL ! -d 192.168.1.0/24 -j MARK --set-mark 13
#
# Shape all traffic inbound to the internal interface according to classid 2:23
# Note: This means no shaping on what is downloaded to the router machine locally.
#  iptables -t mangle -A POSTROUTING -p ALL ! -s 192.168.1.0/24 -d 192.168.1.0/24 -j MARK --set-mark 23

# Let's shape!
TC=/sbin/tc

IFACE_NET=eth0 # Outbound traffic
IFACE_LAN=eth1 # Inbound traffic

tc_start() {
   # root qdisc     1:0
   # (No shaping)    |
   # child class    1:1--------------------
   # (Max rate)    /   \     \      \      \
   # leaf classes /     \     \      \      \
   # (Shaping)  1:11   1:12   1:13   1:14   1:15
   #
   # Internet Interface
   $TC qdisc add dev $IFACE_NET root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8

#### This is the class which was missing on the outbound interface:
   $TC class add dev $IFACE_NET parent 1:0 classid 1:1  cbq bandwidth 1000Mbit rate 97kbps weight 9kb allot 1514 avpkt 1000 bounded

   $TC class add dev $IFACE_NET parent 1:1 classid 1:11 cbq bandwidth 1000Mbit rate 50kbps allot 1514 avpkt 1000 isolated
   $TC class add dev $IFACE_NET parent 1:1 classid 1:12 cbq bandwidth 1000Mbit rate 15kbps allot 1514 avpkt 1000
   $TC class add dev $IFACE_NET parent 1:1 classid 1:13 cbq bandwidth 1000Mbit rate 15kbps allot 1514 avpkt 1000
   $TC class add dev $IFACE_NET parent 1:1 classid 1:14 cbq bandwidth 1000Mbit rate 10kbps allot 1514 avpkt 1000
   $TC class add dev $IFACE_NET parent 1:1 classid 1:15 cbq bandwidth 1000Mbit rate 7kbps allot 1514 avpkt 1000
   $TC filter add dev $IFACE_NET parent 1:0 protocol ip handle 11 fw flowid 1:11
   $TC filter add dev $IFACE_NET parent 1:0 protocol ip handle 12 fw flowid 1:12
   $TC filter add dev $IFACE_NET parent 1:0 protocol ip handle 13 fw flowid 1:13
   $TC filter add dev $IFACE_NET parent 1:0 protocol ip handle 14 fw flowid 1:14
   $TC filter add dev $IFACE_NET parent 1:0 protocol ip handle 15 fw flowid 1:15
   #
   # LAN Interface
   $TC qdisc add dev $IFACE_LAN root handle 2:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8

#### This is the class which was missing on the inbound interface:
   $TC class add dev $IFACE_LAN parent 2:0 classid 2:2  cbq bandwidth 1000Mbit rate 1275kbps weight 127kb allot 1514 avpkt 1000 bounded

   $TC class add dev $IFACE_LAN parent 2:2 classid 2:21 cbq bandwidth 1000Mbit rate 550kbps allot 1514 avpkt 1000 isolated
   $TC class add dev $IFACE_LAN parent 2:2 classid 2:22 cbq bandwidth 1000Mbit rate 50kbps  allot 1514 avpkt 1000
   $TC class add dev $IFACE_LAN parent 2:2 classid 2:23 cbq bandwidth 1000Mbit rate 275kbps allot 1514 avpkt 1000
   $TC class add dev $IFACE_LAN parent 2:2 classid 2:24 cbq bandwidth 1000Mbit rate 275kbps allot 1514 avpkt 1000
   $TC class add dev $IFACE_LAN parent 2:2 classid 2:25 cbq bandwidth 1000Mbit rate 125kbps allot 1514 avpkt 1000
   $TC filter add dev $IFACE_LAN parent 2:0 protocol ip handle 21 fw flowid 2:21
   $TC filter add dev $IFACE_LAN parent 2:0 protocol ip handle 22 fw flowid 2:22
   $TC filter add dev $IFACE_LAN parent 2:0 protocol ip handle 23 fw flowid 2:23
   $TC filter add dev $IFACE_LAN parent 2:0 protocol ip handle 24 fw flowid 2:24
   $TC filter add dev $IFACE_LAN parent 2:0 protocol ip handle 25 fw flowid 2:25
}

tc_stop() {
    $TC qdisc del dev $IFACE_NET root
    $TC qdisc del dev $IFACE_LAN root
}

tc_restart() {
    tc_stop
    sleep 1
    tc_start
}

tc_show() {
    echo ""
    echo $IFACE_NET
    $TC qdisc show dev $IFACE_NET
    $TC class show dev $IFACE_NET
    $TC filter show dev $IFACE_NET
    echo ""
    echo $IFACE_LAN
    $TC qdisc show dev $IFACE_LAN
    $TC class show dev $IFACE_LAN
    $TC filter show dev $IFACE_LAN
    echo ""
}

case "$1" in
  start)
    echo -n "Starting traffic shaping: "
    tc_start
    echo "done"
    ;;
  stop)
    echo -n "Stopping traffic shaping: "
    tc_stop
    echo "done"
    ;;
  restart)
    echo -n "Restarting traffic shaping: "
    tc_restart
    echo "done"
    ;;
  show)
    tc_show
    ;;
  *)
    echo "Usage: /etc/rc.d/rc.tc {start|stop|restart|show}"
    ;;

esac

exit 0

Limiting the connection to 75% of capacity...? Why?
Packet loss drops to 0% and ping responses are rock steady (given that your connection and the server you're pinging is known to not suffer from packet loss and/or unstable ping responses) even when there's a healthy amount of unrelated traffic entering and leaving the local network.

Are there caveats which might end up throwing endless hours of sheer pain out the window?
It is important to note that you can only shape inbound traffic (to the best of my knowledge anyway) on the interface which is connected to the local network (eth1 in my case) and only shape outbound traffic on the interface which is connected to the rest of the world (eth0).
Traffic which is inbound to the router itself (localhost) does not pass through the iptables PREROUTING and INPUT chains and thus that traffic can't be shaped (as far as I know).
Traffic destined to the local network (eth1) passes the chains PREROUTING, FORWARD and POSTROUTING which makes it possible to restrict the inbound rate on those packets.

Short version: We have no control over the rate at which the traffic arrives to routers localhost but we can "slow down" traffic after it has hit the router and is passing through on its way to the eth1 interface. However, traffic leaving the routers localhost _does_ pass through the OUTPUT chain before joining the other traffic in the POSTROUTING chain which makes it possible to mark and shape those packets.

Hope this is of help and if anything is unclear or simply wrong, please do let me know.

MensaWater · 07-06-2010, 01:05 PM

Rather than deleting your question you should re-post it then post your solution. That way if anyone else runs into the issue you did they may resolve it the same way. While the answer may have appeared obvious to you in hindsight, it might not to potential newbies.

kslen · 07-11-2010, 01:49 AM

Done.

abhelp · 07-14-2010, 06:23 PM

Hello
Thanks again for the complete post.
I am modifying your scripts for my setup.I do not need to shape inbound traffic. Also I have simplified it to not add so many leaf classes. All I want to do is limit different rates on eth0 to test our software.
So essentially I can have only one leaf class at one fixed rate. I can then change the rate for this class. Now my question is how did you test if the bandwidth was limited to that rate. So for e.g if I set maximum to 30kbps on the child class and lets say 10kbps on the leaf class it should be between 10 to about 30. I tried with wget ftp://filename to download a file but shows much higher rate than this. How did you test the download rate? I hope above makes sense. I am new to tc class framework though I have read the documentation.

Thanks a lot for your help.

kslen · 07-15-2010, 10:45 AM

Don't worry. I still don't quite have the hang of it all myself to be honest, but it seems to work as it should here so I'll do what I can to assist.

If I understand correctly, what you want is one child class and two leaf classes. One leaf class is designated to testing and the other leaf is for general traffic on the outbound interface. This should be achieved by removing all but two of the leaf classes and marking the packets with iptables as it seems you have done.

Make sure that the following statements matches your firewall rules:
To shape traffic leaving the router machines localhost, mark packets in the OUTPUT chain.
To shape traffic leaving another machine connected to the router, mark packets in the FORWARD chain.

One way to check if traffic is passing through your classes is to set both leaf classes 'bounded', restart tc and generate some traffic to see if the class rates are being obeyed. My favorite tool for this is 'iftop'.

Code:

iftop -P -B -i <ifname>

Another way to check if packets are passing through your classes is by poking tc every second or so for information.

Code:

watch -n 1 /sbin/tc -s -d class show dev eth0

Make sure your child class is 'bounded' during and after testing or the leaf classes will have no supreme overlord to obey. A class have to be 'bounded' in order to not exceed the specified rate limit. If they aren't bound, the child and leafs will borrow whatever bandwidth is available from the root qdisc.

I corrected a minor error in the script. Have a look at it and look for the underlined weight statements. This did however not affect the overall performance, at least as far as I have noticed.