tc rate control over network

abhelp · 07-02-2010, 03:11 AM

Hello all

I want to implement rate control over network interface. So I have configured Linux PC as a router with netem installed and having two interface cards.

device1----> Linux router with netem ---> device2

device1 connects to eth0 and device2 connects to eth1 of the Linux router. eth0 is configured to connect to the internet and eth1 has a static IP address on a local network. I want to limit bandwidth on devices connected to eth0.

So I applied the below rules using tc and tbf.

tc qdisc add dev eth0 root handle 1: prio
tc qdisc add dev eth0 parent 1:3 handle 30: netem
tc qdisc add dev eth0 parent 30:1 tbf rate 256kbit buffer 1600 limit 3000

Will the above work or should I use htb instead. I want to simulate the network conditions using different bandwidths.

tc qdisc add dev eth0 root handle 1: htb
tc class add dev eth0 parent 1: classid 1:1 htb rate 1024kbps
tc class add dev eth0 parent 1:1 classid 1:5 htb rate 512kbps ceil 640kbps prio 1
tc class add dev eth0 parent 1:1 classid 1:6 htb rate 100kbps ceil 160kbps prio 0
tc filter add dev eth0 parent 1:0 prio 1 protocol ip handle 5 fw flowid 1:5
tc filter add dev eth0 parent 1:0 prio 0 protocol ip handle 6 fw flowid 1:6

What is the best way of doing this?

Thanks

kslen · 07-02-2010, 05:03 PM

Hello there!

This is my interpretation of tc after many hours of torture.

http://www.linuxquestions.org/questi...-rates-817246/

Note that the leaf classes are _not_ bounded or isolated except 1:11 and 2:21. This means all classes can borrow bandwidth from each other, but no other class can borrow from 1:11 and 2:21 as they are isolated.

Hope this helps.

abhelp · 07-06-2010, 01:44 AM

Hello

Thanks a lot for your reply. Any particular reason for using cbq versus htb or tbf?

kslen · 07-11-2010, 02:51 AM

Hello again!

Pardon the late reply. It has been a busy weekend.

Quote:

Any particular reason for using cbq versus htb or tbf?

Not really. The documentation I found to be the easiest to wrap my head around was that which were written for cbq, yet htb and tbf are supposed to be easier even according to the documentation explaining cbq. Makes no sense, but so it is.

Werther better or easier to find, I'm not sure which is true. I ended up with cbq and it works great.

Oh, btw.
I moved the script to my initial thread so that I don't have to maintain two copies if I should happen to encounter any errors or expand it. So far the script is working as advertised though.

abhelp · 07-14-2010, 06:25 PM

Hello
Thanks again for the complete post.
I am modifying your scripts for my setup.I do not need to shape inbound traffic. Also I have simplified it to not add so many leaf classes. All I want to do is limit different rates on eth0 to test our software.
So essentially I can have only one leaf class at one fixed rate. I can then change the rate for this class. Now my question is how did you test if the bandwidth was limited to that rate. So for e.g if I set maximum to 30kbps on the child class and lets say 10kbps on the leaf class it should be between 10 to about 30. I tried with wget ftp://filename to download a file but shows much higher rate than this. How did you test the download rate? I hope above makes sense. I am new to tc class framework though I have read the documentation.

Thanks a lot for your help.

kslen · 07-15-2010, 11:12 AM

Have a look at the other thread for my reply.

http://www.linuxquestions.org/questi...-rates-817246/

apnicsolutions · 07-17-2010, 01:29 PM

Please find my script as follows for bandwidth rate limiting. I used cbq first, maybe I misconfigured it but after enough work and research, it wasn't doing what it was supposed to, in terms of limits that I'd set. Here's an HTB one that works well for me:

# Set some variables
#!/bin/bash
EXT_IFACE=”eth0″
INT_IFACE=”eth1″
TC=”tc”
UNITS=”kbit”
LINE=”10000″ #maximum ext link speed
LIMIT=”5000″ #maximum that we’ll allow
# Set some variables for individual “classes” that we’ll use to shape internal upload speed, i.e. shaping eth0
CLS1_RATE=”200″ # High Priority traffic class has 200kbit
CLS2_RATE=”300″ # Medium Priority class has 300kbit
CLS3_RATE=”4500″ # Bulk class has 4500kbit
# (We’ll set which ones can borrow from which later)
# Set some variables for individual “classes” that we’ll use to shape internal download speed, i.e. shaping eth1
INT_CLS1_RATE=”1000″ #Priority
INT_CLS2_RATE=”4000″ #Bulk
# Delete current qdiscs. i.e. clean up
${TC} qdisc del dev ${INT_IFACE} root
${TC} qdisc del dev ${EXT_IFACE} root
# Attach root qdiscs. We are using HTB here, and attaching this qdisc to both interfaces. We’ll label it “1:0″
${TC} qdisc add dev ${INT_IFACE} root handle 1:0 htb
${TC} qdisc add dev ${EXT_IFACE} root handle 1:0 htb
# Create root classes, with the maximum limits defined
# One for eth1
${TC} class add dev ${INT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}
# One for eth0
${TC} class add dev ${EXT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}
# Create child classes
# These are for our internal interface eth1
# Create a class labelled “1:2″ and give it the limit defined above
${TC} class add dev ${INT_IFACE} parent 1:1 classid 1:2 htb rate ${INT_CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}
# Create a class labelled “1:3″ and give it the limit defined above
${TC} class add dev ${INT_IFACE} parent 1:1 classid 1:3 htb rate ${INT_CLS2_RATE}${UNITS} ceil ${INT_CLS2_RATE}${UNITS}
# EXT_IF (upload) now. We also set which classes can borrow and lend.
# This class is guaranteed 200kbit and can burst up to 5000kbit if available
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:2 htb rate ${CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}
# This class is guaranteed 300kbit and can burst up to 5000kbit-200kbit=4800kbit if available
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:3 htb rate ${CLS2_RATE}${UNITS} ceil `echo ${LIMIT}-${CLS1_RATE}|bc`${UNITS}
# This class can is guaranteed 4500kbit and can’t burst past it (5000kbit-200kbit-300kbit=4500kbit).
# I.e. even if our bulk traffic goes crazy, the two classes above are still guaranteed availability.
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:4 htb rate ${CLS3_RATE}${UNITS} ceil `echo ${LIMIT}-${CLS1_RATE}-${CLS2_RATE}|bc`${UNITS}
# Add pfifo. Read more about pfifo elsewhere, it’s outside the scope of this howto.
${TC} qdisc add dev ${INT_IFACE} parent 1:2 handle 12: pfifo limit 10
${TC} qdisc add dev ${INT_IFACE} parent 1:3 handle 13: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:2 handle 12: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:3 handle 13: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:4 handle 14: pfifo limit 10
### Done adding all the classes, now set up some rules! ###
# INT_IFACE
# Note the ‘dst’ direction. Traffic that goes OUT of our internal interface and to our servers is out server’s download speed, so SOME_IMPORTANT_IP is allocated to 1:2 class for download.
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst SOME_IMPORTANT_IP/32 flowid 1:2
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst SOME_OTHER_IMPORTANT_IP/32 flowid 1:2
#All other servers download speed goes to 1:3 – not as important as the above two
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst 0.0.0.0/0 flowid 1:3
# EXT_IFACE
# Prioritize DNS requests
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_IP/32 match ip sport 53 0xffff flowid 1:2
# SSH is important
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_IP/32 match ip sport 22 0xffff flowid 1:2
# Our exim SMTP server is important too
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_MAILSERVER/32 match ip sport 25 0xffff flowid 1:3
# The bulk
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src 0.0.0.0/0 flowid 1:4

apnicservices · 07-18-2010, 11:03 AM

I used CBQ at first although couldn't get it to work the way I wanted. I spent a fair bit of time researching although ultimately the figures I put in didn't correspond to what was actually happening and I got stuck trying to debug. I rewrote in HTB as follows:

#!/bin/bash
EXT_IFACE=”eth0″
INT_IFACE=”eth1″
TC=”tc”
UNITS=”kbit”
LINE=”10000″ #maximum ext link speed
LIMIT=”5000″ #maximum that we’ll allow
# Set some variables for individual “classes” that we’ll use to shape internal upload speed, i.e. shaping eth0
CLS1_RATE=”200″ # High Priority traffic class has 200kbit
CLS2_RATE=”300″ # Medium Priority class has 300kbit
CLS3_RATE=”4500″ # Bulk class has 4500kbit
# (We’ll set which ones can borrow from which later)
# Set some variables for individual “classes” that we’ll use to shape internal download speed, i.e. shaping eth1
INT_CLS1_RATE=”1000″ #Priority
INT_CLS2_RATE=”4000″ #Bulk
# Delete current qdiscs. i.e. clean up
${TC} qdisc del dev ${INT_IFACE} root
${TC} qdisc del dev ${EXT_IFACE} root
# Attach root qdiscs. We are using HTB here, and attaching this qdisc to both interfaces. We’ll label it “1:0″
${TC} qdisc add dev ${INT_IFACE} root handle 1:0 htb
${TC} qdisc add dev ${EXT_IFACE} root handle 1:0 htb
# Create root classes, with the maximum limits defined
# One for eth1
${TC} class add dev ${INT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}
# One for eth0
${TC} class add dev ${EXT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}
# Create child classes
# These are for our internal interface eth1
# Create a class labelled “1:2″ and give it the limit defined above
${TC} class add dev ${INT_IFACE} parent 1:1 classid 1:2 htb rate ${INT_CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}
# Create a class labelled “1:3″ and give it the limit defined above
${TC} class add dev ${INT_IFACE} parent 1:1 classid 1:3 htb rate ${INT_CLS2_RATE}${UNITS} ceil ${INT_CLS2_RATE}${UNITS}
# EXT_IF (upload) now. We also set which classes can borrow and lend.
# This class is guaranteed 200kbit and can burst up to 5000kbit if available
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:2 htb rate ${CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}
# This class is guaranteed 300kbit and can burst up to 5000kbit-200kbit=4800kbit if available
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:3 htb rate ${CLS2_RATE}${UNITS} ceil `echo ${LIMIT}-${CLS1_RATE}|bc`${UNITS}
# This class can is guaranteed 4500kbit and can’t burst past it (5000kbit-200kbit-300kbit=4500kbit).
# I.e. even if our bulk traffic goes crazy, the two classes above are still guaranteed availability.
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:4 htb rate ${CLS3_RATE}${UNITS} ceil `echo ${LIMIT}-${CLS1_RATE}-${CLS2_RATE}|bc`${UNITS}
# Add pfifo. Read more about pfifo elsewhere, it’s outside the scope of this howto.
${TC} qdisc add dev ${INT_IFACE} parent 1:2 handle 12: pfifo limit 10
${TC} qdisc add dev ${INT_IFACE} parent 1:3 handle 13: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:2 handle 12: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:3 handle 13: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:4 handle 14: pfifo limit 10
### Done adding all the classes, now set up some rules! ###
# INT_IFACE
# Note the ‘dst’ direction. Traffic that goes OUT of our internal interface and to our servers is out server’s download speed, so SOME_IMPORTANT_IP is allocated to 1:2 class for download.
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst SOME_IMPORTANT_IP/32 flowid 1:2
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst SOME_OTHER_IMPORTANT_IP/32 flowid 1:2
#All other servers download speed goes to 1:3 – not as important as the above two
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst 0.0.0.0/0 flowid 1:3
# EXT_IFACE
# Prioritize DNS requests
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_IP/32 match ip sport 53 0xffff flowid 1:2
# SSH is important
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_IP/32 match ip sport 22 0xffff flowid 1:2
# Our exim SMTP server is important too
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_MAILSERVER/32 match ip sport 25 0xffff flowid 1:3
# The bulk
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src 0.0.0.0/0 flowid 1:4

abhelp · 08-03-2010, 08:42 PM

okay thanks a lot for your help. Anyway now I can clear my requirements what I want is to do some VOIP testing with my set up. We want to limit the bandwidth going over eth0 which will be the outgoing interface. The eth1 interface will be used by the VOIP client to receive the calls. So essentially what I have set up now is
Linux PC with NAT and enabled port forwarding having two network interfaces. eth0 is the outbound interface to which we want to limit traffic which will make outgoing calls. eth1 is the incoming interface which has the VOIP client connected to receive calls.
So what I did is
tc qdisc add dev $IF root handle 1: htb default 30
tc class add dev $IF parent 1: classid 1:1 htb rate $MAX_DNLD
tc class add dev $IF parent 1:1 classid 1:5 htb rate $RATE_CTRL ceil $MAX_DNLD prio 1
tc filter add dev $IF parent 1:0 prio 1 protocol ip handle 5 fw flowid 1:5
/sbin/iptables -A OUTPUT -t mangle -p tcp --sport 80 -j MARK --set-mark 5

So I have created once child class. I am not clear as to if traffic shaping is required on both the interfaces. So I have two PCs now one connected to eth0 interface and one connected to eth1 interface. The PC connected to eth0 interface will make calls to the PC connected to eth1 interface.The PC which is making the calls connected to eth0 interface needs to have bandwidth shaping rule applied. eth0 interface on the Linux PC has DHCP enabled and the PC is connected to it via a network switch. So do I need to set up leaf classes for outgoing interface and one for the eth1 interface? Do I need to do anything for the other interface as well. Can you please explain?

Thanks a lot.