how do I build a bridged network rate limiter

wastingtime · 06-24-2015, 07:37 PM

I want to test some equipment under limited bandwidth conditions

I set up a machine with two nics in a bridge configuration such that the equipment under test i connected to eth3 and the network connected to eth1. All traffic from/to eth3 is bridged to the network via eth1, and can be monitored using e.g. wireshark listening to the bridge br0.

I now want to limit the traffic going through eth3. I have read many a tutorial regarding using tc, with specific examples for my configuration.

Here's my updated script.

Code:

#!/bin/bash

UPDEV=eth1
DNDEV=eth3
BRDEV=br0
BRIP=10.0.0.2
BRGW=10.0.0.1
BRBC=10.0.0.255
RATE=1000kbit
MAXRATE=30mbit

echo $UPDEV is the sniffer connection to the network
echo $DNDEV is connected to the client machine to be monitored

echo Load class based queuing module
modprobe sch_cbq

echo Enable forwarding
echo "1" > /proc/sys/net/ipv4/ip_forward
echo Enable redirection of packets with foreign destination ip to local interface
echo "1" > /proc/sys/net/ipv4/conf/eth1/route_localnet 

echo Reset previous setting
###########################

tc qdisc del dev $UPDEV root
tc qdisc del dev $DNDEV root

iptables -D FORWARD -t mangle -m physdev --physdev-in $DNDEV -j MARK --set-mark 5
iptables -D FORWARD -t mangle -m physdev --physdev-out $DNDEV -j MARK --set-mark 5

iptables -D INPUT -i $BRDEV  -j ACCEPT
iptables -D INPUT -i $DNDEV -j ACCEPT
iptables -D INPUT -i $UPDEV -j ACCEPT

route del default gateway $BRGW $BRDEV
ifconfig $BRDEV down

brctl delif $BRDEV $DNDEV
brctl delif $BRDEV $UPDEV
brctl delbr $BRDEV

ifconfig $DNDEV down
ifconfig $UPDEV down

if [ "$1" = "clear" ] ; then 
ifconfig $UPDEV $BRIP netmask 255.255.255.0 broadcast $BRBC up
route add default gateway $BRGW $UPDEV
exit 0
fi

echo Setup bridge
#################

echo Get rid of interface IP addresses
ifconfig $UPDEV 0 0.0.0.0 promisc up
ifconfig $DNDEV 0 0.0.0.0 promisc up

echo Create the bridge
brctl addbr $BRDEV # create bridge device
brctl addif $BRDEV $UPDEV # Add $UPDEV to bridge
brctl addif $BRDEV $DNDEV # Add $DNDEV to bridge

echo Enable bridge with upstream network ip address
ifconfig $BRDEV $BRIP netmask 255.255.255.0 broadcast $BRBC up
route add default gateway $BRGW $BRDEV

echo Ensure firewall allows traffic across bridge
iptables -A INPUT -i $UPDEV -j ACCEPT
iptables -A INPUT -i $DNDEV -j ACCEPT
iptables -A INPUT -i $BRDEV -j ACCEPT

echo Mark $DNDEV packets
iptables -A FORWARD -t mangle -m physdev --physdev-in $DNDEV -j MARK --set-mark 5
iptables -A FORWARD -t mangle -m physdev --physdev-out $DNDEV -j MARK --set-mark 5

echo Setup upstream queues with rate limit
tc qdisc add dev $UPDEV handle 1:0 root htb default 2
tc class add dev $UPDEV parent 1:0 classid 1:1 htb rate ${RATE} ceil ${RATE}
tc class add dev $UPDEV parent 1:0 classid 1:2 htb rate ${MAXRATE} ceil ${MAXRATE}

echo "Limit egress rate; drop everything coming too fast"
tc filter add dev $UPDEV parent 1:0 protocol all prio 50 handle 5 \
 estimator 1sec 8sec fw classid 1:1 police rate ${RATE} burst ${RATE} drop

echo Setup downstream queues with rate limit
tc qdisc add dev $DNDEV handle 2:0 root htb default 2
tc class add dev $DNDEV parent 2:0 classid 2:1 htb rate ${RATE} ceil ${RATE}
tc class add dev $DNDEV parent 2:0 classid 2:2 htb rate ${MAXRATE} ceil ${MAXRATE}

echo "Limit ingress rate; drop everything coming too fast"
tc filter add dev $DNDEV parent 2:0 protocol all prio 50 handle 5 \
 estimator 1sec 8sec fw classid 2:1 police rate ${RATE} burst ${RATE} drop
#end

smallpond · 06-24-2015, 10:33 PM

Before you can do class-based queuing you need to load the sch_cbq kernel module.

wastingtime · 06-24-2015, 11:35 PM

Thanks! now upstream responds to RATE setting.

Downstream continues to be unaffected.

wastingtime · 06-25-2015, 12:54 AM

I was missing an estimator

Quote:

If you use this method you must provide an estimator, although
nothing will complain if you don't. It just won't police any
packets.

http://linux-tc-notes.sourceforge.net/tc/doc/police.txt

It also turns out that there's no need for special ingress filter.
It is clearer and simpler to define class based policies on
UPDEV and DNDEV that look for the marked packets.

So the updated configuration uses fw matching to place the packets into queues.

I test using scp to copy a file downstream or upstream.
When run separately the downstream is slightly above RATE (1mbit/s).
The upstream start fast and stabilizes slightly below rate.

However, when both run concurrently the upstream copy slows down to 10% of RATE,
while the downstream works as well as before.

I'm not sure what's causing that. Could it be that the upstream tcp ack's (which
travel in opposite direction downstream) are crowded by the downstream copy.

If so how can I control that?