LinuxQuestions.org - Shaping egress upload problem

Hi everybody,

This is my first message in this linux community service so please dont scream.
I have a problem with setting up my traffic shaper for users in particular vlans to limit
their upload speed. The download bandwith is limited properly and it didnt cause any problems
for me. Maybe I will start from giving you some information about my link and topology.

Policy that I want to achieve:
- each user has a fixed bandwith of 512kbit down and 64kbit up
- between vlan traffic is shaped by sfq on max LAN speed

Information:
- I use debian 5.03 64bit
- I have 3 vlans (vlan10, vlan20, vlan30) on 802.1q protocol
- I aggregate 2 network devices (eth0, eth1) using ifenslave and bonding module
- This is an LACP aggregation
- The aggregated interfaces goes to Cisco CAT. 2950 and from there is multiplexed through
particular vlans
- My total bandwith is 2048kbit/256kbit as for up/down

Network topology image:

http://img42.imageshack.us/img42/7317/nettopology.jpg

Problems:

Now as I said before everything works fine with shaping traffic that comes to my users
(download). I do it that way:

Code:

# Setting queue for vlan10 subinterface

echo "[+] Configuring htb queue for vlan10"

tc qdisc del root dev vlan10 2> /dev/null > /dev/null

tc qdisc add dev vlan10 root handle 1:0 htb default 3



tc class add dev vlan10 parent 1:0 classid 1:1 htb rate 95000kbit



tc class add dev vlan10 parent 1:1 classid 1:2 htb rate 2020kbit burst 15k

tc class add dev vlan10 parent 1:1 classid 1:3 htb rate 92000kbit ceil 92500kbit



tc class add dev vlan10 parent 1:2 classid 1:4 htb rate 512kbit ceil 512kbit



for i in {5..10}

do

echo "[+] Setting filter for address: 172.16.10.$i"

tc filter add dev vlan10 protocol ip parent 1:0 u32 match ip dst 172.16.10.$i/32 flowid 1:4

done



tc filter add dev vlan10 protocol ip parent 1:0 u32 match ip src 172.16.10.1/32 flowid 1:3



tc qdisc add dev vlan10 parent 1:4 handle 4:0 sfq perturb 10

tc qdisc add dev vlan10 parent 1:3 handle 3:0 sfq perturb 10

echo "[+] Done: vlan10"

So if you have some precautions please notice me.
The problem starts with shaping traffic that goes from my lan users and I am well aware of
the fact that I should use ingress shaping and imq issues but I want to bypass this kind
of configuration as I heard that on routers you can easily manage inbound and outbound traffic
because the condition is to shape only outgoing traffic and this type of traffic depends from
the perspective we look at. For example on my ppp0 interface the outgoing traffic is the
data that users send to the internet and so on. So I tried to do the same fing with ppp0
interface and I got things quite messy:

The first attempt was that:

Code:

# Setting queue for ppp0 modem interface <<< TESTING >>>

echo "[+] Configuring htb queue for ppp0"

tc qdisc del root dev ppp0 2> /dev/null > /dev/null

tc qdisc add dev ppp0 root handle 1:0 htb



tc class add dev ppp0 parent 1:0 classid 1:1 htb rate 230kbit burst 6k



tc class add dev ppp0 parent 1:1 classid 1:2 htb rate 64kbit ceil 64kbit



tc filter add dev ppp0 protocol ip parent 1:0 u32 match ip src 172.16.10.7 flowid 1:2



tc qdisc add dev ppp0 parent 1:2 handle 2:0 sfq perturb 10

echo "[+] Done: ppp0"

And on host 172.16.10.7 nothing happend, since I think the SNAT/MASQ do the trick and
changed private ip to public, so the speedtest showed me down: 512kbit and up: 256kbit :(

The second Attempt:

Code:

# Setting queue for ppp0 modem interface <<< TESTING >>>

echo "[+] Configuring htb queue for ppp0"

tc qdisc del root dev ppp0 2> /dev/null > /dev/null

tc qdisc add dev ppp0 root handle 1:0 htb



tc class add dev ppp0 parent 1:0 classid 1:1 htb rate 230kbit burst 6k



tc class add dev ppp0 parent 1:1 classid 1:2 htb rate 64kbit ceil 64kbit



#tc filter add dev ppp0 protocol ip parent 1:0 u32 match ip dport 80 0xffff flowid 1:2

tc filter add dev ppp0 parent 1:0 protocol ip prio 1 handle 7 fw flowid 1:2



iptables -t mangle -A PREROUTING -i vlan10 -s 172.16.10.7 -j MARK --set-mark 7



tc qdisc add dev ppp0 parent 1:2 handle 2:0 sfq perturb 10

echo "[+] Done: ppp0"

I figured out that I can mark a packet that has know source address and then use fwmark
to enqueue it even then when nat mangles it. The result was unfortunately the same. The
iptables rule was incrementing but the upload rate was the same.

I read some LARTC documentation and they say that you must to turn off reverse path filtering
to prevent complication between NAT and packet marking.

So I tried:

Code:

echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter

echo "0" > /proc/sys/net/ipv4/conf/ppp0/rp_filter

echo "0" > /proc/sys/net/ipv4/conf/vlan10,20,30/rp_filter

And it worked, the rate per user was correct approx. 512/64 in the speedtest.

I am worried about that I maked some kind of broadcast storm because the counters in
iptables in mangle table and prerouting chain is growing very fast (marking packets):

Code:

# iptables -t mangle -nvL

Chain PREROUTING (policy ACCEPT 21373 packets, 14M bytes)

 pkts bytes target    prot opt in    out    source              destination

 4470  785K MARK      all  --  vlan10 *      172.16.10.7          0.0.0.0/0          MARK xset 0x7/0xffffffff

I am actually alone in this vlan and dont create almost any traffic and the counters are
getting higher and higher. So this fact is suspicious for me.

Another problem is that I had to turn off rp filtering and this is very dangerous
for my routing table, that could be changed by some spoofed routing advertisement
altough I dont use any routing protocols like bpg,ospf.

So I dont know if I should use IMQ, I heard that it has some drawbacks and its difficult to setup.

Sorry for the big lenght of this post but I couldnt explain what I mean in a different way.
Please help me and advise me some way that is appropriate to solve this manner properly.