Traffic shaping with tc and htb

gosssamer · 09-01-2014, 11:46 AM

Hi,

I have a fedora20 mail server connected to a 10mbs ethernet link on a 1Gbs DMZ. This server has about 1000 users. Periodically, a corporate email is sent to a few hundred users at a time, many of which then forward their mail off to a remote system, such as to their gmail account.

The problem is the 10mbs link is being saturated with all this mail being forwarded off the server to remote mailboxes. I'm trying to shape that outbound mail to prevent it from consuming all the available bandwith.

At the same time, I don't want to throttle access to the system from the hosts on the local network, including the backup server, for example.

There are other hosts on the network that need to be able to use this 10mbs link as well, so I've allocated 2mbit for outbound SMTP traffic from this host, during peak times.

I've put together the following tc rules:

Code:

tc qdisc add dev eth0 root handle 1: htb default 20
tc class add dev eth0 parent 1: classid 1:1 htb rate 1000mbit burst 15k
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 512kbit ceil 2mbit burst 15k
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 1mbit ceil 1000mbit burst 15k
tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10 
tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10 
U32="tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32"
$U32 match ip dport 25 0xffff flowid 1:10

I don't believe it's correct, however, because it's still suffering from high latency when there are large messages being sent from the queue.

Any ideas greatly appreciated.
Thanks,
Alex

unSpawn · 09-01-2014, 06:48 PM

Quote:

Originally Posted by gosssamer

The problem is the 10mbs link is being saturated with all this mail being forwarded off the server to remote mailboxes. I'm trying to shape that outbound mail to prevent it from consuming all the available bandwith. (..) Any ideas greatly appreciated.

Before you get into bandwidth shaping have a look at what your MTA can do for you. Any decent MTA should be able to throttle sending mail itself. Sendmail for example has a concept of "cost" of sending mail and it can also work with different queues that exhibit different characteristics (slow / fast). Then you could also use an iptables rule limiting the amount of outbound new connections to remote TCP/25. Also realize that bandwidth shaping this way does not apply to remote systems (as you exert no control over those) directly but by way of throttling your systems responses.

gosssamer · 09-01-2014, 08:01 PM

Quote:

Originally Posted by unSpawn

Before you get into bandwidth shaping have a look at what your MTA can do for you. Any decent MTA should be able to throttle sending mail itself. Sendmail for example has a concept of "cost" of sending mail and it can also work with different queues that exhibit different characteristics (slow / fast).

Do you happen to know what mechanisms postfix has available, or can point me to the related docs?

Quote:

Then you could also use an iptables rule limiting the amount of outbound new connections to remote TCP/25.

I've seen several references to using tc with iptables. Is this what you're referring to?

Why would someone choose iptables over tc for rate limiting?

Quote:

Also realize that bandwidth shaping this way does not apply to remote systems (as you exert no control over those) directly but by way of throttling your systems responses.

Yes, you cannot control inbound rates, but it should be possible to control outbound smtp connections to remote machines using some combination of tc and/or iptables.

Thanks,
Alex

unSpawn · 09-02-2014, 12:14 PM

Quote:

Originally Posted by gosssamer

Do you happen to know what mechanisms postfix has available, or can point me to the related docs?

No I don't but a quick search for something like "postfix throttle outbound" should yield a cornucopia of leads.

Quote:

Originally Posted by gosssamer

I've seen several references to using tc with iptables. Is this what you're referring to?

No, more something like "-A OUTPUT -p tcp -m state --state NEW --dport 25 -m limit --limit 2/s".

Quote:

Originally Posted by gosssamer

Why would someone choose iptables over tc for rate limiting?

No, I said before you get into tc...

Quote:

Originally Posted by gosssamer

Yes, you cannot control inbound rates, but it should be possible to control outbound smtp connections to remote machines using some combination of tc and/or iptables.

I clarified this as some people seem to think tc is a complete solution, that's all.

gosssamer · 09-04-2014, 12:08 AM

Quote:

Originally Posted by unSpawn

No I don't but a quick search for something like "postfix throttle outbound" should yield a cornucopia of leads.

That doesn't work. You can't control the concurrency for just outbound mail. When I did it, users were complaining about a lack of SMTP connections and their mail being queued and/or temporarily rejected.

Quote:

No, more something like "-A OUTPUT -p tcp -m state --state NEW --dport 25 -m limit --limit 2/s".

I'd really like to know if that's a competing solution to using tc, or why you would use it conjunction with tc when you can specify which protocols are sent to which filters using tc alone.

Do you have any idea where I might go for specific tc assistance?

For example, I'm learning now one of my filters have a negative amount of tokens, and I have no idea what that means:

Code:

class htb 1:30 parent 1:1 prio 0 quantum 1000 rate 1Kbit ceil 8Mbit linklayer ethernet burst 15Kb/1 mpu 0b overhead 0b cb
urst 1600b/1 mpu 0b overhead 0b level 0
 Sent 80813962 bytes 111342 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
 lended: 3070 borrowed: 87144 giants: 0
 tokens: -4738044 ctokens: 18906

Could that be somehow related to borrowed tokens?

Thanks,
Alex

unSpawn · 09-04-2014, 02:46 AM

Quote:

Originally Posted by gosssamer

That doesn't work. You can't control the concurrency for just outbound mail. When I did it, users were complaining about a lack of SMTP connections and their mail being queued and/or temporarily rejected.

I contest the "doesn't work" part simply by the fact of you not having posted what you re-configured, the amount of connections you would need on average and any statistics of your trials. Plus like I said before some MTAs know how to handle different queues so you could filter on message submission for certain characteristics and decide to spool them for later sending and send for instance high priority messages immediately. In short this isn't the simple tweaking of just a few dials but a strategy to devise.

Quote:

Originally Posted by gosssamer

I'd really like to know if that's a competing solution to using tc, or why you would use it conjunction with tc when you can specify which protocols are sent to which filters using tc alone.

Until you actually do that you don't. The "protocol ip" in the tc line given before only means "ip" as in "Internet Protocol Suite protocols". No port designations given. The way I see it an MTA will respond to network conditions, for example when it receives certain errors it will, per mail RFCs, just re-queue the message for later resending. The quicker it can re-queue those messages (as opposed to a lengthy trickle traffic state due to how existing bandwidth must be divided continuously between more and more connections) the better, more efficient. At least that's how things seem to me.

Quote:

Originally Posted by gosssamer

Do you have any idea where I might go for specific tc assistance?

For anything wrt bandwidth shaping first see the LARTC and the iproute mailing list.

gosssamer · 09-16-2014, 10:01 PM

I figured this out, and thought I would follow-up with a working solution. First, an outline of the problem:

- fedora20 postfix/dovecot/http server on a 1Gbit DMZ/LAN connected to the Internet on a 10mbs link
- Need to throttle bulk outbound SMTP connections to undetermined external mail servers to prevent saturation of 10mbs Internet link
- Need to avoid restricting connections by other hosts in the DMZ to prevent ssh backups from being throttled
- Postfix does not have the ability to throttle at a packet or message level, only to specific, identified hosts

You should have a basic understanding of how tc works. I've chosen HTB because it's simpler than CBQ and the fine-grained level of control wasn't really necessary here, especially with the addition of SFQ on top of HTB.

I'd certainly welcome any tips for improvement.

Code:

#!/bin/sh

BACKUP=4.3.2.1

# This line sets a HTB qdisc on the root of eth0, and it specifies that the class 1:30 is 
# used by default. It sets the name of the root as 1:, for future references.
tc qdisc add dev eth0 root handle 1: htb default 50

# This creates a class called 1:1, which is direct descendant of root (the parent is 1:), 
# this class gets assigned also an HTB qdisc, and then it sets a max rate of 1000mbits, with a burst of 15k
tc class add dev eth0 parent 1: classid 1:1 htb rate 1000mbit burst 15k

# Internet traffic limited to about 8mbs, to allow for any overhead and leaving some bandwidth
# for other hosts on the LAN to connect to the Internet
# The previous class has this branches:

# smtp traffic (limited to 4mbs)
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 2mbit ceil 4mbit burst 15k prio 1

# ssh backups on local LAN (no effective limit)
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 2mbit ceil 1000mbit burst 15k prio 1

# http traffic (limited to 4mbs)
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 4mbit burst 15k prio 1

# imap/pop traffic (limited to 4mbs)
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 3mbit ceil 4mbit burst 15k prio 1

# ssh/interactive traffic (limited to 1mbs)
tc class add dev eth0 parent 1:1 classid 1:40 htb rate 512kbit ceil 1mbit burst 15k prio 1

# default traffic (everything else; limited to 2mbs)
tc class add dev eth0 parent 1:1 classid 1:50 htb rate 3mbit ceil 2mbit burst 15k prio 1

# Martin Devera, author of HTB, then recommends SFQ for beneath these classes:
tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10 
tc qdisc add dev eth0 parent 1:11 handle 11: sfq perturb 10 
tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10 
tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10 
tc qdisc add dev eth0 parent 1:40 handle 40: sfq perturb 10 
tc qdisc add dev eth0 parent 1:50 handle 50: sfq perturb 10 

#
# Filter Section
#
# These commands adds a filter to the qdisc 1: of dev eth0, set the
# priority of the filters to 1 (all get the same priority), matches packages
# by source port, and set the class to the corresponding classid.
#
# The 0xffff designates this is for all protocols

# ssh/interactive traffic
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 22 0xffff flowid 1:40

# smtp traffic
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip dport 25 0xffff flowid 1:10

# This filter is attached to the qdisc 1: of dev eth0, has a
# priority of 1, and matches the destination ip address 4.3.2.1 exactly, and
# matches packages with a source port of 22, then makes class
# 1:11 process the packages that match
#
# ssh backups from backup server
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip dst $BACKUP match ip sport 22 0xffff flowid 1:11

# http traffic
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 80 0xffff flowid 1:20
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 443 0xffff flowid 1:20

# imap/pop traffic
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 110 0xffff flowid 1:30
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 143 0xffff flowid 1:30
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 993 0xffff flowid 1:30
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 995 0xffff flowid 1:30
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 465 0xffff flowid 1:30
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 587 0xffff flowid 1:30