[SOLVED] Linux router, non-local uploads have horrible performance

Rz_Ten1 · 05-20-2014, 09:14 PM

I've been trying to pin down this issue for quite awhile now, and I'm hoping someone may have an idea as to what is going wrong.

The short version is I have a Linux router running PPPoE on a DSL modem in bridge mode. Everything works great on the router itself but the problem is anything else on the LAN using this machine as a router has crap upload performance. WAN downloads via the LAN are unimpaired and run at the maximum DSL speed. Uploads from the router itself will hit ~550KBps. Uploads from any other machine on the LAN to the WAN struggle to exceed 10KBps. Wireshark/tcpdump show a massive number of retransmits on client machines, but only when trying to send data over the WAN. Oddly not all of these packets seem to be passing though ppp0.

Transfers to and from the router over the local LAN will hit the line speed, so it's not related to the switch or eth0. On a whim I've also replaced the network cards in several machines as well as the switch to no avail. Turning off network offload on all devices has no impact as well. The MTU is set the same on all devices.

ifconfig looks like this:

Quote:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1492
inet 10.0.0.50 netmask 255.255.255.0 broadcast 10.0.0.255
ether 00:1b:21:11:81:27 txqueuelen 100 (Ethernet)
RX packets 46225 bytes 4888156 (4.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 71448 bytes 88078506 (83.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19 memory 0xfbde0000-fbe00000

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255
ether 00:50:04:0a:c1:a2 txqueuelen 100 (Ethernet)
RX packets 66929 bytes 87846559 (83.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 38632 bytes 5802810 (5.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 base 0xec00

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 0 (Local Loopback)
RX packets 7872 bytes 1286826 (1.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7872 bytes 1286826 (1.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1492
inet X.X.X.X netmask 255.255.255.255 destination Y.Y.Y.Y
ppp txqueuelen 3 (Point-to-Point Protocol)
RX packets 66226 bytes 86334475 (82.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 38231 bytes 4937221 (4.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0 is the LAN connection, eth1 is the DSL bridge, and ppp0 is the PPPoE connection.

sysctl.conf is changing the following settings from their defaults:

Quote:

# Disable inbound source routed packets to prevent folks
# from spoofing their IP address. No legitimate users
# require source routing any more.
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable TCP SYN cookies to keep us from suffering from
# syn-flood DoS or DDoS attacks. See DJB's page at
# http://cr.yp.to/syncookies.html if you want to know
# how SYN cookies work - it's cool.
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_no_metrics_save = 1

# Don't send any redirects either. (Only use if you're
# not acting as a router that needs to send redirects.)
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Do not respond to packets that would cause us to go out
# a different interface than the one to which we're responding.
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Log any packets that have IP addresses that shouldn't exist
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Packet forwarding
net.ipv4.ip_forward = 1

# Ignore broadcast pings
# (Don't participate in smurf attacks)
net.ipv4.icmp_echo_ignore_broadcasts = 1

#Speed tweaks
net.core.rmem_max = 10485760
net.core.wmem_max = 10485760
net.core.netdev_max_backlog = 2500
net.ipv4.tcp_rmem = 8192 87380 10485760
net.ipv4.tcp_wmem = 8192 65536 10485760
net.ipv4.udp_mem = 4096 65536 10485760
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_slow_start_after_idle = 0

The PPPoE startup and iptables configuration script sets the following:

Quote:

PUBLIC_IP=X.X.X.X
INTERFACE=eth0
MODEM=ppp0

echo "Setting iptables rules..."

#Set buffer lengths
/sbin/ifconfig eth0 txqueuelen 100
/sbin/ifconfig eth1 txqueuelen 100

#Reset iptables
/usr/sbin/iptables --flush
/usr/sbin/iptables -t nat --flush
/usr/sbin/iptables -t mangle --flush
#Reject packets from RFC1918 class networks
/usr/sbin/iptables -A INPUT -i $MODEM -s 10.0.0.0/8 -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -s 127.0.0.0/8 -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -s 169.254.0.0/16 -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -s 172.16.0.0/12 -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -s 192.168.0.0/16 -j DROP
# Drop invalid packets immediately
/usr/sbin/iptables -A INPUT -i $MODEM -m state --state INVALID -j NFLOG --nflog-group 0
/usr/sbin/iptables -A INPUT -i $MODEM -m state --state INVALID -j DROP
# Allow most ICMP packets to be received (so people can check our
# presence), but restrict the flow to avoid ping flood attacks
/usr/sbin/iptables -A INPUT -i $MODEM -p icmp -m icmp --icmp-type address-mask-request -j NFLOG --nflog-group 1
/usr/sbin/iptables -A INPUT -i $MODEM -p icmp -m icmp --icmp-type address-mask-request -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -p icmp -m icmp --icmp-type timestamp-request -j NFLOG --nflog-group 1
/usr/sbin/iptables -A INPUT -i $MODEM -p icmp -m icmp --icmp-type timestamp-request -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -p icmp -m icmp --icmp-type any -m limit --limit 5/second -j ACCEPT
# Drop excessive RST packets to avoid SMURF attacks.
/usr/sbin/iptables -A INPUT -i $MODEM -p tcp -m tcp --tcp-flags RST RST -m limit --limit 5/second --limit-burst 5 -j ACCEPT
#25 SMTP, 110 POP3, 143 IMAP, 993 IMAP SSL
#80 HTTP, 443 HTTPS
/usr/sbin/iptables -A INPUT -i $MODEM -p tcp -m multiport --dports 25,80,443,993 -j ACCEPT
/usr/sbin/iptables -A INPUT -i $MODEM -p udp -m multiport --dports 1,67,513 -j DROP
/usr/sbin/iptables -A INPUT -i $MODEM -m state --state NEW -j NFLOG --nflog-group 0
/usr/sbin/iptables -A INPUT -i $MODEM -m state --state NEW -j DROP
/usr/sbin/iptables -P FORWARD DROP
/usr/sbin/iptables -A FORWARD -i $MODEM -d 10.0.0.0/8 -j ACCEPT
/usr/sbin/iptables -A FORWARD -i $INTERFACE -s 10.0.0.6 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
/usr/sbin/iptables -A FORWARD -i $INTERFACE -s 10.0.0.0/8 -j ACCEPT
/usr/sbin/iptables -A FORWARD -j NFLOG --nflog-group 1

#Wireless route
route add -net 172.16.1.0 netmask 255.255.255.0 dev eth0 gw 10.0.0.8 metric 1 > /dev/null 2>&1
/usr/sbin/iptables -A PREROUTING -t nat -p udp -i $MODEM -d 172.16.1 -j DNAT --to-destination 10.0.0.8

#Routing!
/usr/sbin/iptables -A POSTROUTING -t nat -o $MODEM -j SNAT --to $PUBLIC_IP

# Traffic Control
# Reset everything to a known state (cleared)
/usr/sbin/tc qdisc del dev $MODEM root > /dev/null 2>&1
/usr/sbin/tc qdisc add dev $MODEM root pie limit 500 target 10ms

I've tried pulling the qdisc out, but that also has no impact. Does anyone know what may be the issue here?

jailbait · 05-20-2014, 10:17 PM

Check your contract with your ISP. Many ISPs provide upload speeds that are much lower than the download speeds.

--------------------
Steve Stites

Rz_Ten1 · 05-20-2014, 10:53 PM

The problem isn't with the ISP; it's on my end. It's a 20mbit/5mbit line. I can hit a solid ~18mbit down everywhere but only ~4mbit up from the router. Every other machine has the slow upload issue.

Rz_Ten1 · 05-21-2014, 03:53 AM

Also, one more oddity: Passive FTP uploads from any client machine hang as soon as the file transfer tries to start. Passive FTP uploads from the router itself work fine. The conntrack FTP helper isn't loaded, but from my understanding it's only needed for incoming connections and not outgoing passive connections.

jailbait · 05-21-2014, 11:28 AM

Quote:

Originally Posted by Rz_Ten1

It's a 20mbit/5mbit line.

Therefore your maximum upload speed is 5mbit.

-----------------------
Steve Stites

suicidaleggroll · 05-21-2014, 11:49 AM

Quote:

Originally Posted by jailbait

Therefore your maximum upload speed is 5mbit.

-----------------------
Steve Stites

And since he's getting 10kB/s from all machines except the router (which hits 550kB/s), I'd say there's a problem...wouldn't you?

Rz_Ten1 · 05-21-2014, 12:42 PM

Some news, I think I'm looking at a kernel regression. I just recompiled 3.10.40 and all the client machines are getting 18.81 mbps down and 4.91 mbps now, which is basically line speed. I suspect this is an error in conntrack, but I'm not totally sure yet.

I'm going to try moving up older kernel revisions to see if I can pin down where it broke.

Rz_Ten1 · 05-21-2014, 01:14 PM

Yep, looks like it was a change in 3.14.

3.10.40: OK
3.12.20: OK
3.13.11: OK
3.14.0: FAIL
3.14.4: FAIL

Rz_Ten1 · 05-21-2014, 02:36 PM

More kernels:
3.14.0-rc1: OK
3.14.0-rc2: OK
3.14.0-rc3: OK
3.14.0-rc4: OK
3.14.0-rc5: OK
3.14.0-rc6: OK
3.14.0-rc7: OK
3.14.0-rc8: OK
3.14.0: FAIL

Rz_Ten1 · 05-21-2014, 03:38 PM

Looks like there's a patch for this already, although it isn't included in the current stable build.

From http://patchwork.ozlabs.org/patch/339082/:
Vlad Yasevich - April 14, 2014, 9:37 p.m.

Sometimes, when the packet arrives at skb_mac_gso_segment()
its skb->mac_len already accounts for some of the mac lenght
headers in the packet. This seems to happen when forwarding
through and OpenSSL tunnel.

When we start looking for any vlan headers in skb_network_protocol()
we seem to ignore any of the already known mac headers and start
with an ETH_HLEN. This results in an incorrect offset, dropped
TSO frames and general slowness of the connection.

We can start counting from the known skb->mac_len
and return at least that much if all mac level headers
are known and accounted for.

Fixes: 53d6471cef17262d3ad1c7ce8982a234244f68ec (net: Account for all vlan headers in skb_mac_gso_segment)
CC: Eric Dumazet <eric.dumazet@gmail.com>
CC: Daniel Borkman <dborkman@redhat.com>
Tested-by: Martin Filip <nexus+kernel@smoula.net>
Signed-off-by: Vlad Yasevich <vyasevic@redhat.com>
---
net/core/dev.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)