strange routing issue

keyzersuze · 07-11-2015, 06:28 AM

Hi

I have a rather interesting routing issue, I was getting slow internet
access. my first impression is that my router (linux soekris box) was
dying its about 6-8 years old.

But decided to do some investigating.

My test setup

BoxA (192.168.11.5) it has 2 1G ports bonded together

router - only has 100M ports from memory
eth0 192.168.11.1
eth1 192.168.10.1
eth3 ... eth link to adsl modem

ap 192.168.10.2 1G port.

on BoxA i run
mtr -c 10 -n 192.168.10.2

and
tcpdump -pni bond0 icmp and host 192.168.10.2

on router I run
tcpdump -pni eth0 icmp and host 192.168.10.2

Code:

20:09:32.861115 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 59520, length 44
20:09:32.961351 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 59776, length 44
20:09:32.962078 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 59776, length 44
20:09:33.061597 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 60032, length 44
20:09:33.062117 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 60032, length 44
20:09:33.395425 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 60288, length 44
20:09:33.729170 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 60544, length 44
20:09:33.729694 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 60544, length 44
20:09:34.229846 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 60800, length 44
20:09:34.730498 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 61056, length 44
20:09:34.731007 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 61056, length 44
20:09:35.231176 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 61312, length 44
20:09:35.731808 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 61568, length 44
20:09:35.732482 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 61568, length 44
20:09:36.232509 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 61824, length 44
20:09:36.733151 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 62080, length 44
20:09:36.733678 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 62080, length 44
20:09:37.233825 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 62336, length 44
20:09:37.734425 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 62592, length 44
20:09:37.735037 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 62592, length 44
20:09:38.235106 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 62848, length 44
20:09:38.735778 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 63104, length 44
20:09:38.736368 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 63104, length 44
20:09:39.236444 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 63360, length 44
20:09:39.737061 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 63616, length 44
20:09:39.737592 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 63616, length 44
20:09:40.237726 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 63872, length 44
20:09:40.738406 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 64128, length 44
20:09:40.738970 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 64128, length 44
20:09:41.239096 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 64384, length 44
20:09:41.739716 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 64640, length 44
20:09:41.740414 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 64640, length 44

This matches up exactly what I see sent on from BoxA, slightly different times

tcpdump -pni eth1 icmp and host 192.168.10.2

Code:

20:09:32.961663 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 59776, length 44
20:09:32.961934 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 59776, length 44
20:09:33.061725 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 60032, length 44
20:09:33.061979 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 60032, length 44
20:09:33.729323 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 60544, length 44
20:09:33.729564 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 60544, length 44
20:09:34.730636 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 61056, length 44
20:09:34.730874 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 61056, length 44
20:09:35.732101 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 61568, length 44
20:09:35.732343 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 61568, length 44
20:09:36.733309 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 62080, length 44
20:09:36.733546 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 62080, length 44
20:09:37.734633 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 62592, length 44
20:09:37.734899 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 62592, length 44
20:09:38.735982 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 63104, length 44
20:09:38.736227 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 63104, length 44
20:09:39.737218 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 63616, length 44
20:09:39.737452 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 63616, length 44
20:09:40.738565 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 64128, length 44
20:09:40.738824 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 64128, length 44
20:09:41.740045 IP 192.168.11.5 > 192.168.10.2: ICMP echo request, id
47832, seq 64640, length 44
20:09:41.740279 IP 192.168.10.2 > 192.168.11.5: ICMP echo reply, id
47832, seq 64640, length 44

This is what i see going out eth1. this is fairly consistent.

I am not seeing any errors on the interfaces via ifconfig.

I am not seeing high cpu usage. in fact when i turn the intervale down
to 0.1 sec between pings its the same error.

I double checked my iptables rules and eth0 <-> eth1 is just allowed
I disabled my TC rules, but they were meant for outbound to internet.

check ip li every thing seems fine, queue length of 1000

I'm at a loss now as where to look next... I am planning on replacing
the box so I get 1G ports but my curiosity has been raised now...

Ser Olmy · 07-11-2015, 10:49 AM

If your router has 100Base-TX ports, it may represent a bottleneck for traffic between the 192.168.10.0/24 and 192.168.11.0/24 networks, if the devices on those networks are capable of greater speeds. However, unless the bandwidth of your Internet connection is quite high (say, 40+ Mbps), even a simple router should be able to handle the traffic.

Some follow-up questions:

How fast is your Internet connection, exactly?
What are the exact specs of your router (CPU, memory)?
How does this "slowness" manifest itself? Are download speeds below what you'd expect? Does it take long to open web pages?
Do you experience the problem on all devices on your network?
Is the problem always present?
Are you using applications that open a very large number of connections simultaneously, such as a BitTorrent client? (This may cause some routers to run out of space in the NAT/connection tracking table.)
Does name resolution (DNS) work properly? (There should be no significant delay between issuing a host or nslookup command and seeing the results.)

keyzersuze · 07-12-2015, 12:23 AM

Quote:

Originally Posted by Ser Olmy

If your router has 100Base-TX ports, it may represent a bottleneck for traffic between the 192.168.10.0/24 and 192.168.11.0/24 networks, if the devices on those networks are capable of greater speeds. However, unless the bandwidth of your Internet connection is quite high (say, 40+ Mbps), even a simple router should be able to handle the traffic.

Some follow-up questions:

How fast is your Internet connection, exactly?
What are the exact specs of your router (CPU, memory)?
How does this "slowness" manifest itself? Are download speeds below what you'd expect? Does it take long to open web pages?
Do you experience the problem on all devices on your network?
Is the problem always present?
Are you using applications that open a very large number of connections simultaneously, such as a BitTorrent client? (This may cause some routers to run out of space in the NAT/connection tracking table.)
Does name resolution (DNS) work properly? (There should be no significant delay between issuing a host or nslookup command and seeing the results.)

eth0 and eth1 are not internet eth3 is.

eth0 is local lan
eth1 is local wifi (via same switch but different vlan)

Yes I thought it might be the 1G -> 100M thing, but there is hardly any traffic on there when i test and you can see its just mtr sending icmp's !
This has been a new ish problem, last 4-6 months. symptoms was slow internet access. testing got me to here..

The above testing was done with very little traffic

if you look at the tcpdumps the packet makes it in eth0 but doesn't make it out of eth1. No firewall rules and I removed all the traffic shaping rules, even double checked the other inferface to see if the packet was being sent there but no ....

EDIT: think i have had a bit of brain fart... I'm going to miss some packets on eth1, its mtr ... traceroute...