How do I detect where network problems are occurring?
 

I'm frequently having problems with slow network speeds. I was wondering if someone has a definitive system/procedure or program for detecting where and what is happening?

I know if I do a traceroute I can find the latency between me and every hop. But how do I detect if something else is the problem or where? Especially for systems I don't have admin access too? Is there a way to detect throttling or bit-shaping and to see which hop is doing this?

There is a few reasons I ask this:

1. I have 2 - 5MBit/s DSL lines. If I download from my server (which is supposed to be 20Mbit/s) on the web using two connections I should get a combined speed of ~10MBit/s. I usually only get a top speed of 5Mbit/s. I want to figure out which piece of equipment is causing the problem.

2. My server out on the web keeps dropping to speeds of 40kB/s. I know they are running it on a Virtual machine. So I want to see if it's a problem in my area or theirs.

3. I know Bell throttles bit-torrent connections after 4:00 pm for all their third party vendors. Sometimes I see similar slow downs with http and ssh connections. I want to try to track down where it's being held up to see if it's the throttling or bit-shaping causing a problem.

I don't think there is a product that will do what you want. You would have to have a point of presence on the routers themselves. For number one, I don't think it works that way. If you have 2 5MB lines then it's going to pick one, not combine them. I've seem software that combines lines for greater throughput, but even then it's more of traffic cop kind of approach. Send traffic down different lines so it doesn't get congested. For number 2, are you ever getting better than 40KB? I can't see how it would be your problem. If number 3 I read in that you imply the hosting company is restricting bandwidth based on how much you pay them probably. That sounds like your problem. They are over subscribed and you don't pay them enough. Find a new one.

Whilst I'm not sure these will tell you exactly what's causing a network problem they might help you to localize it. I use Nagios (http://www.nagios.org/) to monitor hardware/routers/virtual servers and MRTG (http://oss.oetiker.ch/mrtg/) to monitor individual ports on my router, DSL modem and the network traffic on them.

I found both fairly easy to setup on ubuntu and they have certainly helped me to see traffic spikes and when I've got low or no bandwidth (eg when my DSL line gets lost or reset I can get an alert).

-Alex

if you are not using ipv6 disable it,sometimes it is the cause.

Great suggestions thanks. I use nagios for doing various monitoring, and mrtg only works if you have access to snmp (and it reports correct figures) But I'll have another look.

As for my dual wan router. The way it works is it sends traffic by choosing whichever line has the least amount of traffic on connection. However, the test case I used, I set up the download to pull through both IP addresses thus using both lines. I only got 5MBit/s. If I then started surfing the web or downloading from somewhere else I got 7-9MBit/s total reported from my router. IF I setup an iperf on one line (one IP) I get 5MBit/s, if I test the other line I get 5MBit/s. If I test both at the same time I get ~2.5MBit/s from each. However, if I test using ipref from my machine and somewhere else I can get higher than the 5MBit/s. (Just don't have two places with a full 5Mbit/s upload rate)

Am I getting better then 40KB/s. When I tested this last I was getting 20,000 KBit/s (2.4 MBytes/s) up and down. I just tested it now and I'm getting 5,000 KBit/s (625 KBytes/s) down and 1,000 KBit/s up. Just waited a few minutes and now I'm getting 500 KBit/s down and 602 KBit/s up.

As for my ISP. I use a third party company called acanac.net. I pay for unlimited, untrottled internet. However, the company acanac.net rents their lines from (Bell.ca) who as decided they don't want anyone to offer any service better then they do, so they do all sorts of nasty stuff to the lines. (Big legal fight going on about that) So I'm left with the problem of detecting when a slowdown is something I've miss-configured or their bit-shaping stuff bleeding into areas it shouldn't (as it often does)

What I'm looking for is not so much a single solution to all my problems but if anyone has a good system or procedure they follow to detect problems. An instruction page I could follow.

Ie: for Oracle if I want to trace a problem the first thing we do is run a SQL Trace. Then from there a start analyzing each statement with a problem to check for indexes and such.

iperf is another tool you can use to look at network performance, you'll need a box on the outside though. It'll let you see if you are dropping packets.