Tool for finding ftp bottleneck required

emmalg · 09-24-2009, 09:32 AM

Dear all

As part of my job, I frequently have to download large amounts of data from servers sat behind a firewall in a different country, to my computer also sat behind a firewall.

We never get more than 250KB/s and frequently only 40KB/s which is really annoying when the files are often 5+GB!

I would really like to try and find out whether it's our system or the other (or neither) which is responsponsible for the bottleneck as the IT departments always blame it on each other. It would be great to be armed with some output like a traceroute (which wont work due to the firewalls) to be able to prove where the problem lies.

If you can suggest a good tool I can run on Ubuntu, Fedora (or even Windoze) which will accept credentials for the external site I would really appreciate it. Some tips on interpreting the output would also be great!

I sometimes do the ftp by command line, but often actually use Filezilla as the sites are flaky and it seems to manage the connection well.

I've searched for info but nothing seems equipped for the required username and password!

Cheers in advance
Emma

Caballero del norte · 09-30-2009, 11:31 PM

I suggest you look into running wget.

wget can use password and username, so that is covered for you most likely.

If you can get wget running, run it with the option -nc (non-clobber) and run another instance of it at the same time. If the first instance does not appreciably slow down and the second instance runs at roughly the same speed as the first... the problem is not on the uploading side.

So, try running a third instance: if things are essentially the same as running two instancia

you simply need to be a bit more multi-channeled on your side.

Sometimes when I need to do a speedy download of a lot of data (from several files obviously, because -nc limits instances to only one per file) I will run several instances of wget at one time in different terminal windows:

$ wget -nc {insert other options here} -i getme_1
$ wget -nc {insert other options here} -i getme_2
$ wget -nc {insert other options here} -i getme_3
...

where the getme_* files are text files with the URLs that I wish to download. (They can all be one file, too, but you'll waste some time -nc'ing files that have already been downloaded.)

(There are of course other ways than a getme file to let wget know which files to download, but this certainly works.)

I hope this sheds a little light into your room.

emmalg · 10-01-2009, 04:20 AM

Hola Caballero!

Thank you for the suggestion, it is a useful idea, unfortunately wget wont give me the output I need to make the IT departments pay attention. I don't think I've ever noticed the bit rate affected when downloading from two different sites but I can run a test to double check though.

What I really need is something which tells me the where the delays are, are the in the other site, on the internet portion of the transfer or in house? It would need some sort of textual output I can wave under their noses!!

Cheers
Emma

chrism01 · 10-01-2009, 09:08 PM

What exactly is the problem with traceroute?
Do you mean you can't traceroute from your internal box to the internal box in the other ctry because traceroute is blocked by the firewall(s)?
If so you should be able to traceroute from your system to the firewall.
You'd then have to ask you mgr to let you have an acct on the firewall so you can traceroute the international section or persuade a network guy to do that bit for you.
International cables tend to have high reliability, but the throughput can vary tremendously, eg by time of day.

markseger · 10-02-2009, 08:24 AM

It might be worthwhile to just watch what is happening on your system while the download is running - are you sure you're not getting network errors? saturating one of your CPU cores with interrupts? hitting high queue wait times in your disk queue? these and other questions can be easily answered with http://collectl.sourceforge.net. Odds are the problem is in your network or remote system, but it can't hurt to rule out your side of the connection.
-mark

emmalg · 10-05-2009, 04:48 AM

Hi Mark

That tools looks really useful for all sorts of applications! I have another machine that always runs slow after a particular piece of software has been used so I'll probably be installing this all over the place!

It'll be good as well to see if there is a phyical problem slowing things down. Thank you very much!

Emma