Welcome to the most active Linux Forum on the web.
Go Back > Forums > Linux Forums > Linux - Software
User Name
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.


  Search this Thread
Old 09-24-2009, 09:32 AM   #1
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Rep: Reputation: 16
Question Tool for finding ftp bottleneck required

Dear all

As part of my job, I frequently have to download large amounts of data from servers sat behind a firewall in a different country, to my computer also sat behind a firewall.

We never get more than 250KB/s and frequently only 40KB/s which is really annoying when the files are often 5+GB!

I would really like to try and find out whether it's our system or the other (or neither) which is responsponsible for the bottleneck as the IT departments always blame it on each other. It would be great to be armed with some output like a traceroute (which wont work due to the firewalls) to be able to prove where the problem lies.

If you can suggest a good tool I can run on Ubuntu, Fedora (or even Windoze) which will accept credentials for the external site I would really appreciate it. Some tips on interpreting the output would also be great!

I sometimes do the ftp by command line, but often actually use Filezilla as the sites are flaky and it seems to manage the connection well.

I've searched for info but nothing seems equipped for the required username and password!

Cheers in advance

Old 09-30-2009, 11:31 PM   #2
Caballero del norte
Registered: Apr 2007
Location: Sonoran Desert, USA
Distribution: Ubuntu 8.04
Posts: 30

Rep: Reputation: 16
I suggest you look into running wget.

wget can use password and username, so that is covered for you most likely.

If you can get wget running, run it with the option -nc (non-clobber) and run another instance of it at the same time. If the first instance does not appreciably slow down and the second instance runs at roughly the same speed as the first... the problem is not on the uploading side.

So, try running a third instance: if things are essentially the same as running two instancia you simply need to be a bit more multi-channeled on your side.

Sometimes when I need to do a speedy download of a lot of data (from several files obviously, because -nc limits instances to only one per file) I will run several instances of wget at one time in different terminal windows:

$ wget -nc {insert other options here} -i getme_1
$ wget -nc {insert other options here} -i getme_2
$ wget -nc {insert other options here} -i getme_3

where the getme_* files are text files with the URLs that I wish to download. (They can all be one file, too, but you'll waste some time -nc'ing files that have already been downloaded.)

(There are of course other ways than a getme file to let wget know which files to download, but this certainly works.)

I hope this sheds a little light into your room.
Old 10-01-2009, 04:20 AM   #3
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Hola Caballero!

Thank you for the suggestion, it is a useful idea, unfortunately wget wont give me the output I need to make the IT departments pay attention. I don't think I've ever noticed the bit rate affected when downloading from two different sites but I can run a test to double check though.

What I really need is something which tells me the where the delays are, are the in the other site, on the internet portion of the transfer or in house? It would need some sort of textual output I can wave under their noses!!

Old 10-01-2009, 09:08 PM   #4
LQ Guru
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 18,120

Rep: Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669Reputation: 2669
What exactly is the problem with traceroute?
Do you mean you can't traceroute from your internal box to the internal box in the other ctry because traceroute is blocked by the firewall(s)?
If so you should be able to traceroute from your system to the firewall.
You'd then have to ask you mgr to let you have an acct on the firewall so you can traceroute the international section or persuade a network guy to do that bit for you.
International cables tend to have high reliability, but the throughput can vary tremendously, eg by time of day.
Old 10-02-2009, 08:24 AM   #5
Registered: Jul 2003
Posts: 244

Rep: Reputation: 26
It might be worthwhile to just watch what is happening on your system while the download is running - are you sure you're not getting network errors? saturating one of your CPU cores with interrupts? hitting high queue wait times in your disk queue? these and other questions can be easily answered with Odds are the problem is in your network or remote system, but it can't hurt to rule out your side of the connection.
Old 10-05-2009, 04:48 AM   #6
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Hi Mark

That tools looks really useful for all sorts of applications! I have another machine that always runs slow after a particular piece of software has been used so I'll probably be installing this all over the place!

It'll be good as well to see if there is a phyical problem slowing things down. Thank you very much!


Last edited by emmalg; 10-05-2009 at 04:49 AM. Reason: spelling


ftp, networking

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
DB Tool required callsreenu Linux - Software 0 09-11-2008 08:56 AM
E-Mail Server Load - Finding Bottleneck fukawi2 Linux - Server 2 02-28-2007 06:09 PM
Finding the bottleneck in my system? sneakyimp Linux - Software 10 09-27-2005 01:04 PM
Profiling tool required Sambatyon Programming 1 07-30-2005 05:17 PM
trouble finding libraries required for apache 2.0.50 manicajk Linux - Software 4 08-16-2004 06:05 PM > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 12:51 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration