LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 09-24-2009, 08:32 AM   #1
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Rep: Reputation: 16
Question Tool for finding ftp bottleneck required


Dear all

As part of my job, I frequently have to download large amounts of data from servers sat behind a firewall in a different country, to my computer also sat behind a firewall.

We never get more than 250KB/s and frequently only 40KB/s which is really annoying when the files are often 5+GB!

I would really like to try and find out whether it's our system or the other (or neither) which is responsponsible for the bottleneck as the IT departments always blame it on each other. It would be great to be armed with some output like a traceroute (which wont work due to the firewalls) to be able to prove where the problem lies.

If you can suggest a good tool I can run on Ubuntu, Fedora (or even Windoze) which will accept credentials for the external site I would really appreciate it. Some tips on interpreting the output would also be great!

I sometimes do the ftp by command line, but often actually use Filezilla as the sites are flaky and it seems to manage the connection well.

I've searched for info but nothing seems equipped for the required username and password!

Cheers in advance
Emma

 
Old 09-30-2009, 10:31 PM   #2
Caballero del norte
Member
 
Registered: Apr 2007
Location: Sonoran Desert, USA
Distribution: Ubuntu 8.04
Posts: 30

Rep: Reputation: 16
I suggest you look into running wget.

wget can use password and username, so that is covered for you most likely.

If you can get wget running, run it with the option -nc (non-clobber) and run another instance of it at the same time. If the first instance does not appreciably slow down and the second instance runs at roughly the same speed as the first... the problem is not on the uploading side.

So, try running a third instance: if things are essentially the same as running two instancia you simply need to be a bit more multi-channeled on your side.

Sometimes when I need to do a speedy download of a lot of data (from several files obviously, because -nc limits instances to only one per file) I will run several instances of wget at one time in different terminal windows:

$ wget -nc {insert other options here} -i getme_1
$ wget -nc {insert other options here} -i getme_2
$ wget -nc {insert other options here} -i getme_3
...

where the getme_* files are text files with the URLs that I wish to download. (They can all be one file, too, but you'll waste some time -nc'ing files that have already been downloaded.)

(There are of course other ways than a getme file to let wget know which files to download, but this certainly works.)

I hope this sheds a little light into your room.
 
Old 10-01-2009, 03:20 AM   #3
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Hola Caballero!

Thank you for the suggestion, it is a useful idea, unfortunately wget wont give me the output I need to make the IT departments pay attention. I don't think I've ever noticed the bit rate affected when downloading from two different sites but I can run a test to double check though.

What I really need is something which tells me the where the delays are, are the in the other site, on the internet portion of the transfer or in house? It would need some sort of textual output I can wave under their noses!!

Cheers
Emma
 
Old 10-01-2009, 08:08 PM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,289

Rep: Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034
What exactly is the problem with traceroute?
Do you mean you can't traceroute from your internal box to the internal box in the other ctry because traceroute is blocked by the firewall(s)?
If so you should be able to traceroute from your system to the firewall.
You'd then have to ask you mgr to let you have an acct on the firewall so you can traceroute the international section or persuade a network guy to do that bit for you.
International cables tend to have high reliability, but the throughput can vary tremendously, eg by time of day.
 
Old 10-02-2009, 07:24 AM   #5
markseger
Member
 
Registered: Jul 2003
Posts: 241

Rep: Reputation: 25
It might be worthwhile to just watch what is happening on your system while the download is running - are you sure you're not getting network errors? saturating one of your CPU cores with interrupts? hitting high queue wait times in your disk queue? these and other questions can be easily answered with http://collectl.sourceforge.net. Odds are the problem is in your network or remote system, but it can't hurt to rule out your side of the connection.
-mark
 
Old 10-05-2009, 03:48 AM   #6
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Hi Mark

That tools looks really useful for all sorts of applications! I have another machine that always runs slow after a particular piece of software has been used so I'll probably be installing this all over the place!

It'll be good as well to see if there is a phyical problem slowing things down. Thank you very much!

Emma

Last edited by emmalg; 10-05-2009 at 03:49 AM. Reason: spelling
 
  


Reply

Tags
ftp, networking


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
DB Tool required callsreenu Linux - Software 0 09-11-2008 07:56 AM
E-Mail Server Load - Finding Bottleneck fukawi2 Linux - Server 2 02-28-2007 05:09 PM
Finding the bottleneck in my system? sneakyimp Linux - Software 10 09-27-2005 12:04 PM
Profiling tool required Sambatyon Programming 1 07-30-2005 04:17 PM
trouble finding libraries required for apache 2.0.50 manicajk Linux - Software 4 08-16-2004 05:05 PM


All times are GMT -5. The time now is 07:39 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration