LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 09-19-2006, 02:02 PM   #1
toko2
LQ Newbie
 
Registered: Oct 2005
Location: Toronto, Ontario
Distribution: SuSE, Ubuntu
Posts: 2

Rep: Reputation: 0
Large file transfers over a network connection.


I have recently run into a problem at work. I manage quite a bit of data, however it is all images (jpeg, gif etc). I have one server that houses roughly 2TB of this data but I need to be able to transfer this from one server to another server. For various reasons I have to use my gigabit network connection to do this. At the present time we are using the "tar cf - stuff | ssh bob@wendy.no.where.com tar xf - -C /home/dir" method but the through put is horrible. We have also tried using the " -c blowfish" option on ssh but this doesn't really help, on top of that we have also tried changing the tar block size which also....doesn't help. Any ideas?

Last edited by toko2; 09-19-2006 at 02:18 PM.
 
Old 09-19-2006, 03:38 PM   #2
thorn168
Member
 
Registered: Oct 2004
Location: USA
Distribution: Vector Linux 5.1 Std., Vector Linux 5.8 Std., Win2k, XP, OS X (10.4 & 10.5)
Posts: 344

Rep: Reputation: 42
I have a feeling that your bottleneck maybe within the server itself rather then in the network connection. What sort of internal bus do you have: SATA or PATA? What is the max rated speed at which the bus is running? Also check to make sure that the hard drives are not excessively fragmented. File fragmentation can significantly slow down the transfer of data.

Now if the hardware internals turn out not to be the cause of the slow down then you have to start analyzing how much bandwidth you are using and how much bandwidth you have. I would suggest that you use Ping to find out how fast each network component in your network is transmitting to determine whether or not the problem is in the machine that is transmitting, or in the machine that is receiving or is some other machine between the two.

Let us know what you find.
 
Old 09-19-2006, 05:06 PM   #3
toko2
LQ Newbie
 
Registered: Oct 2005
Location: Toronto, Ontario
Distribution: SuSE, Ubuntu
Posts: 2

Original Poster
Rep: Reputation: 0
Well upon further inspection it looks like the problem is on the destination server. It looks like it can't handle the amount of data that is being thrown at it.

We are using: 3Ware SATA2 cards with a RAID5 configuration. The drives are formated as Reiserfs which has given us excellent performance in the past.
 
Old 09-21-2006, 12:05 PM   #4
thorn168
Member
 
Registered: Oct 2004
Location: USA
Distribution: Vector Linux 5.1 Std., Vector Linux 5.8 Std., Win2k, XP, OS X (10.4 & 10.5)
Posts: 344

Rep: Reputation: 42
What is the speed of the hard drives in the destination server? They should be as fast as the hard drives in the source server.

Also check that the hard drive buffers are also the same size.

i.e. 8mb 7500rpm = 8mb 7500rpm drives.

The only other suggestion is to check the destination server's drives for signs of malfuction; as in future drive failure.

Did you determine if fragmentation was an issue on the destination server's drives?
 
  


Reply

Tags
files, network, ssh, tar, transfer


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Proftp Server 60min timeout on large file transfers how cna i change this? methodh2k Linux - General 3 10-19-2005 08:39 AM
Large file transfers kick me off my wireless network. unholy Linux - Wireless Networking 4 10-03-2005 05:29 PM
NFS drops connection for medium-to-large transfers dddaaannn Linux - Networking 1 07-25-2005 07:51 PM
RH 9.0 / Samba Large File Transfers rjesse Linux - General 3 01-26-2005 09:59 AM
Large file transfers crash Slack 9.1? steyr Slackware 3 02-14-2004 02:23 PM


All times are GMT -5. The time now is 04:35 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration