LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 05-08-2006, 07:16 PM   #1
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Rep: Reputation: 51
Need to move large file(s) to lots of machines on my network. suggestions?


I need to get lots of files around to about 45 machines so they can all chew on a task together. Currently, it's setup to distribute the files via a little script I wrote that rsyncs the correct directory from one machine to the other 45. However, when we start getting large amounts of data (2+GB), the transfer takes well over an hour (for all machines). I'm looking for a way to speed that up. Accessing all the files over an NFS (or similar) share is not an option.

My first thought was to setup a bittorrent on my local network. I'm just not incredibly familiar with setting one up and I'm not sure how to launch a client on all the machines simultaneouly.

Is bittorrent the way to go? Are there other options?

Thanks
 
Old 05-09-2006, 03:19 AM   #2
tredegar
Guru
 
Registered: May 2003
Location: London, UK
Distribution: Ubuntu 10.04, mostly
Posts: 6,007

Rep: Reputation: 366Reputation: 366Reputation: 366Reputation: 366
When you do the sums, I think you'll find that it is your network bandwidth that is the limiting factor here.

Bittorrent will not move the data any faster if the network is saturated.

Upgrade to Gigabit ethernet? Fibreoptic? Change your network topology?

HTH
 
Old 05-11-2006, 06:11 PM   #3
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Quote:
Originally Posted by tredegar
When you do the sums, I think you'll find that it is your network bandwidth that is the limiting factor here.

Bittorrent will not move the data any faster if the network is saturated.

Upgrade to Gigabit ethernet? Fibreoptic? Change your network topology?
hmm...

I am using GigE switches. 10 of the nodes have GigE, the other 35 are 100Mb/s. I've never seen a file transfer go faster than 40MB/s - I don't know if the switch or the computer is the limiting factor, but I would guess it's the computer. Regardless, I would think that there's enough internal bandwidth on the switch(es) for at least 4-5 full 100Mb/s transfers at once - assuming they aren't all coming from the same machine.

That said, I hadn't thought about network bandwidth until you mentioned it - thanks for that.
 
Old 05-12-2006, 01:25 AM   #4
BSchindler
Member
 
Registered: May 2006
Location: Los Angeles, CA USA
Distribution: RHAS3, RHAS4, RHEL4, RH6, RH7.2, FC2-7, Gentoo, DSL
Posts: 49

Rep: Reputation: 15
You don't say anything about your network topology, so all I can do is just give you some food for thought. You were on the right track when you thought about bittorrent.

tredegar was correct when he indicated that network bandwidth is a limiting factor -- but that is not the only limit. The source node can only deliver so much data so fast. Remember, the data is read from a spinning disk and there is latentcy and transfer rate to consider. Really fast drive systems will give you perhaps 50MB transfer rate. Also, just because you have GBit interfaces on machines does not mean you will get 100MB transfer rates (1GBit/10bits/byte), you are using TCP/IP so there are layers of network wrappers and TCP packets still get acknowledged (that's what makes it a fault-tolerant protocol). So when you say you are transferring data at 50MB that's a good number.

Back to your torrent idea. What you are really sensing is that you want multiple machines to be performing simultaneous transfers. Consider that your nodes with GBit interfaces are G0 to G9 and your slow nodes are S0 to S34 you have a way to accomplish your "torrent". I will use the "->" symbol to indicate a transfer (rsync or whatever) from one node to another. Divide your transfer into steps like this:

Step 1: This step is the fastest it can be (2GB files should take approx 35-40 seconds)
G0 -> G5

Step 2: Now, we have two "masters" so use them to populate others (high speed nodes). As the transfers are taking place with a limited number of nodes the network is not limited as (I assume) you have GBit switches which are for-all-intents non-blocking. This won't be quite as fast but should complete in less than 3 minutes (assuming 2GB file).
G0 -> G1, G2, G3, G4
G5 -> G6, G7, G8, G9

Step 3: We now have ten masters, let them populate the S<n> nodes. These are the slowest transfers but we can expect to get these transfers done in 5 to 7 minutes.
G0 -> S0, S1, S2, S3
G1 -> S4, S5, S6, S7
G2 -> S8, S9, S10, S11
G3 -> S12, S13, S14, S15
G4 -> S16, S17, S18, S19
G5 -> S20, S21, S22
G6 -> S23, S24, S25
G7 -> S26, S27, S28
G8 -> S29, S30, S31
G9 -> S32, S33, S34

Again, I don't know your topology, but you should be able to move your 2GB files across all nodes in less than 10 minutes.

HTH
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Large file transfers kick me off my wireless network. unholy Linux - Wireless Networking 4 10-03-2005 05:29 PM
Samba Large File Transfor aborts - The specified network name is no longer available rjesse Linux - Networking 3 01-26-2005 12:37 AM
Move process between machines Flibble Linux - General 1 01-18-2004 07:59 PM
Display of Licsnese agreement is to large to click, can't move or resize when trying scottforrester Linux - Software 3 11-16-2003 12:50 PM
Suggestions on How to Move to Software RIAD gboutwel Linux - Software 1 02-02-2003 11:57 PM


All times are GMT -5. The time now is 07:57 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration