LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-15-2008, 05:01 PM   #1
ufmale
Member
 
Registered: Feb 2007
Posts: 385

Rep: Reputation: 30
copy terabyte of data from one machine to another


what is a good way to copy large amount of data in the size of 3-4 terabytes (over the network). What i did was

Code:
mount  //remotemachine/data /mnt/remote - o username=root
Then i just copy them from the console

Code:
cp -rv /mnt/remote/* /mnt/md0
It will run for awhile, then the console hangs for some reason.
I have to reboot the machine, then use rsync to do another copy session

I am using Redhat el4 with ext3 filesystem.

I am lookiing for a good way to do the job more efficiently and reliable.
 
Old 05-15-2008, 07:08 PM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,580
Blog Entries: 14

Rep: Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969
Why not use rsync?

The benefits to using it:
1) You can compress the data before it is transferred so it limits bandwidth usage. That impacts cpu on both hosts but usually the network is your bottleneck so it helps to use the compression flag.
2) If it fails you can restart the rsync and and it will only transfer the files that haven't already transferred.
3) You can actually start multiple sessions of rsync to do separate sets of files if you have enough cpu/memory/bandwidth to accommodate the multiple sessions. This would allow you to transfer more quickly though of course isn't quite as simple as just saying transfer it all in one session that takes longer.
 
Old 05-15-2008, 08:10 PM   #3
SlowCoder
Member
 
Registered: Oct 2004
Location: Southeast, U.S.A.
Distribution: Fedora (Desktop), CentOS (Server), Knoppix (Diags)
Posts: 934

Rep: Reputation: 38
I wish I had that kind of storage space!

I think you should try to determine what is causing the freeze of your session.
- Is it some sort of inactivity timeout? If so, maybe you can disable the timeout, or set up a cron job that sends a keypress event to the session?
- Maybe you've got a periodic network outage issue, possibly caused by mispatched duplex/speed settings on one of the machines?
- Is the server you are transferring from a Windows box that likes to freak randomly during such a large transfer?
- Would it be viable to multihome each machine and create a direct connection between them, removing all outside influences like network switches, routers, etc.?

Just my limited thinking on this subject.
 
Old 05-16-2008, 01:17 AM   #4
ufmale
Member
 
Registered: Feb 2007
Posts: 385

Original Poster
Rep: Reputation: 30
Quote:
- Is it some sort of inactivity timeout? If so, maybe you can disable the timeout, or set up a cron job that sends a keypress event to the session?
How do I check and change it?

Quote:
- Maybe you've got a periodic network outage issue, possibly caused by mispatched duplex/speed settings on one of the machines?
Possibly. is there a log file i can investigate?

The other machine is also linux, but i think it has different journal file system.
Mine has ext3
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Copy directories/files from Fedora 8 machine to SuSE 10.3 machine r-t Linux - Newbie 6 03-11-2008 08:54 AM
scp: copy a file from local machine to remote machine seran Linux - Newbie 8 10-30-2007 01:23 PM
trying to copy files from another machine to the local machine using telnet,..how? shrike_912 Programming 6 03-14-2006 05:45 PM
copy from windows machine to a linux machine s_deepblue Linux - Networking 3 03-12-2005 02:43 PM
Using SMB to copy files from one linux machine to second linux machine. coppersky Linux - Networking 9 06-24-2003 01:07 PM


All times are GMT -5. The time now is 05:28 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration