LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 12-14-2008, 01:39 AM   #1
superc0w
LQ Newbie
 
Registered: Dec 2008
Location: Sunnyvale, CA
Distribution: Ubuntu, Solaris
Posts: 5

Rep: Reputation: 0
Question copy one large directory or multiple cp's on the sub directories


Does anyone know what would be faster? Copying 1 large directory or copying the sub directories with multiple copy commands

example:
cp -r /backups/one_large_dir /backups/one_large_dir1

OR

cp -r /backups/one_large_dir/smaller_dir1 /backups/one_large_dir1/smaller_dir1 &

cp -r /backups/one_large_dir/smaller_dir2 /backups/one_large_dir1/smaller_dir2 &

cp -r /backups/one_large_dir/smaller_dir3 /backups/one_large_dir1/smaller_dir3 &

etc. etc.
 
Old 12-14-2008, 02:20 AM   #2
kaz2100
Senior Member
 
Registered: Apr 2005
Location: Penguin land, with apple, no gates
Distribution: SlackWare > Debian testing woody(32) sarge etch lenny squeeze(+64) wheezy jessie stretch buster
Posts: 1,643

Rep: Reputation: 98
Hya,

I am not sure if whatever you want is just a shellscript.

I somewhat think that many penguins used to use tar to archive a huge file into several volumes.

If shiellscript, it would be something like
Code:
@ a = 10
while (a > 1) cp -r blah$a blAH$a
@ a--
end
Happy Penguins!
 
Old 12-14-2008, 02:39 AM   #3
superc0w
LQ Newbie
 
Registered: Dec 2008
Location: Sunnyvale, CA
Distribution: Ubuntu, Solaris
Posts: 5

Original Poster
Rep: Reputation: 0
Thanks much for the answer, but it wasn't quite what I was looking for.

I'm just wondering if there is a faster way to copy large amounts of data (~750G).

I am just curious what the difference is time-wise between copying one large directory or copying the sub-directories over on multiple threads.


Thanks!
 
Old 12-14-2008, 02:48 AM   #4
billymayday
LQ Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
I don't know the answer, but my gut feel is that it's unlikely to make much if any difference, because the constraint on the process will be (i) the disk drive, and (ii) the IO bus. I would fully imagine that one of these will be the bottleneck, and splitting the process into multiple processes won't speed things up. More likely it will slow things down.

Basically, throwing CPU or other power at an IO bound problem probably won't help.

If for some reason throwing CPU power at the problem could relieve this then the answer changes (for example compressing data to copy between systems).

WHat would be interesting would be for you to try a sample and post the results.
 
Old 12-14-2008, 02:53 AM   #5
superc0w
LQ Newbie
 
Registered: Dec 2008
Location: Sunnyvale, CA
Distribution: Ubuntu, Solaris
Posts: 5

Original Poster
Rep: Reputation: 0
Thanks that's much more along the lines of what I'm looking for.

I will run the tests over the next few days and post the results.
 
Old 12-15-2008, 01:06 AM   #6
rob.rice
Senior Member
 
Registered: Apr 2004
Distribution: slack what ever
Posts: 1,036

Rep: Reputation: 185Reputation: 185
do a cp -r just before you go to bed and let the computer run over night
this way it doesn't matter how long it tacks your sleeping
 
Old 12-15-2008, 01:49 PM   #7
superc0w
LQ Newbie
 
Registered: Dec 2008
Location: Sunnyvale, CA
Distribution: Ubuntu, Solaris
Posts: 5

Original Poster
Rep: Reputation: 0
Thanks for your advice!

The issue is this isn't your average copy, its nearly 750GB of data. Running a cp -r over night is what the cron runs every single day. The whole process (it's a backup system) take over 24 hours, and for some strange reason stops the backup in it's tracks. So I'm trying to make each piece of the puzzle more efficient.

Does anyone know a good benchmarking software so I can test to see which is faster? I have munin right now constantly monitoring the system, but I don't feel it's good for a benchmark of which method would be faster.
 
Old 12-15-2008, 02:14 PM   #8
billymayday
LQ Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
try

time cp -r ....

Do this on a smal job to see if time gives the output you want.
 
  


Reply

Tags
processing


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Copy files from multiple directories into one directory MadRabbit Linux - Newbie 8 02-07-2014 07:56 PM
Append md5sum or similar to filename, batch copy from multiple directories to one migla Linux - Newbie 5 02-03-2008 05:54 PM
Copy one file to multiple vhost directories thetawaverider Linux - Newbie 5 03-24-2006 04:57 PM
Copy one file to multiple directories cmfarley19 Linux - General 9 11-10-2004 12:57 PM
find and copy files into multiple directories avargas22 Linux - Newbie 2 04-01-2004 11:11 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 05:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration