LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 07-04-2016, 11:40 AM   #1
gdpr004
LQ Newbie
 
Registered: Jul 2016
Posts: 4

Rep: Reputation: Disabled
What is a command for fast parallel copying of whole directory to two different disks of notebook?


I need command that reads files ONCE and then simultaneously copies them to multiply destinations, namely external USB-HDD and internal HDD of my notebook. Something like "cp -vax /from/here /to/here01 /to/here02" (of course in reality this example doesn't work because "cp" just can't do it). Parallel execution of "cp" doesn't count because in this case command will read files twice. I have many files and complex directory tree to copy, so just figuring out how to simultaneously copy ONE file in different destinations isn't enough.
 
Old 07-05-2016, 04:35 PM   #2
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,270

Rep: Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458
Hello and welcome to LQ.

If you mean to use this for live state then I doubt it will work as expected. If you just want to limit read on original then maybe look at the ideas here and see if they would do. http://serverfault.com/questions/137...-same-filelist
 
Old 07-05-2016, 09:15 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 20,177

Rep: Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752Reputation: 3752
Quote:
Originally Posted by roboq6 View Post
Parallel execution of "cp" doesn't count because in this case command will read files twice.
Just proves once again it's a good thing (operating) system designers take a more holistic view than the users.
What counts is not how often each file/directory is read, but how often the disk is physically accessed.

Kick off two cp in background and let the system handle things appropriately.
 
Old 07-11-2016, 08:24 AM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by syg00 View Post
Just proves once again it's a good thing (operating) system designers take a more holistic view than the users.
What counts is not how often each file/directory is read, but how often the disk is physically accessed.

Kick off two cp in background and let the system handle things appropriately.
Yup. One place would do complete reinstall of a lab containing 15-20 machines (I think it was). As long as all the client systems were booted within a couple of seconds the reinstall only took an hour. If one or two were balky and need to take 5 minutes to get the boot started, it could take up to four hours as everything started thrashing (they found it better to just wait an hour and then do the balky systems in a separate pass).
 
Old 07-15-2016, 03:02 PM   #5
slackartist
Member
 
Registered: Oct 2015
Location: Where the buffalo's roam
Distribution: Slackware Current Multilib
Posts: 150
Blog Entries: 20

Rep: Reputation: 26
i dont understand that, are you talking about a pxe install?.
 
Old 07-15-2016, 11:19 PM   #6
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1012Reputation: 1012Reputation: 1012Reputation: 1012Reputation: 1012Reputation: 1012Reputation: 1012Reputation: 1012
I'm pretty sure the original idea would not actually save much, if any, time. Although, theoretically it could.
 
Old 07-16-2016, 06:53 AM   #7
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by AwesomeMachine View Post
I'm pretty sure the original idea would not actually save much, if any, time. Although, theoretically it could.
Nearly none.

Two copies of the same file can be done reading a file ONCE. As long as the complete file will fit in memory cache, the second copy will read the cache... 2/3/4/.. copies in parallel starting at the same time will use the cache only, even if the file is too large for cache. The only time problems show up is when the speed of one of the destinations is REALLY slow, causing a cache reload - and thus starting to thrash the cache.

The DISADVANTAGE of doing the parallel write mentioned is that the copies are now limited to the speed of the slowest device...
 
Old 07-16-2016, 06:55 AM   #8
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 17,544

Rep: Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929Reputation: 5929
what about the command tee:
Code:
cat inputfile | tee out1 out2 .... outn
to OP:
would be nice to tell us if that really helps. It depends on a lot of things, for example the size of the file, cache, i/o speed ....

Last edited by pan64; 07-16-2016 at 06:58 AM.
 
Old 07-17-2016, 03:33 AM   #9
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,137
Blog Entries: 1

Rep: Reputation: 873Reputation: 873Reputation: 873Reputation: 873Reputation: 873Reputation: 873Reputation: 873
Any reason there couldn't just be one copy of the file(s) on a network share? I suppose rsync could do what you want, but it's gonna read for each instance. A technicality if you make one copy to a clone and then read the clone multiple times. Although tee would probably more accurately follow your rules. If you consider each pipe as not a read. A series of daisy chained mirrors perhaps if network bandwidth is setup that way, but more of a faster way, than best way. Assuming no breaks in the chain. Pros and Cons whichever route is chosen.
 
Old 07-17-2016, 05:43 AM   #10
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
As long as all the computers start copying in a second or two, it would work just fine.

If a couple are out of sync then the server could start thrashing - depending on the size of the file(s) being copied.
 
Old 07-17-2016, 11:39 AM   #11
gdpr004
LQ Newbie
 
Registered: Jul 2016
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by syg00 View Post
Just proves once again it's a good thing (operating) system designers take a more holistic view than the users.
What counts is not how often each file/directory is read, but how often the disk is physically accessed.

Kick off two cp in background and let the system handle things appropriately.
Okay, it seems like you're right, my OS will just use cache in order to achieve better performance.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Copying home folder from command line without copying hidden directories and files rm_-rf_windows Linux - General 2 04-12-2016 09:24 AM
[SOLVED] Executing a command in parallel | GNU parallel or xargs the_gripmaster AIX 3 05-08-2012 08:41 AM
Linux command for copying a directory in windows to the Linux partition thelink123 Linux - Newbie 6 08-21-2008 05:27 PM
Copying OS disks rwalkerphl Linux - General 1 04-20-2006 09:46 AM
Copying Disks Stephanie Linux - General 12 09-04-2002 09:47 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:33 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration