LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 01-17-2017, 07:38 PM   #1
cilbuper
Member
 
Registered: Mar 2008
Posts: 141

Rep: Reputation: 0
Copying/moving many files at once via Dolphin or similar - creating a que & transfering more efficiently


I often have to sort through a few folders and select files I want to copy or move to another folder/drive/media. I can't just highlight them all at once and do it in one process, so I select them individually and chose appropriate action and proper destination (lets say same destination for all files/folders).

Now once a few files are being copies (300MB - 4+GB each) the speed really takes a hit where I'm getting like total of 30MB/sec where copying one file alone I was getting 90-150MB sustained after the first 10 second speed burst. Adding more files can drop each file transfer rate down to 1-3MB/sec at times when I get like 10-15 files copying.

the drives are pretty robust, new 4TB WD Red and Samsung 830 SSD (source).

Is there a way to make a que where it copies 1 fiels at a time then moves on to the next? There is a program in Windows that does this where you can select 1,000 files with 1,000 different destinations and it will fly through the transfer doing one at a time (can be modded to a couple at a time if I remember correctly, especially if from different source and or destination drives).

Is there anything like this in Linux?

The strange thing is that this seems to be automatic on the command line if many files are named to be moved, the are done one at a time and speeds are much better.
 
Old 01-17-2017, 08:36 PM   #2
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (15 current), Slack15, Ubuntu studio, MX Linux, FreeBSD 13.1, WIn10
Posts: 10,342

Rep: Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242Reputation: 2242
have you ever thought about learning BASH scripting?
 
Old 01-18-2017, 03:47 AM   #3
cilbuper
Member
 
Registered: Mar 2008
Posts: 141

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by BW-userx View Post
have you ever thought about learning BASH scripting?
I would really like to do this, and have started a number of times but I've yet to find a good method for learning. Some examples (when reading online "how to's" for specific operations, I'll copy the script exactly as the poster says they are using and it won't run in my machine. While I know that isn't writing my own code, it was a way to start seeing how things worked.

Can you suggest any books, sites, or methods on how to start on this? I would like to eventually learn python (from what I have read from its benefits and "power" it seems very versitile and a good investment in my time).

As far as my origninal question on how the files copy, can what I asked be done with BASH scripting? I'm thinking of a program called Teracopy in Windows that I like for a number of reasons.

Here is an example of a program I would like to write but don't know how to start. Let's say I take 500 different files from MANY different directories across different drives (let's say 200+GB of data) and with destinations varying across different drives and folders. The program would assign each file a variable related to the physical drive it is on (sda, sdb, sdc, etc) but disregard the partitions as there could be many partitions on one drive. This would be the source variable. Then for each destination location for each file, it could cross reference the drive as well and assign another variable as the destination variable. If 100GB from dev/sda needs to be copied to dev/sdc then that is no problem, if 25GB of /dev/sdb copied to /dev/sdf, then that could be done at the same time as the 100GB transfer as the read/write of the hard drives shouldn't interfere with each other. Now if 75GB from /dev/sdh needs to be copied to /dev/sda then that would interfere with the first 100GB transfer of source files from /dev/sda so that would be put inthe que to be run after the initial 100GB transfer is completed.

Now I know much of this will depend on the type of computer that is being used (processor(s) type/speed as well as what type of hard drives are being used.

I would also like to incorporate SSD cache drives (or partitions for this) or possibly RAM drives when sufficient RAM is available. These could be used in the last example of the 75GB transfer that was to be copied to the busy /dev/sda. The data is copied from source to the cache, and when sda is ready, the data can be copied from the cache drive at top speed while freeing up the source drive of the 75GB for any other activities that needed to be done (copying, loading files:video, audio, Virtual machines, etc) without having to wait for the copying to finish.

What kind of programming would I have to know to do a project such as this.

On top of this, the program would run performance tests on drives when they are installed so it knows how it should perform and how best to handle copy/move requests as well as a few other operations I have in mind to make things a little more slick.
 
Old 01-21-2017, 05:02 AM   #4
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 19,872
Blog Entries: 12

Rep: Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053
my experience is:

- graphical file managers are much slower at copying than 'cp'
- the underlying filesystem knows how to optimize everything, no need to worry about it.

both arguments point to shell scripting.
simply construct a statement that will find the files you want to copy, and copy them with 'cp'. it will be MUCH faster than your gui file manager.
'nuff said.

if you give us a concrete example, we might even be able to help with that.
 
Old 01-21-2017, 01:34 PM   #5
rob.rice
Senior Member
 
Registered: Apr 2004
Distribution: slack what ever
Posts: 1,076

Rep: Reputation: 205Reputation: 205Reputation: 205
on the command line run "mc" midnight commander a CLI file manager
it has a select group command built in under the files menu
once selected you can run any command on the group
while not a substitute for bash scripting it would do what you want
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Copying files in dolphin - how to monitor progress and or stop process cilbuper Linux - Software 4 05-26-2018 12:57 AM
preserve date time stamp copying files over home lan with samba, dolphin glorsplitz Slackware 1 07-22-2016 05:38 AM
Networking & Transfering files stu_mueller Slackware 3 05-15-2008 10:15 AM
copying / moving files in console Four Linux - Newbie 4 03-25-2007 10:34 PM
errors while copying/moving files plisken Linux - General 14 06-07-2003 05:57 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 08:17 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration