LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 08-06-2016, 02:46 AM   #1
crack00r
LQ Newbie
 
Registered: Aug 2016
Posts: 5

Rep: Reputation: Disabled
Parallel transfer with inotifywait ore other


Hi, i use your tut, and it works great so far,
when i use
https://github.com/yadayada/acd_cli
to mount ACD and i get uploadspeed with
1 File = 30mb/s
4 Files parallel 120mb/s
4 Files with "acd_cli ul -x 8 -r 5" = 60mb/s

Here are Screenshots: http://imgur.com/a/Hqy1w

shouldnt it allways at maxpseed?

Im at 1gig/1gig so bottleneck is not upload/downlad, and it seams on parallel connections amazon isnt the bottleneck too



I also try this script, and push files over and over to the folder, but i dont get more speed of 30mb/s like 1 file transfer
it reconize new files, but dont speed up parallel transfers, it transfer 1 file, and when its finish, its transfer next file...

#!/bin/bash
FOLDER="/home/plex/toupload"
inotifywait -m $FOLDER -e create -e moved_to |
while read path action file; do
echo "The file '$file' appeared in directory '$path' via '$action'"
acd_cli ul $FOLDER/"$file" /Media/Unsorted/
rm -rf $FOLDER/"$file"
done

is there any other way to open a parallel transfer everytime a file is join the folder?
So, FIle 1 needs 5min. to copy to the Cloud, and after 30 sec. next file joins, transfer this instant too, and not waiting for finish the 1 file


i dont stick to inotifywait, i just want to transfer everything instant when it joins the folder

i get higher mb/s rate with parallel transfers
1 file after onther is to slow



ore a cron check every minute the folder, and when a NEW file joins, transfer only this... PARALlEL
 
Old 08-06-2016, 07:45 AM   #2
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,948
Blog Entries: 11

Rep: Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421Reputation: 5421
Quote:
Originally Posted by crack00r View Post
your
who is this "your" you are addressing?
linuxquestions.org is a forum for linux questions, but not specifically for the application you are talking about.
for somebody not familiar with this app, it is totally impossible to understand what you are saying.
i think you should open an issue on the github repo you linked.
 
Old 08-06-2016, 08:02 AM   #3
crack00r
LQ Newbie
 
Registered: Aug 2016
Posts: 5

Original Poster
Rep: Reputation: Disabled
Sorry, was a a copy paste from some where else
http://makeshift.ninja
 
Old 08-06-2016, 08:03 AM   #4
crack00r
LQ Newbie
 
Registered: Aug 2016
Posts: 5

Original Poster
Rep: Reputation: Disabled
Its not a product i have problem with.
I just want to move files when the appear in a folder.
And not 1 after other, it should move the instant when the get to the folder
 
Old 08-06-2016, 08:11 AM   #5
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,383

Rep: Reputation: 406Reputation: 406Reputation: 406Reputation: 406Reputation: 406
Hi

You can add a & after a command to have it run in the background. Here you probably don't want to delete the file before the upload is done. But you can add () around many, so they run after one another in the background

Code:
...
(
acd_cli ul $FOLDER/"$file" /Media/Unsorted/
rm -rf $FOLDER/"$file"
) &
...
 
1 members found this post helpful.
Old 08-06-2016, 01:51 PM   #6
crack00r
LQ Newbie
 
Registered: Aug 2016
Posts: 5

Original Poster
Rep: Reputation: Disabled
thx, this works perfect

Code:
#!/bin/bash
FOLDER="/root/Daten"
inotifywait -r -m $FOLDER -e close_write -e moved_to |
        while read path action file; do
                 (
                   acd_cli ul -x 4 -r 4 -f -rsf $FOLDER/ /
                  ) &
        done
 
Old 08-07-2016, 01:45 AM   #7
crack00r
LQ Newbie
 
Registered: Aug 2016
Posts: 5

Original Poster
Rep: Reputation: Disabled
it works so perfect, it create to much backgrounds

is is may possible to put in a waiting loop when there are allready 10 acd_cli pids working?

Last edited by crack00r; 08-07-2016 at 03:38 AM.
 
Old 08-08-2016, 11:58 AM   #8
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,383

Rep: Reputation: 406Reputation: 406Reputation: 406Reputation: 406Reputation: 406
Yes it's possible, but a bit more complicated.

The script would have to count how many are working, and if 10 or more, keep checking again and again. The difficult thing is to get from 0 to 10 jobs working all the time. And you don't know how long time each one takes.

But there are tools for this. You need GNU parallel. Install it if you need, and make sure it's not a very old version.

https://www.gnu.org/software/parallel/man.html

One example is with inotify and parallel:

Quote:
If you have a dir in which users drop files that needs to be processed you can do this on GNU/Linux (If you know what inotifywait is called on other platforms file a bug report):

inotifywait -q -m -r -e MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |\
parallel -u echo

This will run the command echo on each file put into my_dir or subdirs of my_dir.
You can add "-j 10" to the parallel command to run max 10. By default, it will count how many CPU cores you have, because it's usually used to keep all CPU cores busy.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Nvidia and ARM: It's a parallel, parallel, parallel world LXer Syndicated Linux News 0 03-21-2013 06:10 PM
Apt Ign(ore) translation-nb Joachim Schrader Ubuntu 0 04-25-2011 06:26 AM
Any alternatives to Fastlynx? (parallel file transfer software) mike333md Linux - Software 1 01-31-2007 07:44 PM
File transfer over parallel port. nethunter Linux - Networking 1 03-18-2006 01:46 PM
parallel file transfer in slackware Andriy Slackware 9 01-25-2006 07:17 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration