Parallel transfer with inotifywait ore other

crack00r · 08-06-2016, 02:46 AM

Hi, i use your tut, and it works great so far,
when i use
https://github.com/yadayada/acd_cli
to mount ACD and i get uploadspeed with
1 File = 30mb/s
4 Files parallel 120mb/s
4 Files with "acd_cli ul -x 8 -r 5" = 60mb/s

Here are Screenshots: http://imgur.com/a/Hqy1w

shouldnt it allways at maxpseed?

I´m at 1gig/1gig so bottleneck is not upload/downlad, and it seams on parallel connections amazon isnt the bottleneck too

I also try this script, and push files over and over to the folder, but i dont get more speed of 30mb/s like 1 file transfer
it reconize new files, but dont speed up parallel transfers, it transfer 1 file, and when its finish, its transfer next file...

#!/bin/bash
FOLDER="/home/plex/toupload"
inotifywait -m $FOLDER -e create -e moved_to |
while read path action file; do
echo "The file '$file' appeared in directory '$path' via '$action'"
acd_cli ul $FOLDER/"$file" /Media/Unsorted/
rm -rf $FOLDER/"$file"
done

is there any other way to open a parallel transfer everytime a file is join the folder?
So, FIle 1 needs 5min. to copy to the Cloud, and after 30 sec. next file joins, transfer this instant too, and not waiting for finish the 1 file

i dont stick to inotifywait, i just want to transfer everything instant when it joins the folder

i get higher mb/s rate with parallel transfers
1 file after onther is to slow

ore a cron check every minute the folder, and when a NEW file joins, transfer only this... PARALlEL

ondoho · 08-06-2016, 07:45 AM

Quote:

Originally Posted by crack00r

your

who is this "your" you are addressing?
linuxquestions.org is a forum for linux questions, but not specifically for the application you are talking about.
for somebody not familiar with this app, it is totally impossible to understand what you are saying.
i think you should open an issue on the github repo you linked.

crack00r · 08-06-2016, 08:02 AM

Sorry, was a a copy paste from some where else
http://makeshift.ninja

crack00r · 08-06-2016, 08:03 AM

Its not a product i have problem with.
I just want to move files when the appear in a folder.
And not 1 after other, it should move the instant when the get to the folder

Guttorm · 08-06-2016, 08:11 AM

Hi

You can add a & after a command to have it run in the background. Here you probably don't want to delete the file before the upload is done. But you can add () around many, so they run after one another in the background

Code:

...
(
acd_cli ul $FOLDER/"$file" /Media/Unsorted/
rm -rf $FOLDER/"$file"
) &
...

crack00r · 08-06-2016, 01:51 PM

thx, this works perfect

Code:

#!/bin/bash
FOLDER="/root/Daten"
inotifywait -r -m $FOLDER -e close_write -e moved_to |
        while read path action file; do
                 (
                   acd_cli ul -x 4 -r 4 -f -rsf $FOLDER/ /
                  ) &
        done

crack00r · 08-07-2016, 01:45 AM

it works so perfect, it create to much backgrounds

is is may possible to put in a waiting loop when there are allready 10 acd_cli pids working?

Guttorm · 08-08-2016, 11:58 AM

Yes it's possible, but a bit more complicated.

The script would have to count how many are working, and if 10 or more, keep checking again and again. The difficult thing is to get from 0 to 10 jobs working all the time. And you don't know how long time each one takes.

But there are tools for this. You need GNU parallel. Install it if you need, and make sure it's not a very old version.

https://www.gnu.org/software/parallel/man.html

One example is with inotify and parallel:

Quote:

If you have a dir in which users drop files that needs to be processed you can do this on GNU/Linux (If you know what inotifywait is called on other platforms file a bug report):

inotifywait -q -m -r -e MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |\
parallel -u echo

This will run the command echo on each file put into my_dir or subdirs of my_dir.

You can add "-j 10" to the parallel command to run max 10. By default, it will count how many CPU cores you have, because it's usually used to keep all CPU cores busy.