Script to copy files, rsync?

Ruffe42 · 04-28-2014, 04:21 PM

Hi,

I need some help to make a script to copy files from a local folder to a network drive.
First of all I want to tell you this is my first time in linux environment, so be kind and clear. tongue.gif

My setup: Raspberry Pi with Raspbian wheezy, with a local connected USB-drive and a network drive. I have mounted both drives so they are available from the pi.

/home/pi DOWNLOAD > completed (local USB-drive)
MOVIES > HD (Network Drive part.1)
TV > SERIES > X (Network Drive part.2)
Y (Network Drive part.2)
Z (Network Drive part.2)

I want a script that copy all specific files (.avi etc) from /home/pi/DOWNLOAD/completed which dosen't already exist on the destination and put those in the predetermined folder.

etc. /home/pi/DOWNLOAD/completed/xxxxxYxx.avi goes to /home/pi/TV/SERIES/Y
/home/pi/DOWNLOAD/completed/xxxxxXxx.avi goes to /home/pi/TV/SERIES/X
/home/pi/DOWNLOAD/completed/xxxxxZxx.avi goes to /home/pi/TV/SERIES/Z
/home/pi/DOWNLOAD/completed/xxxxxxxx.avi goes to /home/pi/MOVIES/HD

I've struggled with rsync for a few days now and I think I got a good structure but somehow nothing happens when I run the .sh-script and when I try to run the rsync command manually it just say it is a (dry-run) even though I don't have the parameter -n in the command.

here is my "script" ^^'

#log-file
LOG_FILE=/home/pi/TV/rsynclogg.log

#the header for log-file
echo '-----------------------------------------------------------------------------' >> $LOG_FILE
date >> $LOG_FILE

#rsync commands
rsync -av --include='*/' --include='*X*.avi' --exclude '*' -prune-emty-dirs /home/pi/DOWNLOAD/completed /home/pi/TV/SERIES/X &>> $LOG_FILE
rsync -av --include='*/' --include='*Y*.avi' --exclude '*' -prune-emty-dirs /home/pi/DOWNLOAD/completed /home/pi/TV/SERIES/Y &>> $LOG_FILE
rsync -av --include='*/' --include='*Z*.avi' --exclude '*' -prune-emty-dirs /home/pi/DOWNLOAD/completed /home/pi/TV/SERIES/Z &>> $LOG_FILE
rsync -av --include='*/' --include='*.avi' --exclude '*' --exclude='*X*.avi' --exclude='*Y*.avi' --exclude='*Z*.avi' -prune-emty-dirs /home/pi/DOWNLOAD/completed /home/pi/MOVIES/HD &>> $LOG_FILE

I would really appreciate some help, maybe there is another tool I can use? I'm stuck and really desperate..

Please help me!
br J

unSpawn · 04-28-2014, 05:47 PM

Something like this?

Code:

#!/bin/bash --
# Unset debug when done testing:
set -vxe
# Set default behaviour:
LANG=C; LC_ALL=C; export LANG LC_ALL
RSYNC_ARGS="--dry-run --checksum --itemize-changes --human-readable --stats --log-file=/home/pi/TV/rsync.log"

# Throw in some ways to nice things:
# which nice >/dev/null 2>&1 && nice -n +20 $$ >/dev/null 2>&1
# which ionice >/dev/null 2>&1 && ionice -c3 -p $$ >/dev/null 2>&1
# RSYNC_ARGS="$RSYNC_ARGS --bwlimit=1m"

for ITEM in X Y Z; do
 _MYTMPFILE=`mktemp XXXXXXXXXX` && {
 find /home/pi/DOWNLOAD/completed/ -type f -name \*${ITEM}\*.avi > "${_MYTMPFILE}"
 rsync $RSYNC_ARGS --files-from="${_MYTMPFILE}" "/home/pi/TV/SERIES/${ITEM}/"
 }; rm -f "${_MYTMPFILE}"
done

_MYTMPFILE=`mktemp XXXXXXXXXX` && {
find /home/pi/DOWNLOAD/completed/ -type f \( -not -name \*X\*.avi -a -not -name \*Y\*.avi -a -not -name \*Z\*.avi \)
rsync $RSYNC_ARGS --files-from="${_MYTMPFILE}" /home/pi/MOVIES/HD/
}; rm -f "${_MYTMPFILE}"

exit 0

See 'man rsync' for explanation of rsync options.
Remove the "--dry-run" when you think it looks OK.

Ruffe42 · 04-28-2014, 06:33 PM

Quote:

Originally Posted by unSpawn

Something like this?

Code:

#!/bin/bash --
# Unset debug when done testing:
set -vxe
# Set default behaviour:
LANG=C; LC_ALL=C; export LANG LC_ALL
RSYNC_ARGS="--dry-run --checksum --itemize-changes --human-readable --stats --log-file=/home/pi/TV/rsync.log"

# Throw in some ways to nice things:
# which nice >/dev/null 2>&1 && nice -n +20 $$ >/dev/null 2>&1
# which ionice >/dev/null 2>&1 && ionice -c3 -p $$ >/dev/null 2>&1
# RSYNC_ARGS="$RSYNC_ARGS --bwlimit=1m"

for ITEM in X Y Z; do
 _MYTMPFILE=`mktemp XXXXXXXXXX` && {
 find /home/pi/DOWNLOAD/completed/ -type f -name \*${ITEM}\*.avi > "${_MYTMPFILE}"
 rsync $RSYNC_ARGS --files-from="${_MYTMPFILE}" "/home/pi/TV/SERIES/${ITEM}/"
 }; rm -f "${_MYTMPFILE}"
done

_MYTMPFILE=`mktemp XXXXXXXXXX` && {
find /home/pi/DOWNLOAD/completed/ -type f \( -not -name \*X\*.avi -a -not -name \*Y\*.avi -a -not -name \*Z\*.avi \)
rsync $RSYNC_ARGS --files-from="${_MYTMPFILE}" /home/pi/MOVIES/HD/
}; rm -f "${_MYTMPFILE}"

exit 0

See 'man rsync' for explanation of rsync options.
Remove the "--dry-run" when you think it looks OK.

Looks neat, would love to get that working.
Would you mind explain more? I don't really know what to replace to make it run, if you could highlight or something. ITEM, MYTMPFILE

Ruffe42 · 04-28-2014, 08:37 PM

Got some nice ideas from your code, thx!

I'm looking for something lighter and came up with this:

Code:

sudo find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*X*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/X" \;
sudo find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*Y*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/Y" \;
sudo find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*Z*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/Z" \;
sudo find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*W*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/W" \;
sudo find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*V*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/V" \;

sudo find /home/pi/DOWNLOAD -type f \( -name '*.mkv' -a -not -iname '*X*.mkv' -a -not -iname '*Y*.mkv' -a -not -iname '*Z*.mkv' -a -not -iname '*W*.mkv' -a -not -iname 'V*.mkv' -a -not -iname '*sample*.mkv' \) -exec cp -i -n {} "/home/pi/MOVIES/HD" \;

Each line runs smooth and good in the cmd, but when I run the .sh file, nothing happens.
How do I turn the code above to a working script?

br J

unSpawn · 04-30-2014, 02:51 PM

Quote:

Originally Posted by Ruffe42

I don't really know what to replace to make it run, if you could highlight or something. ITEM, MYTMPFILE

If you save the code as script, say "/usr/local/bin/rsync_movies.sh" you can run it as '/bin/bash /usr/local/bin/rsync_movies.sh'.
The output will show you what it does (how it works) but it will essentially move no files.
If you no longer want to see what it does comment out the "set -vxe" line (should look like "#set -vxe" then). You should still see output in "/home/pi/TV/rsync.log".
If you think it's what you want then you have to delete the "--dry-run" switch from the RSYNC_ARGS= line (looks like RSYNC_ARGS="--checksum --itemize-changes --human-readable --stats --log-file=/home/pi/TV/rsync.log" then).

unSpawn · 04-30-2014, 02:55 PM

Quote:

Originally Posted by Ruffe42

How do I turn the code above to a working script?

Save this:

Code:

#!/bin/bash

find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*X*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/X" \;
find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*Y*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/Y" \;
find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*Z*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/Z" \;
find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*W*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/W" \;
find /home/pi/DOWNLOAD -type f \( -not -iname '*sample*.mkv' -a -iname '*V*.mkv' \) -exec cp -i -n {} "/home/pi/TV/SERIES/V" \;
find /home/pi/DOWNLOAD -type f \( -name '*.mkv' -a -not -iname '*X*.mkv' -a -not -iname '*Y*.mkv' -a -not -iname '*Z*.mkv' -a -not -iname '*W*.mkv' -a -not -iname 'V*.mkv' -a -not -iname '*sample*.mkv' \) -exec cp -i -n {} "/home/pi/MOVIES/HD" \;

exit 0

as say "/usr/local/bin/move_movies.sh" and make it executable ('chmod 0755 /usr/local/bin/move_movies.sh'). Then if you execute '/usr/local/bin/move_movies.sh' (shouldn't need sudo if your home is /home/pi/ else use 'sudo /usr/local/bin/move_movies.sh') files should be moved according to your script. IIRC the curly braces need to be escaped BTW.