[SOLVED] Bash backup script with scp, multiple processes

mky · 09-25-2010, 01:12 AM

Hi,i wrote a script for backing up my data over the network with scp in crontab, but i get over 50 processes and the script is only by the first file ,and that make my servers load very high, how can i lock the process ? Or is it better when i use rsync or nfs+copy ?

Code:

REMOTE=1.2.3.4

SOURCE=/data/some/directory
TARGET=/mnt/some/directory

LOG=/root/bck.log
 
DATE=`date +%y\.%m\.%d\.`

USER=root


ssh $USER@$REMOTE mkdir $TARGET/$DATE


if [ -d "$SOURCE" ]; then
    for i in `ls $SOURCE | grep 'data'`;do
	     echo "Begining copy of" $i  >> $LOG
	     scp  $SOURCE/$i $USER@$REMOTE:$TARGET/$DATE
	     echo $i "completed" >> $LOG
		
		if [ -n `ssh $USER@$REMOTE ls $TARGET/$DATE/$i 2>/dev/null` ];then
		    rm $SOURCE/$i
		    echo $i "removed" >> $LOG
		    echo "####################" >> $LOG
				else
					echo "Copy not complete" >> $LOG
					exit 0
		fi 
    done
     

else

    echo "Directory is not present" >> $LOG
    exit 0
fi

Meson · 09-25-2010, 01:37 AM

Fifty of what procs? That scp command, concurrently? How often is your cronjob set for?

Also, just a tip, if you're trying to copy over all files from $SOURCE with "data" in the filename, try:

Code:

scp $SOURCE/*data* $USER@$REMOTE:$TARGET/$DATE

you could do even better with rsync. Don't create the directory on the other side until you know $SOURCE contains files. You can take care of all of this with rsync's prune empty dirs option and relative path's option.

mky · 09-25-2010, 02:21 AM

Thanks for the quick reply.
I get 50 of the same scp proc,yes. The cronjob is set to everyday at eight.

Code:

* 8 * * * root /root/daily-backup

and my first script looked as you wrote in your code, but there was this problem too.

I try what you said, make a script with rsync.
If anyone know why i get this procs, please reply.

kurumi · 09-25-2010, 02:47 AM

Quote:

Originally Posted by mky

Code:

    for i in `ls $SOURCE | grep 'data'`;do
	     echo "Begining copy of" $i  >> $LOG
	     scp  $SOURCE/$i $USER@$REMOTE:$TARGET/$DATE
	     echo $i "completed" >> $LOG
		
		if [ -n `ssh $USER@$REMOTE ls $TARGET/$DATE/$i 2>/dev/null` ];

Let's analyse this portion of your code. The for loop can be written,

Code:

for i in *data*

using shell expansion. There is no need to use ls and grep. These cost you extra processes.
Next, for every data file found (say for eg 500 of them), you call scp. that means calling scp 500 times.
Next, you also did a ssh, this will also be called 500 times using the for loop.

unSpawn · 09-25-2010, 04:50 AM

Quote:

Originally Posted by Meson

you could do even better with rsync.

I agree. Rsync would be the efficient way to accomplish this task.

mky · 09-25-2010, 05:17 AM

Quote:

Let's analyse this portion of your code. The for loop can be written
...

Thank's for the example, i tought that the script is going sequentially so
the first scp begin, and the script goes forward when scp finished with copy, my fault.But would it be a solution when i use "&&" ?

Code:

scp  $SOURCE/$i $USER@$REMOTE:$TARGET/$DATE &&

Quote:

I agree. Rsync would be the efficient way to accomplish this task.

I try a new script with rsync but originally was the script with rsync and my server cached all the backup data, and the CPU and I/O was on 100%.
That's why i search a sequentially solution.

Meson · 09-25-2010, 08:33 AM

Give this a try

Code:

#!/bin/bash

REMOTE=1.2.3.4

SOURCE=/data/some/directory
TARGET=/mnt/some/directory

LOG=/root/bck.log

DATE=`date +%y\.%m\.%d\.`

USER=root

if [[ -d "$SOURCE" ]]; then

        echo "Directory is not present" >> $LOG
        exit 0
fi

echo "($(date)) Starting Backup" >> $LOG

rsync -aih --remove-source-files \
      $SOURCE/*data* \
      $USER@$REMOTE:$TARGET/$DATE &>> $LOG

if [[ $? ]]; then
        
        echo "($(date)) Complete" >> $LOG
else
        
        echo "($(date)) Fail" >> $LOG
fi

mky · 09-29-2010, 08:35 AM

Thank You Meson ! It worked, and my servers load was not so high, i don't understand, but i'm happy for the solution.

Meson · 09-29-2010, 06:28 PM

There're two keys here. One with scp or rsync, you can do file globbing with the command itself; meaning you don't need to loop through all of the files, you can transfer them with a single instance. Also, for convenience, rsync has that option to remove the source file, so you know it will only be removed if the transfer was successful. Read the rsync man page a few times over, it's very powerful.