LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 09-25-2010, 01:12 AM   #1
mky
LQ Newbie
 
Registered: Sep 2010
Posts: 4

Rep: Reputation: 0
Bash backup script with scp, multiple processes


Hi,i wrote a script for backing up my data over the network with scp in crontab, but i get over 50 processes and the script is only by the first file ,and that make my servers load very high, how can i lock the process ? Or is it better when i use rsync or nfs+copy ?

Code:
REMOTE=1.2.3.4

SOURCE=/data/some/directory
TARGET=/mnt/some/directory

LOG=/root/bck.log
 
DATE=`date +%y\.%m\.%d\.`

USER=root


ssh $USER@$REMOTE mkdir $TARGET/$DATE


if [ -d "$SOURCE" ]; then
    for i in `ls $SOURCE | grep 'data'`;do
	     echo "Begining copy of" $i  >> $LOG
	     scp  $SOURCE/$i $USER@$REMOTE:$TARGET/$DATE
	     echo $i "completed" >> $LOG
		
		if [ -n `ssh $USER@$REMOTE ls $TARGET/$DATE/$i 2>/dev/null` ];then
		    rm $SOURCE/$i
		    echo $i "removed" >> $LOG
		    echo "####################" >> $LOG
				else
					echo "Copy not complete" >> $LOG
					exit 0
		fi 
    done
     

else

    echo "Directory is not present" >> $LOG
    exit 0
fi
 
Old 09-25-2010, 01:37 AM   #2
Meson
Member
 
Registered: Oct 2007
Distribution: Arch x86_64
Posts: 606

Rep: Reputation: 66
Fifty of what procs? That scp command, concurrently? How often is your cronjob set for?

Also, just a tip, if you're trying to copy over all files from $SOURCE with "data" in the filename, try:

Code:
scp $SOURCE/*data* $USER@$REMOTE:$TARGET/$DATE
you could do even better with rsync. Don't create the directory on the other side until you know $SOURCE contains files. You can take care of all of this with rsync's prune empty dirs option and relative path's option.
 
1 members found this post helpful.
Old 09-25-2010, 02:21 AM   #3
mky
LQ Newbie
 
Registered: Sep 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Thanks for the quick reply.
I get 50 of the same scp proc,yes. The cronjob is set to everyday at eight.
Code:
* 8 * * * root /root/daily-backup
and my first script looked as you wrote in your code, but there was this problem too.

I try what you said, make a script with rsync.
If anyone know why i get this procs, please reply.
 
Old 09-25-2010, 02:47 AM   #4
kurumi
Member
 
Registered: Apr 2010
Posts: 223

Rep: Reputation: 45
Quote:
Originally Posted by mky
Code:
    for i in `ls $SOURCE | grep 'data'`;do
	     echo "Begining copy of" $i  >> $LOG
	     scp  $SOURCE/$i $USER@$REMOTE:$TARGET/$DATE
	     echo $i "completed" >> $LOG
		
		if [ -n `ssh $USER@$REMOTE ls $TARGET/$DATE/$i 2>/dev/null` ];
Let's analyse this portion of your code. The for loop can be written,
Code:
for i in *data*
using shell expansion. There is no need to use ls and grep. These cost you extra processes.
Next, for every data file found (say for eg 500 of them), you call scp. that means calling scp 500 times.
Next, you also did a ssh, this will also be called 500 times using the for loop.
 
Old 09-25-2010, 04:50 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,285
Blog Entries: 54

Rep: Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854Reputation: 2854
Quote:
Originally Posted by Meson View Post
you could do even better with rsync.
I agree. Rsync would be the efficient way to accomplish this task.
 
Old 09-25-2010, 05:17 AM   #6
mky
LQ Newbie
 
Registered: Sep 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
Let's analyse this portion of your code. The for loop can be written
...
Thank's for the example, i tought that the script is going sequentially so
the first scp begin, and the script goes forward when scp finished with copy, my fault.But would it be a solution when i use "&&" ?
Code:
scp  $SOURCE/$i $USER@$REMOTE:$TARGET/$DATE &&
Quote:
I agree. Rsync would be the efficient way to accomplish this task.
I try a new script with rsync but originally was the script with rsync and my server cached all the backup data, and the CPU and I/O was on 100%.
That's why i search a sequentially solution.
 
Old 09-25-2010, 08:33 AM   #7
Meson
Member
 
Registered: Oct 2007
Distribution: Arch x86_64
Posts: 606

Rep: Reputation: 66
Give this a try

Code:
#!/bin/bash

REMOTE=1.2.3.4

SOURCE=/data/some/directory
TARGET=/mnt/some/directory

LOG=/root/bck.log

DATE=`date +%y\.%m\.%d\.`

USER=root

if [[ -d "$SOURCE" ]]; then

        echo "Directory is not present" >> $LOG
        exit 0
fi

echo "($(date)) Starting Backup" >> $LOG

rsync -aih --remove-source-files \
      $SOURCE/*data* \
      $USER@$REMOTE:$TARGET/$DATE &>> $LOG

if [[ $? ]]; then
        
        echo "($(date)) Complete" >> $LOG
else
        
        echo "($(date)) Fail" >> $LOG
fi
 
Old 09-29-2010, 08:35 AM   #8
mky
LQ Newbie
 
Registered: Sep 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Thank You Meson ! It worked, and my servers load was not so high, i don't understand, but i'm happy for the solution.
 
Old 09-29-2010, 06:28 PM   #9
Meson
Member
 
Registered: Oct 2007
Distribution: Arch x86_64
Posts: 606

Rep: Reputation: 66
There're two keys here. One with scp or rsync, you can do file globbing with the command itself; meaning you don't need to loop through all of the files, you can transfer them with a single instance. Also, for convenience, rsync has that option to remove the source file, so you know it will only be removed if the transfer was successful. Read the rsync man page a few times over, it's very powerful.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash 'read' built-in with multiple processes reading same descriptor ta0kira Programming 41 09-13-2009 04:58 PM
Is 'nice' inherited to child processes? e.g. bash script/php script that calls MySQL SirTristan Linux - Newbie 1 12-04-2008 12:57 AM
Stopping Multiple processes with a script rajdeepbhattacharya Linux - Newbie 1 06-15-2008 05:00 PM
scp in bash script only getting to 11% just_me_then Linux - General 5 04-27-2008 03:02 AM
Bash backup script - If multiple files starting with a exist problem demoncheese Programming 2 07-29-2004 10:47 PM


All times are GMT -5. The time now is 07:01 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration