LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-12-2020, 06:10 AM   #1
snorre
LQ Newbie
 
Registered: Jul 2020
Posts: 20

Rep: Reputation: Disabled
Question rsync-Backup-Script does not exclude directories


Hi and a very good afternoon to all of you!

I am trying to backup a Raspberry Pi on my QNAP NAS and therefore created a bash-script,
which should then be triggered via crontab.

I want to exclude some special folders, like the ones from docker, and have therefore
created an exclude file which I would like to use as parameter for the rsync command.

Long story short. Here is what I have done so far...

Code:
#!/bin/bash

readonly MOUNT_POINT='/mnt/nfs/backup'
readonly TARGET="${MOUNT_POINT}/Pi3A"
readonly EXCLUDEFILE='exclude.txt'

SOURCES=(/mnt/sda1)

# rSync Switches
readonly RSYNCPARAMS='-avR -hh'

# Configuration von rsync
readonly RSYNCCONFIG='--delete'

# Shortcuts for commands
readonly RSYNC='/usr/bin/rsync'

target="${TARGET}/"
readonly LOG="${0}".log

echo "------------------------------------------------------------------------------" > "${LOG}"
date >> "${LOG}"
echo "" >> "${LOG}"
today=$(date +%d)
echo "" >> "${LOG}"

echo "------------------------------------------------------------------------------" >> "${LOG}"
echo "rSync:" >>  "${LOG}"
echo "Params: ${RSYNCPARAMS}" >> "${LOG}"
echo "Config: ${RSYNCCONFIG}" >> "${LOG}"
echo "------------------------------------------------------------------------------" >> "${LOG}"

for source in "${SOURCES[@]}"; do
  echo "" >> "${LOG}"
  echo "Synchronizing directory \"${source}\" ..." >> "${LOG}"
  ${RSYNC} ${RSYNCPARAMS} --exclude-from="${EXCLUDEFILE}" ${RSYNCCONFIG[@]} ${source} ${target}${today} >> "${LOG}" 2>&1 
  if [ ${?} -ne 0 ]; then 
    ERROR=1
  fi 
  echo "------------------------------------------------------------------------------" >> "${LOG}"
done

echo "" >> "${LOG}"
echo "------------------------------------------------------------------------------" >> "${LOG}"
As far as I understood different manuals and descriptions, the path given in the config
file must be relative to the source.

So in my case the source path is

/mnt/sda1

and this is what I tried so far without success...

Code:
# tried this
/sda1/lost+found
#tried that
mnt/sda1/lost+found
The folder lost+found was copied to the server in either case.

Any Ideas?

Many thanks in advance,
Mike
 
Old 07-12-2020, 07:12 AM   #2
markd81
LQ Newbie
 
Registered: Jul 2020
Location: Netherlands
Distribution: Debian
Posts: 17

Rep: Reputation: 3
Try lost+found without a preceding path.

Code:
md@nukkie:~/test$ ls -R syncdir/
syncdir/:
dir1  dir2  dir3

syncdir/dir1:
file1  file2  file3

syncdir/dir2:
file1  file2  file3

syncdir/dir3:
file1  file2  file3
md@nukkie:~/test$ cat excludefile.txt 
dir1
md@nukkie:~/test$ rsync -avr -hh --exclude-from="excludefile.txt" syncdir ./syncin/
sending incremental file list
syncdir/
syncdir/dir2/
syncdir/dir2/file1
syncdir/dir2/file2
syncdir/dir2/file3
syncdir/dir3/
syncdir/dir3/file1
syncdir/dir3/file2
syncdir/dir3/file3

sent 453 bytes  received 146 bytes  1.17K bytes/sec
total size is 0  speedup is 0.00
md@nukkie:~/test$ ls -R syncin
syncin:
syncdir

syncin/syncdir:
dir2  dir3

syncin/syncdir/dir2:
file1  file2  file3

syncin/syncdir/dir3:
file1  file2  file3

Last edited by markd81; 07-12-2020 at 07:13 AM. Reason: forgot last ls
 
Old 07-12-2020, 07:41 AM   #3
snorre
LQ Newbie
 
Registered: Jul 2020
Posts: 20

Original Poster
Rep: Reputation: Disabled
Hi mark,

many thanks for your answer!
I have tried your version and noticed that this does not work either!
So I guessed that the problem has to be somewhere else....Hmm

And then I realized, that I have to set the path to the exclude-file absolute,
since the bash-script is processed via a crontab.

Would there a way to omit that, so the cronjob takes the path of the script itself
for all other accesses to files/directories?

So, all I did was changing the constant for the EXCLUDEFILE to

Code:
readonly EXCLUDEFILE='/usr/local/bin/whateverdirectory/exclude.txt'
and it worked like a charm

Also my first version with
Code:
sda1/lost+found
sda1/usr/accounts
sda1/docker/baikal/mysql
sda1/docker/wallabag/mysql
seems to work now...

Thanks,
Mike
 
Old 07-12-2020, 08:53 AM   #4
markd81
LQ Newbie
 
Registered: Jul 2020
Location: Netherlands
Distribution: Debian
Posts: 17

Rep: Reputation: 3
Hi Mike, good to hear. You're welcome!
I assumed the excludefile was in the same directory as the script. Do you mean that you want the script that is being called by cron uses the scripts directory as base directory for the rest of the files? In other words: /usr/local/bin/whateverdirectory/ contains next to the excludefile also the script?

If so you could either cd to that path at the beginning of your script or you can set a variable that will use the path of the script as base like this:
Code:
#!/bin/bash

get_basedir(){
  # basename $0 gets the name of the script
  # realpath gets the absolute file name
  # substitution cuts of the part after the last / so you'll get the path to the script
  local _realpath=$(realpath $(basename $0))
  
  BASEDIR=${_realpath%/*}
}

get_basedir
echo $BASEDIR
 
Old 07-12-2020, 10:21 AM   #5
snorre
LQ Newbie
 
Registered: Jul 2020
Posts: 20

Original Poster
Rep: Reputation: Disabled
Hi Marc,

once again, thanks for your reply!

Yes, the exclude file is indeed in the same directory as the script and yes, the cronjob
should use the same working directory for the rest of the commands.

I copied/pasted your function into my script and it works like a charm - many thanks for that
It's exactly what I was looking for!

Greetings,
Mike
 
Old 07-12-2020, 10:48 AM   #6
snorre
LQ Newbie
 
Registered: Jul 2020
Posts: 20

Original Poster
Rep: Reputation: Disabled
I just noticed, that rsync did not copy all new files with the --delete option from today, could this be?

Code:
Client:
/mnt/sda1/usr/bash/
  -- script1.sh
  -- script2.sh
  -- script3.sh

Server:
backup/12/mnt/sda1/usr/bash/
  -- script1.sh
  -- script2.sh
  -- script3.sh
After that first backup (today) I created a new file locally, named script4.sh and fired up the cronjob to make the backup
Unfortunately, there is no script4.sh on the server?!

This is how rsync gets called in the bash-script:

Code:
 rsync -avR -hh --exclude-from="exclude.txt" --delete /mnt/sda1 /mnt/nfs/backup/12
Many thanks,
Mike

Last edited by snorre; 07-12-2020 at 10:49 AM. Reason: Reformatted the code
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] rsync: exclude more than two files/directories kaz2100 Linux - Software 2 05-18-2016 06:33 AM
exclude hidden files and directories from rsync rayalab Linux - General 1 07-29-2015 12:18 PM
CVS Exclude : Exclude sub directories from check out On Linux from command line shajay12 Linux - Newbie 1 08-03-2009 12:36 AM
How to exclude specific directories from an rsync backup kaplan71 Linux - Software 2 05-16-2008 12:09 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:00 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration