rsync only 60-day old files

tajamari · 08-28-2007, 04:38 AM

I want to rsync only 60-day old files, what are the parameters needed to transfer only those files?

IBall · 08-28-2007, 07:01 AM

As far as I can see, there is no option to rsync that allows this.

Would it be a possibility to use find to locate files that meet your criteria, and then copy them to a temporary location, rsync to the remote location then remove the temporary copy?

--Ian

tajamari · 08-29-2007, 12:00 AM

Quote:

Originally Posted by IBall

As far as I can see, there is no option to rsync that allows this.

Would it be a possibility to use find to locate files that meet your criteria, and then copy them to a temporary location, rsync to the remote location then remove the temporary copy?

--Ian

Ian,

I used find then execute my rsync script, but it does nor recursively transfer the directories and its subdirectories, it transfers the located file on a directory I defined.

tajamari · 08-29-2007, 12:18 AM

Quote:

Originally Posted by tajamari

Ian,

I used find then execute my rsync script, but it does nor recursively transfer the directories and its subdirectories, it transfers the located file on a directory I defined.

Below are my scripts running on the primary data server, my problem is it does not transfer the subfolders under /home/data/accounts/state to /home/data/accounts/state if the backup machine. only files are transfered. any help please.

#FIND SCRIPT
#!/bin/bash
DAYS=60
DIR="/home/data/accounts/state"

date
FIND=`find $DIR -depth -mindepth 1 -mtime +$DAYS -exec sh transfertobackup.sh \; &`
date

#RSYNC SCRIPT - transfertobackup.sh
# /bin/bash

# Transfer all subfolders under /home/data/accounts
#---------------------------------------------------------------

rsync --stats -auvz -e "ssh -p10022" --delete --recursive --times -og /home/data/accounts/state/ testaccount@192.168.10.15:/home/data/accounts/state/

choogendyk · 08-29-2007, 12:03 PM

I'm having a little trouble with your conceptual logic. You find anything that was last modified 60 days or more ago. Then you want to recursively copy stuff below that. But it seems the find would already include things below that. In addition, some of the items in subfolders may have been modified more recently. They could affect the mtime of their subfolder but not higher up folders. So . . .

I've done somewhat similar things using cpio. Mine are incrementals, but you could change it to oldamentals ;-) Anyway, here is a segment of my script:

Code:

case "$DAY" in
Mon)
  # delete previous contents and then do incrementals from Friday
  rm -r ${ADIR}; mkdir ${ADIR};
  find . -mtime -3 | cpio -oa 2>/dev/null | ( cd ${ADIR} && cpio -imd );;
Tue|Wed|Thu)
  # delete previous contents and then do incrementals
  rm -r ${ADIR}; mkdir ${ADIR};
  find . -mtime -1 | cpio -oa 2>/dev/null | ( cd ${ADIR} && cpio -imd );;
Fri)
  # delete previous contents and then do full copies
  rm -r ${ADIR}; mkdir ${ADIR};
  find . | cpio -oa 2>/dev/null | ( cd ${ADIR} && cpio -imd );;
esac

Of course, ADIR is the daily directory inside the archive directory. Then you can rsync from there to a remote system if you want. Or, what I do, have the script just tar up the daily directory, gzip it, and scp it.

Just another alternative.

colucix · 08-29-2007, 01:40 PM

Quote:

Below are my scripts running on the primary data server, my problem is it does not transfer the subfolders under /home/data/accounts/state to /home/data/accounts/state if the backup machine. only files are transfered

This is true because the result from the find command gives filenames only and you give to rsync no information on where to put files other than /home/data/accounts/state.
First, I suggest to create on the remote machine the same directory structure as in the local machine. This can be done by rsync in this way:

Code:

rsync -av --include '*/' --exclude '*'

Second, you can parse the output from the find command to retrieve the directory name (path) and tell rsync to put every single file into its own directory. Using your specifications, I would do something like:

Code:

#!/bin/bash
DAYS=60
DIR=/home/data/accounts/state
cd $DIR
rsync -av --include '*/' --exclude '*' -e "ssh -p10022" . testaccount@192.168.10.15:$DIR
for file in `find . -depth -mindepth 1 -mtime +$DAYS`
do
  backup_dir=$(dirname $file)
  rsync --stats -auvz -e "ssh -p10022" --delete --times -og $file testaccount@192.168.10.15:$DIR/$backup_dir
done

If you don't want to copy the whole directory tree, but only the relevant directiories (that is only the location of the files you want to backup) you can simply create them via ssh, that is something like:

Code:

#!/bin/bash
DAYS=60
DIR=/home/data/accounts/state
cd $DIR
for file in `find . -depth -mindepth 1 -mtime +$DAYS`
do
  backup_dir=$(dirname $file)
  ssh testaccount@192.168.10.15 mkdir -p $DIR/$backup_dir
  rsync --stats -auvz -e "ssh -p10022" --delete --times -og $file testaccount@192.168.10.15:$DIR/$backup_dir
done

Hope this helps. For obvious reasons I have not tested all the options to the find and rsync commands, so - please - double check and test with dummy files!

tajamari · 08-29-2007, 10:09 PM

Quote:

Originally Posted by colucix

This is true because the result from the find command gives filenames only and you give to rsync no information on where to put files other than /home/data/accounts/state.
First, I suggest to create on the remote machine the same directory structure as in the local machine. This can be done by rsync in this way:

Code:

rsync -av --include '*/' --exclude '*'

Second, you can parse the output from the find command to retrieve the directory name (path) and tell rsync to put every single file into its own directory. Using your specifications, I would do something like:

Code:

#!/bin/bash
DAYS=60
DIR=/home/data/accounts/state
cd $DIR
rsync -av --include '*/' --exclude '*' -e "ssh -p10022" . testaccount@192.168.10.15:$DIR
for file in `find . -depth -mindepth 1 -mtime +$DAYS`
do
  backup_dir=$(dirname $file)
  rsync --stats -auvz -e "ssh -p10022" --delete --times -og $file testaccount@192.168.10.15:$DIR/$backup_dir
done

If you don't want to copy the whole directory tree, but only the relevant directiories (that is only the location of the files you want to backup) you can simply create them via ssh, that is something like:

Code:

#!/bin/bash
DAYS=60
DIR=/home/data/accounts/state
cd $DIR
for file in `find . -depth -mindepth 1 -mtime +$DAYS`
do
  backup_dir=$(dirname $file)
  ssh testaccount@192.168.10.15 mkdir -p $DIR/$backup_dir
  rsync --stats -auvz -e "ssh -p10022" --delete --times -og $file testaccount@192.168.10.15:$DIR/$backup_dir
done

Hope this helps. For obvious reasons I have not tested all the options to the find and rsync commands, so - please - double check and test with dummy files!

thanks guys for your help... give you feedback once i got results

enfran · 03-10-2010, 02:50 PM

For those people who are still looking for a way to implement an rsync restricted to files newer/older than a specific date, as I was, I summarize here the 3 simple steps I finally used.

1) find ./ -daystart -mtime +N >myfile
ie: write down everything that is older than N days

2) sed 's|./||' myfile >EXCL_list
ie: strip off the unneeded prefix

3) rsync -Pav --exclude-from=EXCL_list --delete-excluded source/ dest
ie: apply the restricted rsync, as requested

Hope this could be helpful to someone...