find & rsync script

simon_bagola · 01-13-2009, 02:59 PM

Hi all!

Using Mythbuntu, I need a script, that would make my dream alive...

Since i'm using wireless bridge, connected with my friends NAS, i tried to use it for streaming some movies to my Mythbuntu. Since the wifi connection manages only 500KB/s, i cannot stream some DVD movies etc.

And so... my idea is:
- make a automount folder on my Mythbuntu system
- every night rsync files/folders from that "mount_folder" to "copy_folder"
- BUT, since my "mount_folder" keeps growing (my NAS has some 2,5TB), and my "copy_folder" is only 120GB size, i would need to rsync only the latest files/folders, together 120GB...

Now, as much as i have seen, this is possible using "find" and "rsync" methods, but unfortunately...i have no idea, how to start, where to turn...

Could someone help me with that?!

eco · 01-13-2009, 03:11 PM

Hi,

You can use find to find the files that where created on a specific day, then it can execute an rsync on the folders it found.

You'll need to work on this but it will look like:

Code:

$ find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu

Read the find and rsync man pages to get it just right!

simon_bagola · 01-13-2009, 03:21 PM

Thanks eco!

BUT how to "filter" and rsync only 120GB from all 2TB?

Like said - i need the latest ones...

I know, i'm bugging you, but...

pentode · 01-13-2009, 03:52 PM

That's basically what rsync does. It only updates files that have been changed.

Take a look at rsnapshot - it provides some scripts to automate rsync. It's mainly used for back-ups.

simon_bagola · 01-13-2009, 04:23 PM

Quote:

Originally Posted by pentode

That's basically what rsync does. It only updates files that have been changed.

but, does (and how) scans for new ones, and resyncs the data that way, that in the end there are always only the latest 120GB of whole 2TB data...??

i keep searching for some clues, have to say, i didn't found them till now...

eco · 01-13-2009, 04:30 PM

I'd have one script do 2 things...

The first clears your box of all files older than say 1 week
The other uses the find command to find files less than 7 days old (use -ctime) and rsync them.

Evaluate how many GB/day you generate and adapt the time period accordingly.

something like

Code:

$ find /mnt/Mythbuntu -type f -ctime <days> -iname '*.avi' -exec rm {} \;
$ find /mnt/nas -type f -ctime <days> -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu \;

... or he could add all new movies to a file that you parse with a script.

simon_bagola · 01-13-2009, 04:46 PM

thanks #eco, that could i manage, yes!

and not to make things simpler, but harder - what about the case, these moves are in folders (like dvd's)?

Is the system the same? I persume, i need to change then that '*.avi' part??

chrism01 · 01-13-2009, 05:36 PM

Just FYI

Quote:

Three fields in the inode structure contain the last access, change, and modification times: atime, ctime, and mtime. The atime field is updated each time the pointer to the file's data blocks is followed and the file's data is read. The mtime field is updated each time the file's data changes. The ctime field is updated each time the file's inode changes. The ctime is NOT creation time; there is no way under standard Unix to find a file's creation time.

You might find that mtime is just as good, if not better.

eco · 01-14-2009, 02:19 AM

Quote:

Originally Posted by chrism01

Just FYI

You might find that mtime is just as good, if not better.

Thanks for that... always was a bit of a grey area for me.

I'd recomend your friend keep a text file up to date with say:

date - path
20090114-/foo/bar/LifeOfBrianDVD

then just work of that list.

If you are going to be using find, then you really need to read the man pages but yes, I'd then remove the

Code:

-iname '*.avi'

and change the type to

Code:

-type d

and look at maxdepth.

Best of luck on your project.

simon_bagola · 01-31-2009, 06:21 AM

thanks!

getting further on my (newbie) project...

i want my crontab to exec these commands, so i want to insert following into it:
0 1 * * * find /mnt/Mythbuntu -type d -mtime 550 -exec rm {}
0 2 * * * find /mnt/nas -type d -mtime 550 -exec rsync -av {} /mnt/Mythbuntu

What do you think about that?

Thanks for help once again!

eco · 02-02-2009, 08:15 AM

Quote:

0 1 * * * find /mnt/Mythbuntu -type d -mtime 550 -exec rm {}
0 2 * * * find /mnt/nas -type d -mtime 550 -exec rsync -av {} /mnt/Mythbuntu

Hi,

sorry for the late answer, a bit of work for the mo.

I'd put the above commands into a file and execute the file.
u need an 'r' to remore recursevely.
I think you are missing '\;' at the end of your find.

Code:

find /mnt/Mythbuntu -type d -mtime 550 -exec rm {} \;

You might want a '/' at the end of the rsync command for the destination.

Try something like this:

vi /usr/local/bin/myfile.sh

Code:

#!/bin/bash
find /mnt/Mythbuntu -type d -mtime 550 -exec rm -rf {} \;
find /mnt/nas -type d -mtime 550 -exec rsync -a {} /mnt/Mythbuntu/ \;

crontab -l
0 1 * * * /usr/local/bin/myfile.sh

I'm not sure your script will work though. might want to put an 'echo' in front of the 'rm' to make sure it doesn't remove important files.

Best of luck.

simon_bagola · 02-11-2009, 03:51 PM

vi /usr/local/bin/myfile.sh

Code:

#!/bin/bash
find /mnt/Mythbuntu -type d -mtime 550 -exec rm -rf {} \;
find /mnt/nas -type d -mtime 550 -exec rsync -a {} /mnt/Mythbuntu/ \;

Hi again!

Well... tried that... hmmm, really do not know what is there...

I start the script, ok at the beginning it gives me "FIND" errors, that it cannot find some of the files (generally, these files use long names and some "ä" etc... Think that could be the reason..? And also, these files are not inside some folders, but just files in mounted directory...

Then, it waits for few seconds (like it is searching something or...), and gives no add. info...

like:

Code:

ina@ina-desktop:/var/log/samba$ sudo sh /usr/local/bin/sincro.sh
find: `/videox/08-05-18 - PREMIERE 3 - Das Parfum - Die Geschichte eines M??rders (Perfume_ The Story of a Murderer.ts': No such file or directory
find: `/videox/08-05-20 - PREMIERE 1 - Basilisk - Der Schlangenk??nig (Basilisk_ The Serpent King) - Science-Fictio.ts': No such file or directory
find: `/videox/08-06-03 - PREMIERE 3 - Zoom - Akademie f?\254r Superhelden (Zoom) - Fantasykom??die.ts': No such file or directory
find: `/videox/08-06-10 - PREMIERE 3 - Friends with Money (Friends with Money) - Kom??die.ts': No such file or directory
find: `/videox/08-06-23 - PREMIERE 1 - Grandma_s Boy (Grandma_s Boy) - Kom??die.ts': No such file or directory
find: `/video/Butterfly.on.a.Wheel.2007.LIMITED.CUSTOM.SLOSUB.PAL.DVDR-?uroBits': No such file or directory
ina@ina-desktop:/var/log/samba$

Any ideas?

Oh, and the script is:

Code:

#!/bin/bash
find /videox -type d -mtime 600 -exec rm -rf {} \;
find /video -type d -mtime 600 -exec rsync -a {} /videox/ \;

rehmanlinux · 01-14-2011, 06:00 AM

This is an old thread but it is pretty relevant to what I am trying to do,

This command would copy the files to the local directory,

Code:

find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu

What if I want to copy the files to a remote directory? I have the private/public keys setup already. I can log in without password. Here is what I am trying but it is not working,

Code:

find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av -e ssh {}root@XXX.XXX.XXX.XXX:/mnt/Mythbuntu

Does any one have an idea of what could be wrong?

jv2112 · 01-15-2011, 07:33 AM

2 Cents .........

You may also want to consider adding the --delete option to the rsync command and expanding the "days" so you wont have to manage the space the 160G.