LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-13-2009, 03:59 PM   #1
simon_bagola
LQ Newbie
 
Registered: Jan 2007
Posts: 6

Rep: Reputation: 0
find & rsync script


Hi all!

Using Mythbuntu, I need a script, that would make my dream alive...

Since i'm using wireless bridge, connected with my friends NAS, i tried to use it for streaming some movies to my Mythbuntu. Since the wifi connection manages only 500KB/s, i cannot stream some DVD movies etc.

And so... my idea is:
- make a automount folder on my Mythbuntu system
- every night rsync files/folders from that "mount_folder" to "copy_folder"
- BUT, since my "mount_folder" keeps growing (my NAS has some 2,5TB), and my "copy_folder" is only 120GB size, i would need to rsync only the latest files/folders, together 120GB...

Now, as much as i have seen, this is possible using "find" and "rsync" methods, but unfortunately...i have no idea, how to start, where to turn...

Could someone help me with that?!
 
Old 01-13-2009, 04:11 PM   #2
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Hi,

You can use find to find the files that where created on a specific day, then it can execute an rsync on the folders it found.

You'll need to work on this but it will look like:

Code:
$ find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu
Read the find and rsync man pages to get it just right!
 
Old 01-13-2009, 04:21 PM   #3
simon_bagola
LQ Newbie
 
Registered: Jan 2007
Posts: 6

Original Poster
Rep: Reputation: 0
Thanks eco!

BUT how to "filter" and rsync only 120GB from all 2TB?

Like said - i need the latest ones...

I know, i'm bugging you, but...
 
Old 01-13-2009, 04:52 PM   #4
pentode
Member
 
Registered: Dec 2005
Location: Oregon
Distribution: Debian Testing
Posts: 488

Rep: Reputation: 38
That's basically what rsync does. It only updates files that have been changed.

Take a look at rsnapshot - it provides some scripts to automate rsync. It's mainly used for back-ups.
 
Old 01-13-2009, 05:23 PM   #5
simon_bagola
LQ Newbie
 
Registered: Jan 2007
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pentode View Post
That's basically what rsync does. It only updates files that have been changed.
but, does (and how) scans for new ones, and resyncs the data that way, that in the end there are always only the latest 120GB of whole 2TB data...??

i keep searching for some clues, have to say, i didn't found them till now...
 
Old 01-13-2009, 05:30 PM   #6
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
I'd have one script do 2 things...

The first clears your box of all files older than say 1 week
The other uses the find command to find files less than 7 days old (use -ctime) and rsync them.

Evaluate how many GB/day you generate and adapt the time period accordingly.

something like
Code:
$ find /mnt/Mythbuntu -type f -ctime <days> -iname '*.avi' -exec rm {} \;
$ find /mnt/nas -type f -ctime <days> -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu \;
... or he could add all new movies to a file that you parse with a script.

Last edited by eco; 01-13-2009 at 05:32 PM.
 
Old 01-13-2009, 05:46 PM   #7
simon_bagola
LQ Newbie
 
Registered: Jan 2007
Posts: 6

Original Poster
Rep: Reputation: 0
thanks #eco, that could i manage, yes!

and not to make things simpler, but harder - what about the case, these moves are in folders (like dvd's)?

Is the system the same? I persume, i need to change then that '*.avi' part??
 
Old 01-13-2009, 06:36 PM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
Just FYI
Quote:
Three fields in the inode structure contain the last access, change, and modification times: atime, ctime, and mtime. The atime field is updated each time the pointer to the file's data blocks is followed and the file's data is read. The mtime field is updated each time the file's data changes. The ctime field is updated each time the file's inode changes. The ctime is NOT creation time; there is no way under standard Unix to find a file's creation time.
You might find that mtime is just as good, if not better.
 
Old 01-14-2009, 03:19 AM   #9
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Quote:
Originally Posted by chrism01 View Post
Just FYI

You might find that mtime is just as good, if not better.
Thanks for that... always was a bit of a grey area for me.

I'd recomend your friend keep a text file up to date with say:

date - path
20090114-/foo/bar/LifeOfBrianDVD

then just work of that list.

If you are going to be using find, then you really need to read the man pages but yes, I'd then remove the
Code:
-iname '*.avi'
and change the type to
Code:
-type d
and look at maxdepth.

Best of luck on your project.
 
Old 01-31-2009, 07:21 AM   #10
simon_bagola
LQ Newbie
 
Registered: Jan 2007
Posts: 6

Original Poster
Rep: Reputation: 0
thanks!

getting further on my (newbie) project...

i want my crontab to exec these commands, so i want to insert following into it:
0 1 * * * find /mnt/Mythbuntu -type d -mtime 550 -exec rm {}
0 2 * * * find /mnt/nas -type d -mtime 550 -exec rsync -av {} /mnt/Mythbuntu

What do you think about that?

Thanks for help once again!
 
Old 02-02-2009, 09:15 AM   #11
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Quote:
0 1 * * * find /mnt/Mythbuntu -type d -mtime 550 -exec rm {}
0 2 * * * find /mnt/nas -type d -mtime 550 -exec rsync -av {} /mnt/Mythbuntu
Hi,

sorry for the late answer, a bit of work for the mo.

I'd put the above commands into a file and execute the file.
u need an 'r' to remore recursevely.
I think you are missing '\;' at the end of your find.
Code:
find /mnt/Mythbuntu -type d -mtime 550 -exec rm {} \;
You might want a '/' at the end of the rsync command for the destination.

Try something like this:

vi /usr/local/bin/myfile.sh
Code:
#!/bin/bash
find /mnt/Mythbuntu -type d -mtime 550 -exec rm -rf {} \;
find /mnt/nas -type d -mtime 550 -exec rsync -a {} /mnt/Mythbuntu/ \;
crontab -l
0 1 * * * /usr/local/bin/myfile.sh

I'm not sure your script will work though. might want to put an 'echo' in front of the 'rm' to make sure it doesn't remove important files.

Best of luck.
 
Old 02-11-2009, 04:51 PM   #12
simon_bagola
LQ Newbie
 
Registered: Jan 2007
Posts: 6

Original Poster
Rep: Reputation: 0
vi /usr/local/bin/myfile.sh
Code:
#!/bin/bash
find /mnt/Mythbuntu -type d -mtime 550 -exec rm -rf {} \;
find /mnt/nas -type d -mtime 550 -exec rsync -a {} /mnt/Mythbuntu/ \;
Hi again!

Well... tried that... hmmm, really do not know what is there...

I start the script, ok at the beginning it gives me "FIND" errors, that it cannot find some of the files (generally, these files use long names and some "" etc... Think that could be the reason..? And also, these files are not inside some folders, but just files in mounted directory...

Then, it waits for few seconds (like it is searching something or...), and gives no add. info...

like:
Code:
ina@ina-desktop:/var/log/samba$ sudo sh /usr/local/bin/sincro.sh
find: `/videox/08-05-18 - PREMIERE 3 - Das Parfum - Die Geschichte eines M??rders (Perfume_ The Story of a Murderer.ts': No such file or directory
find: `/videox/08-05-20 - PREMIERE 1 - Basilisk - Der Schlangenk??nig (Basilisk_ The Serpent King) - Science-Fictio.ts': No such file or directory
find: `/videox/08-06-03 - PREMIERE 3 - Zoom - Akademie f?\254r Superhelden (Zoom) - Fantasykom??die.ts': No such file or directory
find: `/videox/08-06-10 - PREMIERE 3 - Friends with Money (Friends with Money) - Kom??die.ts': No such file or directory
find: `/videox/08-06-23 - PREMIERE 1 - Grandma_s Boy (Grandma_s Boy) - Kom??die.ts': No such file or directory
find: `/video/Butterfly.on.a.Wheel.2007.LIMITED.CUSTOM.SLOSUB.PAL.DVDR-?uroBits': No such file or directory
ina@ina-desktop:/var/log/samba$
Any ideas?

Oh, and the script is:
Code:
#!/bin/bash
find /videox -type d -mtime 600 -exec rm -rf {} \;
find /video -type d -mtime 600 -exec rsync -a {} /videox/ \;
 
Old 01-14-2011, 07:00 AM   #13
rehmanlinux
LQ Newbie
 
Registered: Jan 2011
Posts: 1

Rep: Reputation: 0
Find/Rsync Remote Copy

This is an old thread but it is pretty relevant to what I am trying to do,

This command would copy the files to the local directory,


Code:
find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av {} /mnt/Mythbuntu

What if I want to copy the files to a remote directory? I have the private/public keys setup already. I can log in without password. Here is what I am trying but it is not working,

Code:
find /mnt/nas -type f -ctime 1 -iname '*.avi' -exec rsync -av -e ssh {}root@XXX.XXX.XXX.XXX:/mnt/Mythbuntu
Does any one have an idea of what could be wrong?
 
Old 01-15-2011, 08:33 AM   #14
jv2112
Member
 
Registered: Jan 2009
Location: New England
Distribution: Arch Linux
Posts: 719

Rep: Reputation: 103Reputation: 103
2 Cents .........

You may also want to consider adding the --delete option to the rsync command and expanding the "days" so you wont have to manage the space the 160G.

Last edited by jv2112; 01-15-2011 at 08:34 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync config files, cant find em! fyr3 Slackware 1 07-20-2007 03:11 PM
Find & Transfer to FTP Script DjRakso Linux - Newbie 3 03-01-2007 04:44 AM
Is there a script to auto find installed distros & edit Grub clintbrot Linux - Software 3 06-16-2005 11:15 PM
rsync script twantrd Programming 2 01-06-2005 06:46 AM
Rsync Bandwith saver script pixie Linux - General 0 06-03-2004 04:19 AM


All times are GMT -5. The time now is 08:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration