[SOLVED] shell script to copy newly changed files with rsync

genderbender · 04-20-2010, 06:04 AM

I've got quite a decent rsync script setup, however I'd like to invoke it whenever there's change to a file. My initial idea was to use find, however this has two major flaws - the first being my particular unix veriant cant understand -print0 which means this doesn't work, the second is that I'm not 100% sure how to put variables into quotation marks so ls can understand the target:

Code:

for i in `find /shares/ -mtime -1 -print`; do ls -ltr $i;done

If anyone has a better idea on how to do this, please tell me

This was just an idea, which in itself is full of flaws - for example, it means files will only be in sync once an hour (I might as well just run an hourly cron job).

This is between two NAS boxes running some 'weird and wonderful' trimmed down unix variant. Thanks

catkin · 04-20-2010, 06:21 AM

Does your 'weird and wonderful' trimmed down UNIX variant provide inotify?

genderbender · 04-20-2010, 07:27 AM

I'm sure I can compile from source

By weird and wonderful I actually mean crap and super trimmed down, hehe.

I'll continue reading about inotify and see if I get any further, thanks.

genderbender · 04-20-2010, 08:04 AM

OK. Any other ideas? This isn't going to compile as amazingly there's no C compiler! If not, think I'll just call it a day and use cron to keep my files up to date. Nice idea though

tommylovell · 04-20-2010, 09:43 AM

How about not using the for loop?

Code:

find /shares/ -mtime -1 -exec ls -ltr {} \;

genderbender · 04-20-2010, 04:25 PM

The ls doesn't work. Any files/folders with spaces in the name ls tries to print as individual files, e.g:

a test file.mp3

the code would print:
ls: a: No such file or directory
ls: test: No such file or directory
ls: file.mp3: No such file or directory

Which wouldn't be hard to get around if print0 worked (it doesn't). Can I stick it in speech marks somehow? Not entirely sure how to encase variables in "". That way the ls would do ls -ltr "a test file.mp3"

grail · 04-20-2010, 09:09 PM

Quote:

The ls doesn't work. Any files/folders with spaces in the name ls tries to print as individual files

This again must be to do with your weird version as I just tried the same code on a directory with files with spaces
and all was fine.

As some type of parsing seems to happen, how about placing the braces in double quotes?

Code:

find /shares/ -mtime -1 -exec ls -ltr "{}" \;

genderbender · 04-20-2010, 09:31 PM

Same response with that code, interestingly - I tried this:

Code:

ls -ltr "`find /shares/ -mtime -1`"

and it gave a response not unlike this:

ls: /shares/a specific file
/shares/a different file
/shares/some other file: No such file or directory

I think it's treating the contents of find as one long file rather than a consecutive list of files, this is why I initially tried looping it. I'm thankful for your help, but as this is difficult for you to test as I have an obscure version of find, I think it would be difficult to advise (feel free to stop helping me

grail · 04-20-2010, 10:06 PM

I think part of our issue is using the ls, may I ask what is it you really want to do with the files you have identified?

catkin · 04-21-2010, 02:15 AM

That's right "it's treating the contents of find as one long string rather than a consecutive list of files". You can get around it by using the shell's read command to read a line at a time. The only robust way to do this is using find's print0 facility but that's only necessary when you have pathological (!) file names including characters such as line end.

This technique relies on your shell supporting "process substitution". Beware there is a space between the "<" characters after the done.

Code:

read -r file ; do
   <whatever you want using $file> 
done < <(find ...)

If this does not work there are variations not using process substitution.

genderbender · 04-21-2010, 04:31 AM

Quote:

Originally Posted by grail

I think part of our issue is using the ls, may I ask what is it you really want to do with the files you have identified?

rsync newly changed files based on their time signature, ideally by the hour (original idea was rsyncing each file the moment it changes, but this has since proved impossible). The files and folders are all samba shares and it's entirely upto the userbase what is contained.

grail · 04-21-2010, 07:19 AM

Yeah, but my question is, as your find already denotes that they are a <day old, what are you going to do with the files?
But then I was going to suggest catkin's option above as it will negate the word splitting.

genderbender · 04-21-2010, 12:41 PM

Well I'll cross that bridge when I come to it, a day is rather a significant ammount of time for new files. On other forums and posts, users have created a new file with the current timestamp (minus an hour) and then done a find with the new file as a comparison, I'm not quite at that point yet but I think once I've got this first step working the rest should be easy. I'm still in two minds as to whether this is a decent option, it maybe easier to run rsync multiple times an hour.

grail · 04-21-2010, 09:45 PM

Try catkin's option as this will give you access to the files you are looking for, but it will be one at a time,
hence the ls option is not of much use as there will only be the one file.

catkin · 04-22-2010, 01:14 AM

Quote:

Originally Posted by grail

Try catkin's option as this will give you access to the files you are looking for, but it will be one at a time,
hence the ls option is not of much use as there will only be the one file.

I'm no rsync-spert; what are the pros and cons of running rsync once for many files and once for each file?