LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 04-20-2010, 06:04 AM   #1
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Rep: Reputation: 31
shell script to copy newly changed files with rsync


I've got quite a decent rsync script setup, however I'd like to invoke it whenever there's change to a file. My initial idea was to use find, however this has two major flaws - the first being my particular unix veriant cant understand -print0 which means this doesn't work, the second is that I'm not 100% sure how to put variables into quotation marks so ls can understand the target:

Code:
for i in `find /shares/ -mtime -1 -print`; do ls -ltr $i;done
If anyone has a better idea on how to do this, please tell me This was just an idea, which in itself is full of flaws - for example, it means files will only be in sync once an hour (I might as well just run an hourly cron job).

This is between two NAS boxes running some 'weird and wonderful' trimmed down unix variant. Thanks
 
Old 04-20-2010, 06:21 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,541
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Does your 'weird and wonderful' trimmed down UNIX variant provide inotify?
 
Old 04-20-2010, 07:27 AM   #3
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Original Poster
Rep: Reputation: 31
I'm sure I can compile from source By weird and wonderful I actually mean crap and super trimmed down, hehe.

I'll continue reading about inotify and see if I get any further, thanks.
 
Old 04-20-2010, 08:04 AM   #4
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Original Poster
Rep: Reputation: 31
OK. Any other ideas? This isn't going to compile as amazingly there's no C compiler! If not, think I'll just call it a day and use cron to keep my files up to date. Nice idea though
 
Old 04-20-2010, 09:43 AM   #5
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
How about not using the for loop?

Code:
find /shares/ -mtime -1 -exec ls -ltr {} \;
 
Old 04-20-2010, 04:25 PM   #6
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Original Poster
Rep: Reputation: 31
The ls doesn't work. Any files/folders with spaces in the name ls tries to print as individual files, e.g:

a test file.mp3

the code would print:
ls: a: No such file or directory
ls: test: No such file or directory
ls: file.mp3: No such file or directory

Which wouldn't be hard to get around if print0 worked (it doesn't). Can I stick it in speech marks somehow? Not entirely sure how to encase variables in "". That way the ls would do ls -ltr "a test file.mp3"

Last edited by genderbender; 04-20-2010 at 04:42 PM.
 
Old 04-20-2010, 09:09 PM   #7
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,492

Rep: Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891
Quote:
The ls doesn't work. Any files/folders with spaces in the name ls tries to print as individual files
This again must be to do with your weird version as I just tried the same code on a directory with files with spaces
and all was fine.

As some type of parsing seems to happen, how about placing the braces in double quotes?
Code:
find /shares/ -mtime -1 -exec ls -ltr "{}" \;
 
Old 04-20-2010, 09:31 PM   #8
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Original Poster
Rep: Reputation: 31
Same response with that code, interestingly - I tried this:
Code:
ls -ltr "`find /shares/ -mtime -1`"
and it gave a response not unlike this:

ls: /shares/a specific file
/shares/a different file
/shares/some other file: No such file or directory

I think it's treating the contents of find as one long file rather than a consecutive list of files, this is why I initially tried looping it. I'm thankful for your help, but as this is difficult for you to test as I have an obscure version of find, I think it would be difficult to advise (feel free to stop helping me

Last edited by genderbender; 04-20-2010 at 09:40 PM.
 
Old 04-20-2010, 10:06 PM   #9
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,492

Rep: Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891
I think part of our issue is using the ls, may I ask what is it you really want to do with the files you have identified?
 
Old 04-21-2010, 02:15 AM   #10
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,541
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
That's right "it's treating the contents of find as one long string rather than a consecutive list of files". You can get around it by using the shell's read command to read a line at a time. The only robust way to do this is using find's print0 facility but that's only necessary when you have pathological (!) file names including characters such as line end.

This technique relies on your shell supporting "process substitution". Beware there is a space between the "<" characters after the done.
Code:
read -r file ; do
   <whatever you want using $file> 
done < <(find ...)
If this does not work there are variations not using process substitution.
 
Old 04-21-2010, 04:31 AM   #11
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Original Poster
Rep: Reputation: 31
Quote:
Originally Posted by grail View Post
I think part of our issue is using the ls, may I ask what is it you really want to do with the files you have identified?
rsync newly changed files based on their time signature, ideally by the hour (original idea was rsyncing each file the moment it changes, but this has since proved impossible). The files and folders are all samba shares and it's entirely upto the userbase what is contained.

Last edited by genderbender; 04-21-2010 at 04:32 AM.
 
Old 04-21-2010, 07:19 AM   #12
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,492

Rep: Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891
Yeah, but my question is, as your find already denotes that they are a <day old, what are you going to do with the files?
But then I was going to suggest catkin's option above as it will negate the word splitting.
 
Old 04-21-2010, 12:41 PM   #13
genderbender
Member
 
Registered: Jan 2005
Location: US
Distribution: Centos, Ubuntu, Solaris, Redhat
Posts: 396

Original Poster
Rep: Reputation: 31
Well I'll cross that bridge when I come to it, a day is rather a significant ammount of time for new files. On other forums and posts, users have created a new file with the current timestamp (minus an hour) and then done a find with the new file as a comparison, I'm not quite at that point yet but I think once I've got this first step working the rest should be easy. I'm still in two minds as to whether this is a decent option, it maybe easier to run rsync multiple times an hour.
 
Old 04-21-2010, 09:45 PM   #14
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,492

Rep: Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891Reputation: 1891
Try catkin's option as this will give you access to the files you are looking for, but it will be one at a time,
hence the ls option is not of much use as there will only be the one file.
 
Old 04-22-2010, 01:14 AM   #15
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,541
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by grail View Post
Try catkin's option as this will give you access to the files you are looking for, but it will be one at a time,
hence the ls option is not of much use as there will only be the one file.
I'm no rsync-spert; what are the pros and cons of running rsync once for many files and once for each file?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Files seem to take up more space in destination after rsync copy Karderio Linux - Newbie 5 02-24-2012 08:30 PM
SHELL SCRIPT to copy files from cdrom begin_learn Linux - Newbie 2 06-10-2009 08:10 PM
Shell Script to copy and number files extasic Programming 1 12-12-2008 06:29 AM
Copying whole changed files - rsync? deadlock Linux - Software 2 12-04-2007 04:57 AM


All times are GMT -5. The time now is 07:57 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration