LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-09-2016, 02:51 PM   #1
linux4evr5581
Member
 
Registered: Sep 2016
Location: USA
Posts: 275

Rep: Reputation: Disabled
Finding modified files


Hey so I found out that when using rsync subsequently that it only copies files that have been changed. I was wondering can I use this feature of rsync to find files that have changed, and then using diff or sdiff against my saved backup to see exactly what has changed?

Last edited by linux4evr5581; 10-09-2016 at 02:53 PM.
 
Old 10-09-2016, 03:42 PM   #2
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,477
Blog Entries: 3

Rep: Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234
"rsync" has a --dry-run option which will perform a trial run without making changes. When combined with the --itemize-changes option it will show you which files have changed. The paths will be relative to the source directory. "sed" is needed to clean up that output a little though.

I guess from that you could calculate the destination files and then pass both to "diff"

How far can you get with that description?

(I'm assuming different machines for source and destination. If both are on the same machine, then "diff" alone is enough.)
 
Old 10-09-2016, 04:56 PM   #3
linux4evr5581
Member
 
Registered: Sep 2016
Location: USA
Posts: 275

Original Poster
Rep: Reputation: Disabled
I apologize im not quite understanding, are you implying what I said can work but you're just giving me a more efficient way? I do want to try this out on the same machine for now, but I guess what you told me is for doing this across machines? I got lost at "the paths will be relative to the source directory" (what paths?) I know the basic format of this command to make backups is,
rsync -v -a sourcePath destinationPath.. So i'm not sure why I need to any calculate destination files either...

Last edited by linux4evr5581; 10-09-2016 at 05:10 PM.
 
Old 10-09-2016, 10:45 PM   #4
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,477
Blog Entries: 3

Rep: Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234
If both sets of files are on the same machine you can just use "diff -r"

If one set is on another machine, then you'll need to list the changed files using "rsync" and then use "diff" Here's a guess which is dependent on "bash" because of process substitution

Code:
for i in $(rsync --dry-run -ai ./X/ machine2:/Y/ | sed -ne '/^>/{ s/^[^ ]* //;p }'); 
do 
        diff ./X/$i <(ssh machine2 cat /Y/$i);
done;
For that to work, you'll need SSH keys. And I've left out SSH multiplexing, which would make it far faster.
 
Old 10-10-2016, 12:53 AM   #5
linux4evr5581
Member
 
Registered: Sep 2016
Location: USA
Posts: 275

Original Poster
Rep: Reputation: Disabled
Ok I think I understand, so what you're saying is that rsync is an uneeded command when doing this sort of thing localy, as diff is all you need... But across computers ofcourse you need to be "rsync'd" in order for this to work. Got it. So I guess the code you're showing is a mini script of you ssh'ing into machine2 to "update" the changed files to your machine using rsync -n, kinda like apt-get update (ofcourse not actually updating but like listing the files) and then piping those files through sed and diff...

Last edited by linux4evr5581; 10-10-2016 at 01:00 AM.
 
Old 10-10-2016, 01:24 AM   #6
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,477
Blog Entries: 3

Rep: Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234Reputation: 2234
Quote:
Originally Posted by linux4evr5581 View Post
Ok I think I understand, so what you're saying is that rsync is an uneeded command when doing this sort of thing localy, as diff is all you need... But across computers ofcourse you need to be "rsync'd" in order for this to work. Got it.
Yep. Using "diff" is enough for a local comparison, if you add the -r option.

Quote:
Originally Posted by linux4evr5581 View Post
So I guess the code you're showing is a mini script of you ssh'ing into machine2 to "update" the changed files to your machine using rsync -n, kinda like apt-get update (ofcourse not actually updating but like listing the files) and then piping those files through sed and diff...
Yep. Because "rsync" uses "ssh" under the hood to communicate with the other machine we can re-use it to compare the files using "diff". (Re-use would be far faster and more efficient with multiplexing, but I'm leaving that out. For that you'd start an SSH connection as master connection and leave that in the background, then for the looped connections, you'd attach to that. See the options -M and -S for the ssh(1) client or else ControlMaster and ControlPath in ssh_config(5) for more on multiplexing. )

There's just one pipe though. The pipe would be to use "sed" to trim out any lines without file names and to trim the non-file name data from the lines that do have file names.

Compare the output of these two:

Code:
rsync --dry-run -ai ./X/ machine2:/Y/;
 
rsync --dry-run -ai ./X/ machine2:/Y/ | sed -ne '/^>/{ s/^[^ ]* //;p }';
The -n makes "sed" hide the lines by default. The /^>/ says to find lines beginning with a greater than sign and do what's in the curly braces on those lines. The s/// is a substitution and the p is to print out the result in spite of the -n.

And these two:

Code:
cat /Y/foo;

diff ./X/foo <(ssh machine2 cat /Y/foo);
The first should give a file-not-found type of error. The second should give a diff comparing the local file that is in ./X/ and the remote file that is on machine2 in the directory /Y/ Obviously those are made up values, but you can substitute something relevant.
 
1 members found this post helpful.
Old 10-10-2016, 02:01 AM   #7
linux4evr5581
Member
 
Registered: Sep 2016
Location: USA
Posts: 275

Original Poster
Rep: Reputation: Disabled
Awesome I had no idea rsync had a built in ssh, thats good to know. Ill look into multiplexing, sounds like it's faster cause it's a pre-established connection.. Cool thanks for the tips dude, ill try out those commands later, they looked fairly complicated at first but their starting to make sense. Thanks man!
 
Old 10-10-2016, 02:33 AM   #8
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,626

Rep: Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370Reputation: 4370
just a comment: ssh is not built into rsync, but rsync is able to use ssh (if installed and configured properly) to do its task. Otherwise rsync needs rsync on the "other" side too and rsync has its own protocol and can work without ssh too.
 
Old 10-10-2016, 02:38 AM   #9
linux4evr5581
Member
 
Registered: Sep 2016
Location: USA
Posts: 275

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
just a comment: ssh is not built into rsync, but rsync is able to use ssh (if installed and configured properly) to do its task. Otherwise rsync needs rsync on the "other" side too and rsync has its own protocol and can work without ssh too.
Thanks for clearing that up!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Finding Files Modified in the Past Few Days LXer Syndicated Linux News 0 08-16-2013 02:00 AM
[SOLVED] finding modified files by time mahmoodn Ubuntu 4 12-22-2010 05:40 AM
Finding files modified n minutes ago tucs_123 AIX 5 10-09-2009 01:59 AM
Is there any other way of finding out if my file is just modified or is newly created deepti Linux - General 3 01-07-2009 11:35 PM
gnu make recompiles the source files fully even though the files are not modified yasothamani Linux - Software 4 02-07-2007 07:36 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:55 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration