LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 06-30-2010, 10:23 AM   #1
junglepunk
Member
 
Registered: Jun 2010
Posts: 41

Rep: Reputation: 15
synchronize two folders?


I have made a backup of my files on a remote server and I'd like to maintain that backup using rsync. The problem is that the timestamps don't match perfectly between the source and the backup.

What can I do? I'd rather not replace all the files in my backup because there is so much data it would take a very very long time.

Is there perhaps a way to compare checksums and then update the timestamps?

Both are low power boxes with no GUI and only BusyBox CLI access.
 
Old 06-30-2010, 11:01 AM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,953
Blog Entries: 5

Rep: Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755
You should set up Network Time Protocol (NTP) on both boxes so that they have the same UTC time. Even if in different time zones this will make sure the times recorded on files are correct.

rsync should be seeing the files as the same if they are the same - have you used some flag that makes it change access, creation or modification time on the target? The main point of rsync is specifically to avoid having to re-transfer files that have already been transferred so if you're seeing an issue with re-transfers it seems likely you're telling it to do so. (There are flags to say to re-transfer even if it is on the target.) Review the man page (type "man rsync") and be sure you understand what the flags you're using are doing.
 
Old 06-30-2010, 11:26 AM   #3
junglepunk
Member
 
Registered: Jun 2010
Posts: 41

Original Poster
Rep: Reputation: 15
Sorry, I should probably have included this but, the initial transfer was not done using rsync. It was done using wget and the -m switch running wget in mirroring mode, which should have preserved the timestamps. Apparently timestamps aren't preserved.

Both boxes do synchronize to an ntp server so the time should be the same.
 
Old 06-30-2010, 11:28 AM   #4
junglepunk
Member
 
Registered: Jun 2010
Posts: 41

Original Poster
Rep: Reputation: 15
Quote:
There are two ways to correct.

For the first run after the time change, you can run rsync with the --checksum option in order to ensure that only files that have a changed checksum get transferred, and update the modified time on all unchanged files.
Source: http://samba.anu.edu.au/rsync/daylight-savings.html

I've been googling and as you can see from the quote I found there seems to be a way to get rsync to update timestamps anfter it has checked checksums. But the source I have found doesn't explain how. Does anyone know how this can be done?
 
Old 06-30-2010, 11:55 AM   #5
junglepunk
Member
 
Registered: Jun 2010
Posts: 41

Original Poster
Rep: Reputation: 15
I'm not entirely sure what's going on but there is a "-c" switch for rsync that will check checksums. I tried running rsync with that option enabled but apparently my boxes are so week that checking checksums takes as long as transfering the file does. Which means that this is a hopeless way of doing it. Any other suggestions?
 
Old 06-30-2010, 12:13 PM   #6
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,953
Blog Entries: 5

Rep: Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755
If it were me I'd bite the bullet by removing the files done with wget then using rsync to re-transfer. It will send the files again but then the next time you run rsync after that it will only find the "real" changes.
 
Old 06-30-2010, 12:50 PM   #7
junglepunk
Member
 
Registered: Jun 2010
Posts: 41

Original Poster
Rep: Reputation: 15
Yeah, it looks like that's the only way to do it. Damn shame. Took me a week to transfer the files with wget. Rsync is even slower. Oh well . . .
 
Old 06-30-2010, 01:05 PM   #8
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,953
Blog Entries: 5

Rep: Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755
rsync can be sped up by using compression (it does by default) if you have limited bandwidth (of course you CPU takes a hit).

Also if this is something internal to your LAN/WAN and not via internet you can tell rsync to use the old rsh instead of ssh. That will speed it up some because then it won't have to encrypt the connection. At a prior job we regularly would enable rsh to do large rsyncs then disable it when done. (You do NOT want to leave rsh enabled because it is a security risk but for a controlled window of time you can probably get away with it.)
 
Old 06-30-2010, 01:40 PM   #9
junglepunk
Member
 
Registered: Jun 2010
Posts: 41

Original Poster
Rep: Reputation: 15
This is a LAN, and the LAN speed is not really the problem, CPU is however quite limited. I don't think compression would speed things up, rather it would probably slow things down.

Do you have any good resources on rsh for reading through? I've never hared of it.
 
Old 06-30-2010, 01:52 PM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,017
Blog Entries: 54

Rep: Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764Reputation: 2764
Quote:
Originally Posted by junglepunk View Post
Took me a week to transfer the files with wget.
In that case wouldn't it be easier to move the files in the expected tree layout first? After that you could do a recursive rsync dry run with timestamp checking disabled (--update, --size-only, --checksum) to see if at least the structure is OK. Also with rsync you can filter things so you can choose to update half of the 15 thousand small files and leave the 2TB one for later.
 
Old 06-30-2010, 02:07 PM   #11
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,953
Blog Entries: 5

Rep: Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755Reputation: 755
It's been awhile since I used rsh and that wasn't on Linux. However, on checking my CentOS5 system I see there is a "kshell" file in /etc/xinetd.d that runs "kshd" which is:
Quote:
"# description: The kerberized rshell server accepts rshell commands authenticated and encrypted with Kerberos 5."
The rsh command and the kshd are both part of the package:
krb5-workstation

That package contains man pages for rsh and kshd. You can access them by typing:
man rsh
man kshd

The man page for kshd shows a -e to force encryption and the man page for rsh shows a -x to force encryption. It doesn't say whether or not it encrypts by default if both sides are capable of it. (The UNIX rsh didn't allow for encryption at all.) The package for kshell may have been installed by the xinetd package itself as it does bring in some basic services like that but disables them by default.

Also I misspoke earlier when I said rsync encrypts by default. Typically I use "-e ssh" to force it to use ssh for the transport (as opposed to running an rsync daemon) and it is ssh that encrypts by default not rsync itself. Therefore you might try running the rsync daemon on the target and NOT use the "-e ssh" option and may achieve the same effect. I haven't tried that myself to see if it is any faster but since rsync's man page talks about using ssh for encryption it suggests it doesn't do encryption if the daemon is used instead.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script to create folders including making recursive folders.... linux-bandit Linux - Software 8 11-28-2009 01:50 AM
find -exec command to recursively delete files and folders in folders with X name Joan Murt Linux - Newbie 2 07-08-2009 04:35 PM
change permissions of all folders and sub folders cad Linux - General 3 01-28-2007 10:21 AM
Samba can create new files and folders but access denied in any new folders k.king Linux - Networking 2 01-15-2006 06:14 AM
Synchronize folders with different distro mayank_a Linux - Software 4 10-10-2005 04:46 AM


All times are GMT -5. The time now is 03:20 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration