[SOLVED] Trouble making incremental copies/transfers with rsync in cygwin
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
These three lines perform well in that they make a full transfer of the fileserv:/home/ directory on fileserv to the appropriate place on sm-i222 using rsync. I use rsync instead of scp because I have to traverse subdirectories and symbolic links in the /home/... filesystem on fileserv.
What I'm looking to do is use rsync to do an incremental transfer/backup of only the files that have changed since the last full backup. I'll manage the times I do this manually or in crontab.
I've looked on the 'Net for examples and what I've found are examples that are too complex for what I need or too scripted to make sense to me. Sorry.
A colleague says this is do-able, but not how. Rsync.org says this is do-able but not how. Cygwin says this is do-able... see rsync.org.
I believe what I'm looking for is a single rsync line like I have above that only transfers the changed files on fileserv to sm-i222.
Any assistance is greatly appreciated.
If there is any other information I can provide, please let me know.
rsync does incrementals by default, but you're deleting the data from the destination on line 2. If you stopped deleting the data on sm-i222, you'd only transfer what changed from fileserver.
If you want multiple destination trees (/cygdrive/d/fileserv/home, /cygdrive/d/fileserv/home-2011-03-21, /cygdrive/d/fileserv/home-2011-03-20, etc), then I think rsnapshot is what you want.
Well I couldn't try it last night. There is an associated problem. I'll open up a new ticket if you think that's appropriate.
The problem is that the script fails to run from the crontab. That is it starts to run, then dies not even a tenth of the way through the preliminary rsync. As an example, fileserv has about 1 TB I'm trying to rsync once over to sm-i222. Only about 55GB get transferred.
funny thing is, when I run the script either manually or a line at a time, the rsync runs to completion with all the files (1TB) being transferred. As both Cygwin and rsync support point to each other, I'm at a loss.
Thanks and like I said, if this additional problem requires another thread, I understand...
Tried to run it the next night. rsync still crashed. It seems to run fine for about 165GB from the linux box to the Windows server with cygwin. Then rsync seems to crash and lock up.
I don't think I have anything else running except these two app.s & both O/S's. I'm at a loss here. I've checked the BLODA list on Cygwin and found nothing as relates too rsync. I have seen on the web a few references to incompatibility between the two applications. Basically, Cygwin says it's rsync and rsync says it's cygwin.
In this case everyone is pointing fingers at each other and I am left in a quandry.
I assume you are using Samba to export shares on fileserv to Windows 2003? If so you should be able to map these shares to a drive on Win2003. If they are all under /home on RHEL then map /home to drive Z
After that it's a simple matter of syncing Z: to D:
rsync -avz /cygdrive/z/ /cygdrive/d/
No need to ssh into fileserv at all. Needless to say you will need a Samba user on Win2003 with read/execute access to your shares on fileserv, but I presume you have that already?
However a few things. No, I'm not using samba at all. How do I do that within the environment I describe? Sorry, but I'm relatively new to Win2k3 Server side.
Next, how do I create a Samba user as you describe? Again, I must plead ignorance.
Last, I thought I was limited to using cygwin. If I don't need it, I'd be happier.
As a final note, I get no help from my Windows Admin. His response is, *... figure it out yourself ...* I'm trying to.
Thanks for your suggestion! I think we're getting close!
Looks like cygwin uses the Windows event logger for it's logs. I've looked through them and don't see any errors. warnings or informational alerts for rsync. I've looked on both drives I have cygwin map to local windows drives. There are no log directories or log files that I can see.
Can you think of anywhere else I should look for the log files?
It would help me if you could clarify your network topology. What is the Linux RHEL file server for - accessing files from Linux machines or Windows machines? What is the Windows server for? Terminal services? File and print services? Where is the 4 TB RAID - in the Windows server or the RHEL server?
Clarify this and we might be able to make some sense of it before offering a solution.
Let me see if I can clarify. Network topology. Both the hosts are on the same network subnet, so they can see each other. The linux RHEL server is simply a file server for user storage. The Windows server is what is called a *Media Server* in the sense of Symantec's Backup Exec Application. Essentially, I'm using rsync to copy files and subdirectories from fileserv to sm-i222. Then I backup these files onto tape using the Backup Exec application.
The 4TB RAID is attached to the Windows server. I'm using the C: drive(cygwin /cygdrive/c) for the O/S and Application. I'm using the D: drive(cygwin /cygdrive/d) for data.
Does this help claify or do we need more information? I'll be glad to elaborate or fill in any additional details.
The linux RHEL server is simply a file server for user storage.
This is what I'm trying to work out - HOW is RHEL a file server? Through Samba? NFS? SCP? How are users accessing this file server and from what operating systems? And secondly, you say you're "calling a script from the crontab". On what machine are you doing this? It looks as though you're doing this on the Windows machine - calling a script from the crontab on your Windows server?
Sorry to drag this out but the way you have this set up sounds a mess to me. If I have a Windows server and a Linux server on the same network and I want to share files from Linux to Windows I set up Samba on Linux for that purpose. I then map a Samba share on the Linux server - for example /home/smith - to a drive on Windows - for example Drive Z. And when I want to synchronize /home/smith, otherwise known as Drive Z, to a hard disk on my Windows computer - for example Drive D - all I do is get some synchronizing software and do a simple daily sync of Z: to D:
If you are already sharing files from RHEL to Windows then it's quite possible you already have a Samba server running on the RHEL machine.
There are many synchronizing software options on Windows besides Cygwin/rsync.
Sorry if I am completely off track here but how you have done this still perplexes me. There is no need for the asterisk in your rsync command either.
Sorry for the confusion. I'm a little confused myself. To the best of my knowledge, we are not using Samba on the linux machine. I do see that the samba daemon is running on the linux server.
I don't understand why this is important. We're trying to do a file backup/filesync from the native linux rsync command to the cygwin rsync server via this native linux tool.
While scp would be nice, scp doesn't copy down through subdirectories. That's why we're using rsync. It does copy down through subdirectories.
Sorry I've been so long responding. I've been out due to medical reasons.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
