Trouble making incremental copies/transfers with rsync in cygwin
 

Hello...

As an example, I have two servers, sm-i222 and fileserv.

sm-i222 is a Win2k3 system running cygwin. fileserv is a linux box running RHEL 4.7.

sm-i222 maps /cygdrive/c to the c: drive and /cygdrive/d maps to the d: drive(actually a single 4TB RAID).

from the sm-i222 server /cygdrive/c I call a small script from the crontab. The internal IP for fileserv is 10.0.0.7. See below.

-------------------------

cd /cygdrive/d/fileserv/home
rm -rf /cygdrive/d/fileserv/home/*
rsync -avz -e 'ssh -c blowfish' root@10.0.0.7:/home/* .

-------------------------


These three lines perform well in that they make a full transfer of the fileserv:/home/ directory on fileserv to the appropriate place on sm-i222 using rsync. I use rsync instead of scp because I have to traverse subdirectories and symbolic links in the /home/... filesystem on fileserv.

What I'm looking to do is use rsync to do an incremental transfer/backup of only the files that have changed since the last full backup. I'll manage the times I do this manually or in crontab.

I've looked on the 'Net for examples and what I've found are examples that are too complex for what I need or too scripted to make sense to me. Sorry.

A colleague says this is do-able, but not how. Rsync.org says this is do-able but not how. Cygwin says this is do-able... see rsync.org.

I believe what I'm looking for is a single rsync line like I have above that only transfers the changed files on fileserv to sm-i222.

Any assistance is greatly appreciated.

If there is any other information I can provide, please let me know.

Thanks for your time and consideration.

Blaine

rsync does incrementals by default, but you're deleting the data from the destination on line 2. If you stopped deleting the data on sm-i222, you'd only transfer what changed from fileserver.

If you want multiple destination trees (/cygdrive/d/fileserv/home, /cygdrive/d/fileserv/home-2011-03-21, /cygdrive/d/fileserv/home-2011-03-20, etc), then I think rsnapshot is what you want.

Trouble making incremental copies/transfers with rsync in cygwin
 

Thanks, Alucard... I'll try it tonight!

Alucard...

Well I couldn't try it last night. There is an associated problem. I'll open up a new ticket if you think that's appropriate.

The problem is that the script fails to run from the crontab. That is it starts to run, then dies not even a tenth of the way through the preliminary rsync. As an example, fileserv has about 1 TB I'm trying to rsync once over to sm-i222. Only about 55GB get transferred.

funny thing is, when I run the script either manually or a line at a time, the rsync runs to completion with all the files (1TB) being transferred. As both Cygwin and rsync support point to each other, I'm at a loss.

Thanks and like I said, if this additional problem requires another thread, I understand...

Blaine

Tried to run it the next night. rsync still crashed. It seems to run fine for about 165GB from the linux box to the Windows server with cygwin. Then rsync seems to crash and lock up.

I don't think I have anything else running except these two app.s & both O/S's. I'm at a loss here. I've checked the BLODA list on Cygwin and found nothing as relates too rsync. I have seen on the web a few references to incompatibility between the two applications. Basically, Cygwin says it's rsync and rsync says it's cygwin.

In this case everyone is pointing fingers at each other and I am left in a quandry.

Any thoughts from anyone would be appreciated.

Thanks!

Blaine

We'll need to see the error msgs to be able to help. There should be some logfiles somewhere and also you can trap with (generic)

prog >prog.log 2>&1

Thanks, chrism01... I'll loog for those logs right away!

Blaine Miller

I assume you are using Samba to export shares on fileserv to Windows 2003? If so you should be able to map these shares to a drive on Win2003. If they are all under /home on RHEL then map /home to drive Z

After that it's a simple matter of syncing Z: to D:

rsync -avz /cygdrive/z/ /cygdrive/d/

No need to ssh into fileserv at all. Needless to say you will need a Samba user on Win2003 with read/execute access to your shares on fileserv, but I presume you have that already?

Thanks, gezley, good suggestion...

However a few things. No, I'm not using samba at all. How do I do that within the environment I describe? Sorry, but I'm relatively new to Win2k3 Server side.

Next, how do I create a Samba user as you describe? Again, I must plead ignorance.

Last, I thought I was limited to using cygwin. If I don't need it, I'd be happier.

As a final note, I get no help from my Windows Admin. His response is, *... figure it out yourself ...* I'm trying to.

Thanks for your suggestion! I think we're getting close!

Blaine

chrism01,

Looks like cygwin uses the Windows event logger for it's logs. I've looked through them and don't see any errors. warnings or informational alerts for rsync. I've looked on both drives I have cygwin map to local windows drives. There are no log directories or log files that I can see.

Can you think of anywhere else I should look for the log files?

Thanks!

Blaine

Hello...

Anyone else have any ideas why rsync is not completing but failing under cygwin? I don't see anything in the error log files for cygwin or rsync.

Yet when I use the cygwin version of rsync, the transfer seems to die after about 20 or 30 GB of data.

Both cygwin and rsync point to the other for the problem, but no one wants to help resolve the issue.

Thanks for your time and consideration...

Blaine

It would help me if you could clarify your network topology. What is the Linux RHEL file server for - accessing files from Linux machines or Windows machines? What is the Windows server for? Terminal services? File and print services? Where is the 4 TB RAID - in the Windows server or the RHEL server?
Clarify this and we might be able to make some sense of it before offering a solution.

Good questions...

Let me see if I can clarify. Network topology. Both the hosts are on the same network subnet, so they can see each other. The linux RHEL server is simply a file server for user storage. The Windows server is what is called a *Media Server* in the sense of Symantec's Backup Exec Application. Essentially, I'm using rsync to copy files and subdirectories from fileserv to sm-i222. Then I backup these files onto tape using the Backup Exec application.

The 4TB RAID is attached to the Windows server. I'm using the C: drive(cygwin /cygdrive/c) for the O/S and Application. I'm using the D: drive(cygwin /cygdrive/d) for data.

Does this help claify or do we need more information? I'll be glad to elaborate or fill in any additional details.

Thanks for your time and consideration.

Blaine

This is what I'm trying to work out - HOW is RHEL a file server? Through Samba? NFS? SCP? How are users accessing this file server and from what operating systems? And secondly, you say you're "calling a script from the crontab". On what machine are you doing this? It looks as though you're doing this on the Windows machine - calling a script from the crontab on your Windows server?
Sorry to drag this out but the way you have this set up sounds a mess to me. If I have a Windows server and a Linux server on the same network and I want to share files from Linux to Windows I set up Samba on Linux for that purpose. I then map a Samba share on the Linux server - for example /home/smith - to a drive on Windows - for example Drive Z. And when I want to synchronize /home/smith, otherwise known as Drive Z, to a hard disk on my Windows computer - for example Drive D - all I do is get some synchronizing software and do a simple daily sync of Z: to D:
If you are already sharing files from RHEL to Windows then it's quite possible you already have a Samba server running on the RHEL machine.
There are many synchronizing software options on Windows besides Cygwin/rsync.
Sorry if I am completely off track here but how you have done this still perplexes me. There is no need for the asterisk in your rsync command either.

Sorry for the confusion. I'm a little confused myself. To the best of my knowledge, we are not using Samba on the linux machine. I do see that the samba daemon is running on the linux server.

I don't understand why this is important. We're trying to do a file backup/filesync from the native linux rsync command to the cygwin rsync server via this native linux tool.

While scp would be nice, scp doesn't copy down through subdirectories. That's why we're using rsync. It does copy down through subdirectories.

Sorry I've been so long responding. I've been out due to medical reasons.

Thanks!

Blaine