LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 03-21-2011, 06:38 PM   #1
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Rep: Reputation: 1
Trouble making incremental copies/transfers with rsync in cygwin


Hello...

As an example, I have two servers, sm-i222 and fileserv.

sm-i222 is a Win2k3 system running cygwin. fileserv is a linux box running RHEL 4.7.

sm-i222 maps /cygdrive/c to the c: drive and /cygdrive/d maps to the d: drive(actually a single 4TB RAID).

from the sm-i222 server /cygdrive/c I call a small script from the crontab. The internal IP for fileserv is 10.0.0.7. See below.

-------------------------

cd /cygdrive/d/fileserv/home
rm -rf /cygdrive/d/fileserv/home/*
rsync -avz -e 'ssh -c blowfish' root@10.0.0.7:/home/* .

-------------------------


These three lines perform well in that they make a full transfer of the fileserv:/home/ directory on fileserv to the appropriate place on sm-i222 using rsync. I use rsync instead of scp because I have to traverse subdirectories and symbolic links in the /home/... filesystem on fileserv.

What I'm looking to do is use rsync to do an incremental transfer/backup of only the files that have changed since the last full backup. I'll manage the times I do this manually or in crontab.

I've looked on the 'Net for examples and what I've found are examples that are too complex for what I need or too scripted to make sense to me. Sorry.

A colleague says this is do-able, but not how. Rsync.org says this is do-able but not how. Cygwin says this is do-able... see rsync.org.

I believe what I'm looking for is a single rsync line like I have above that only transfers the changed files on fileserv to sm-i222.

Any assistance is greatly appreciated.

If there is any other information I can provide, please let me know.

Thanks for your time and consideration.

Blaine
 
Old 03-22-2011, 11:09 AM   #2
AlucardZero
Senior Member
 
Registered: May 2006
Location: USA
Distribution: Debian
Posts: 4,650

Rep: Reputation: 525Reputation: 525Reputation: 525Reputation: 525Reputation: 525Reputation: 525
rsync does incrementals by default, but you're deleting the data from the destination on line 2. If you stopped deleting the data on sm-i222, you'd only transfer what changed from fileserver.

If you want multiple destination trees (/cygdrive/d/fileserv/home, /cygdrive/d/fileserv/home-2011-03-21, /cygdrive/d/fileserv/home-2011-03-20, etc), then I think rsnapshot is what you want.
 
Old 03-22-2011, 05:24 PM   #3
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Trouble making incremental copies/transfers with rsync in cygwin

Thanks, Alucard... I'll try it tonight!
 
Old 03-24-2011, 02:00 PM   #4
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Alucard...

Well I couldn't try it last night. There is an associated problem. I'll open up a new ticket if you think that's appropriate.

The problem is that the script fails to run from the crontab. That is it starts to run, then dies not even a tenth of the way through the preliminary rsync. As an example, fileserv has about 1 TB I'm trying to rsync once over to sm-i222. Only about 55GB get transferred.

funny thing is, when I run the script either manually or a line at a time, the rsync runs to completion with all the files (1TB) being transferred. As both Cygwin and rsync support point to each other, I'm at a loss.

Thanks and like I said, if this additional problem requires another thread, I understand...

Blaine
 
Old 04-07-2011, 06:34 PM   #5
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Tried to run it the next night. rsync still crashed. It seems to run fine for about 165GB from the linux box to the Windows server with cygwin. Then rsync seems to crash and lock up.

I don't think I have anything else running except these two app.s & both O/S's. I'm at a loss here. I've checked the BLODA list on Cygwin and found nothing as relates too rsync. I have seen on the web a few references to incompatibility between the two applications. Basically, Cygwin says it's rsync and rsync says it's cygwin.

In this case everyone is pointing fingers at each other and I am left in a quandry.

Any thoughts from anyone would be appreciated.

Thanks!

Blaine
 
Old 04-08-2011, 01:24 AM   #6
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,311

Rep: Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040
We'll need to see the error msgs to be able to help. There should be some logfiles somewhere and also you can trap with (generic)

prog >prog.log 2>&1
 
Old 04-08-2011, 12:43 PM   #7
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Thanks, chrism01... I'll loog for those logs right away!

Blaine Miller
 
Old 04-08-2011, 02:16 PM   #8
gezley
Member
 
Registered: Sep 2009
Location: Ireland
Distribution: Slackware64, NetBSD
Posts: 533

Rep: Reputation: 247Reputation: 247Reputation: 247
I assume you are using Samba to export shares on fileserv to Windows 2003? If so you should be able to map these shares to a drive on Win2003. If they are all under /home on RHEL then map /home to drive Z

After that it's a simple matter of syncing Z: to D:

rsync -avz /cygdrive/z/ /cygdrive/d/

No need to ssh into fileserv at all. Needless to say you will need a Samba user on Win2003 with read/execute access to your shares on fileserv, but I presume you have that already?
 
Old 04-14-2011, 12:22 PM   #9
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Thanks, gezley, good suggestion...

However a few things. No, I'm not using samba at all. How do I do that within the environment I describe? Sorry, but I'm relatively new to Win2k3 Server side.

Next, how do I create a Samba user as you describe? Again, I must plead ignorance.

Last, I thought I was limited to using cygwin. If I don't need it, I'd be happier.

As a final note, I get no help from my Windows Admin. His response is, *... figure it out yourself ...* I'm trying to.

Thanks for your suggestion! I think we're getting close!

Blaine
 
Old 04-14-2011, 01:54 PM   #10
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
chrism01,

Looks like cygwin uses the Windows event logger for it's logs. I've looked through them and don't see any errors. warnings or informational alerts for rsync. I've looked on both drives I have cygwin map to local windows drives. There are no log directories or log files that I can see.

Can you think of anywhere else I should look for the log files?

Thanks!

Blaine
 
Old 04-21-2011, 03:19 PM   #11
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Hello...

Anyone else have any ideas why rsync is not completing but failing under cygwin? I don't see anything in the error log files for cygwin or rsync.

Yet when I use the cygwin version of rsync, the transfer seems to die after about 20 or 30 GB of data.

Both cygwin and rsync point to the other for the problem, but no one wants to help resolve the issue.

Thanks for your time and consideration...

Blaine
 
Old 04-22-2011, 03:31 PM   #12
gezley
Member
 
Registered: Sep 2009
Location: Ireland
Distribution: Slackware64, NetBSD
Posts: 533

Rep: Reputation: 247Reputation: 247Reputation: 247
It would help me if you could clarify your network topology. What is the Linux RHEL file server for - accessing files from Linux machines or Windows machines? What is the Windows server for? Terminal services? File and print services? Where is the 4 TB RAID - in the Windows server or the RHEL server?
Clarify this and we might be able to make some sense of it before offering a solution.
 
Old 04-25-2011, 01:26 PM   #13
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Good questions...

Let me see if I can clarify. Network topology. Both the hosts are on the same network subnet, so they can see each other. The linux RHEL server is simply a file server for user storage. The Windows server is what is called a *Media Server* in the sense of Symantec's Backup Exec Application. Essentially, I'm using rsync to copy files and subdirectories from fileserv to sm-i222. Then I backup these files onto tape using the Backup Exec application.

The 4TB RAID is attached to the Windows server. I'm using the C: drive(cygwin /cygdrive/c) for the O/S and Application. I'm using the D: drive(cygwin /cygdrive/d) for data.

Does this help claify or do we need more information? I'll be glad to elaborate or fill in any additional details.

Thanks for your time and consideration.

Blaine
 
Old 04-26-2011, 03:17 AM   #14
gezley
Member
 
Registered: Sep 2009
Location: Ireland
Distribution: Slackware64, NetBSD
Posts: 533

Rep: Reputation: 247Reputation: 247Reputation: 247
Quote:
Originally Posted by blainemiller View Post
The linux RHEL server is simply a file server for user storage.
This is what I'm trying to work out - HOW is RHEL a file server? Through Samba? NFS? SCP? How are users accessing this file server and from what operating systems? And secondly, you say you're "calling a script from the crontab". On what machine are you doing this? It looks as though you're doing this on the Windows machine - calling a script from the crontab on your Windows server?
Sorry to drag this out but the way you have this set up sounds a mess to me. If I have a Windows server and a Linux server on the same network and I want to share files from Linux to Windows I set up Samba on Linux for that purpose. I then map a Samba share on the Linux server - for example /home/smith - to a drive on Windows - for example Drive Z. And when I want to synchronize /home/smith, otherwise known as Drive Z, to a hard disk on my Windows computer - for example Drive D - all I do is get some synchronizing software and do a simple daily sync of Z: to D:
If you are already sharing files from RHEL to Windows then it's quite possible you already have a Samba server running on the RHEL machine.
There are many synchronizing software options on Windows besides Cygwin/rsync.
Sorry if I am completely off track here but how you have done this still perplexes me. There is no need for the asterisk in your rsync command either.
 
Old 05-16-2011, 06:10 PM   #15
blainemiller
Member
 
Registered: Dec 2003
Posts: 78

Original Poster
Rep: Reputation: 1
Sorry for the confusion. I'm a little confused myself. To the best of my knowledge, we are not using Samba on the linux machine. I do see that the samba daemon is running on the linux server.

I don't understand why this is important. We're trying to do a file backup/filesync from the native linux rsync command to the cygwin rsync server via this native linux tool.

While scp would be nice, scp doesn't copy down through subdirectories. That's why we're using rsync. It does copy down through subdirectories.

Sorry I've been so long responding. I've been out due to medical reasons.

Thanks!

Blaine
 
  


Reply

Tags
rsync


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync incremental backups tqz Linux - Newbie 9 09-15-2012 04:22 AM
using rsync for incremental backups cccc Linux - Server 5 01-29-2010 07:02 AM
Rsync for incremental backups Meson Linux - General 1 10-30-2007 10:44 AM
rsync incremental backup sachin_keluskar Linux - Software 1 06-29-2005 03:09 AM
Rsync for incremental backups? Phaethar Linux - Software 3 12-04-2003 02:27 PM


All times are GMT -5. The time now is 07:24 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration