LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-29-2015, 08:11 AM   #1
MikeyCarter
Member
 
Registered: Feb 2003
Location: Orangeville
Distribution: Fedora
Posts: 492

Rep: Reputation: 31
Question Sync (rsync?) that links to latest files rather than copying them.


I've got several (let's say 4 for the sake of this post) NFS mounts which contain various stages and save points of the same files.

*please note I'm over simplifying my example for the sake of this post*

For my build I want to create a blending of the 4 NFS mounts using the latest files from each into a staging area. However, I don't want to copy the files.

So the end result.

code.c -> /net/server1/code.c
code.h -> /net/server4/code.h
index.html -> /net/server2/index.html
index.php -> /net/server2/index.html
somefolder/index.html -> /net/server3/somefolder/index.html

The reason I don't want to copy the files is the 4 mount points are +1T in size... where the stage area is only a few 100M. If I symlink I can build the stage area the way I need it without using up any space.
 
Old 05-29-2015, 11:24 AM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 17,545

Rep: Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608
I don't think you'll manage that without a 'find' expression to locate files after a certain date piped to ln -sf to link them. I wouldn't dream of attempting the arguments without the exact situation. This presumes that any newest file can appear on any server. If you can narrow that down, you will be able to save a lot of disk thrashing which is gloriously inefficient over nfs.
 
Old 05-29-2015, 11:44 AM   #3
MikeyCarter
Member
 
Registered: Feb 2003
Location: Orangeville
Distribution: Fedora
Posts: 492

Original Poster
Rep: Reputation: 31
The exact scenario. One NFS is the master in one datacentre. Then there are at least 4 other data centres, with up to 10 labels each (snapshots of the code at points in time), sync'd from a DB.

When building the DB copy it looks at a stage area. but the master copy is in one city and the closest NFS mount is on one of the other 4. So when building the stage area makes sense to copy as much from the local datacentre as possible, then copy the missing from the master copy.

Problem is we don't know at the time of the script which data centre is the local one, and which of the 10 labels contains the most up to-date file. Hence the hunt. The resent problem is were running out of space on the stage area so the rsync currently in place is falling over.

So the idea is to symlink them to the NFS mount with the latest file... which will be the local datacentre, latest label, and any missing end up pulling from the master.

I'm going to do this by script at the moment but wanted to see if there was any way to get rsync (or other product) to sync as symbolic links to the original location rather than copy.
 
Old 05-29-2015, 02:00 PM   #4
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 17,545

Rep: Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608Reputation: 2608
Sounds like a crazy way to do software development. Has git been considered? I thought that was what it was designed for.
 
Old 05-29-2015, 03:11 PM   #5
MikeyCarter
Member
 
Registered: Feb 2003
Location: Orangeville
Distribution: Fedora
Posts: 492

Original Poster
Rep: Reputation: 31
It's not for designing the software its for distributing the compile objects (or in my case more like the third-party libraries needed to compile the objects and other software).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Rsync not copying all files barbz127 Linux - Newbie 2 08-04-2013 07:56 PM
Copying latest 10 files from one folder to another kingjaffi Linux - Newbie 5 11-20-2010 01:09 AM
rsync - sync files modified within the last 60 days boomklever Linux - Networking 3 06-03-2009 09:41 AM
Copying Latest Files to Another Folder nformosa Programming 9 04-23-2009 06:40 AM
Copying whole changed files - rsync? deadlock Linux - Software 2 12-04-2007 04:57 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 01:53 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration