When installting a new multi-petabyte storage array, how can all the data on the old array be completed within 4 hour maintenance windows?

Using rsync will take weeks and by the time it is done, there will be a lot of changes. Running rsync again will take days, assuming that there are no changes and longer if there are changes.

Installing faster hardware is impractical as the old storage array will be scrapped once the new one is fully online.

Altering the actual storage fielsystems is impractical.

The storage systems are different, but both have in common the ability to mount on Linux.

Some thoughts:

1) Use rsync and find a large enough maintenance window (e.g. holiday)

2) Break the fielsystem up into chunks that can be processed in less than 4 hours and spread the task out over many days.

3) Use a utility like DRBD (anyone have experience)?

All comments or suggestions welcome!

First, it CANNOT be done using any of the tools you're mentioning, within four hours, period. Secondly, you give us zero details that would let us even guess at an answer...version/distro of Linux? How this array is connected? Type of array? Anything?

Best answer (assuming this is a SAN), is that you let the SAN hardware do the replication behind the scenes, and it won't matter how long it takes, because the host system won't ever know. After completion, break the mirror, and mount it elsewhere.

I had to do something like this using robocopy and Windows (robocopy is similar to rsync, but more primitive in many respects).

My solution involved taking advantage of the particular pattern of how files are updated in our system. I was able to write a script to find and target just the folders with more recent potential updates. As such robocopy would only need to search a far smaller list of folders for changes - suitable for "topping off" during the day before switch-over.

The same sort of thing could be done with rsync, but it depends on whether there are patterns to exploit. For example, if changes are made by adding/removing/saving files rather than updating them in place, then the date of the folders could be used to filter what rsync searches. This can speed up the process by orders of magnitude.

But typical log files are updated in place with append instead of replacing the file. So, you would need to find all of the folders with log files and make sure to make the script rsync those folders even if the parent folder's date-time isn't recent. Similarly, database files are updated in place.

The question is partially academic. Some years ago, a friend in IT was faced with upgrading an old EMC storage array, with more than a petabyte of data, to a new array from another manufacturer. The two storage system were incompatable, but both would mount on a linux box.

The task was completed with rsync. He divided the array by directory tree that was small enouhg for rsync to verify in a couple hours. An rsync would run all day, then during the maintenance window it would be run again to catch any changes during the day. Symbolic links were used to redirect traffic from the old array to the new one. When all of the data have been coherently moved, the new array was placed online. It was tedious...

I propsoed to use a metaserver to proxy traffic to the old and new storage systems. All traffic would go through the proxy server to the old storage and would update the new storage system. This would continue until the new server was in sync, then the new storage array could be placed online in a maintenance window of few minues.

The latter technique has been implemented and tested. It has no practica limits, beyond those of linux.

The real point of the question is to see if the solution has value.

Well, if you're going to allow other tools to be used, then a couple things that come to mind are GlusterFS and AUFS. GlusterFS would be a more forward thinking solution that would make future migrations easier. AUFS is more of a hack.

The basic idea with GlusterFS would be to transition from using a simple nfs share into using a GlusterFS mirrored share. Kind of like a transition from a normal partition into a RAID1, but across two different computers rather than across two drives in a single computer. You could use rsync to get the "mirror" somewhat close first. The point is - once it's set up, it will be easier to transition in the future to a different array.

The basic idea with AUFS is to transition from using a simple nfs share to an AUFS union of the existing share (read only) plus the new share (read-write). Once this switch is made, new writes occur only to the new share. You can then incrementally migrate by using rsync from the AUFS union onto the new share. In this case, the new share starts off empty. You choose some down time to convert all relevant /etc/fstab from nfs to aufs. From then on, all new writes will be to the new share.

AUFS has been sidelined and developmnet/support are at an end.

OverlayFS seems to still be supported. But it is fairly ocmplicated and will require proxy servers; because the storage arrays are closed systems.

Yes, Gluster or CEPH could coherently replicate across two proprietary storage systems of a goodly size.
But it would require a lot of tailoring and intermediate servers to haldle the distributed filesystem stuff.

AUFS is in Debian Stable; OverlayFS is not. You still haven't said anything about what linux distribution(s) you are using, or even what sort of file sharing is involved, so I didn't make any assumptions. AUFS is in pretty much anything vaguely current, so that's why I mentioned it.

The initial project was a while ago. I was not directly involved, so do not have particulars, beyond what I shared.
The old array was proprietary EMC had more than a petabyte of data and had only Gbe ports.
The new was larger, faster and also proprietary.

Of course there was no backup of the data, so errors could be disasterous.

I recall that the data was in large part web stuff.
So, lots of relatively small web pages and some relatively large logs and database.

There was another survelance video storage project; mostly relatively large files.
Ultimately, they were OK with loosing the data during the cut-over, so no copy was needed.

The initial intent was to create a coherent way to mirror very large active dataset between different Linux filesystems.

The problem of coherently moving a large data set seemed a common enough problem and likey to continue and expand.

I coded a mirror solution, tested it with many millions of files and it performed as expected.

Lacking a couple of multi-petabyte arrays, testing has been limited in terms of very large datasets.

If the mirror is intersting enough, I intend to create a very large test dataset (likely beyond the yotabyte range) to perform testing on.

As for now, I've ported the mirror code to Debian and RedHat, with no problems, complications or limitations.

My concern and subject of the post is to see what comprable solutions are avaialble and what common practice is.

You know, you could have stated what sort of feedback you were hoping to get, and why, in the first place.

I think that in most cases, the performance of rsync or robocopy is good enough.

In other cases, a better strategy would be to transition to using something like GlusterFS. Unlike the more specialized utility you propose, it can avoid creating a new single point of failure and it would make future expansions/migrations easier.

Since your intent is to gauge interest, I suggest you post new posts here and elsewhere explicitly asking if folks find this utility interesting. I don't think LinuxQuestions is really a big hot spot for server discussions. Obviously, this forum does exist and we are here, but there are other places where there are a lot more enterprise server folks.

1 members found this post helpful.

rsync, robocopy or tar, yes in moving the data. But they are really slow to get started. On the scale of task under consideraton it is problematically slow.

Making petabyte bricks under Gluster is not it's forte...

The thought was to see what is common practice.
Evidently, there are no simple solutions.
Either go slow, like days and weeks or use some really complicted tools.

Most enterprises simply don't have that much data in a single unified structure. Heck, most enterprises don't have that much data, period. Those which do, I would hope most of them are smart enough not to put all their eggs in one basket.

Obviously not all of them were smart enough, but consider - how large is the potential user base of those which weren't smart enough?

The projected started for my buddy.
The performance and OS aspects were intriguing,
so I tinkered with the architecture and coded it up.

As for potential client base. Storage arrays of 100 TB
are almost getting to be a standard and relatively small
shops can have petabyte arrays. So, there is a growing market.

Regardless of the above statement, if you want confirmation of and information about the possible usefulness and/or suitability of AUFS to do the job you've outlined, I can think of no more suitable individual to comment on this than Barry Kauler, the originator of Puppy Linux. (Don't laugh...)

Puppy was an 'independent' OS back in the day, and was built from the ground up, with inspiration from Knoppix. Pup has always used AUFS in its very unique mode of operation, i.e., running in RAM, and saving configurations to a 'save-file' or 'save-folder', which layers the data into, over & around the base OS data in such a way that the user is unaware of it.

It's still in use to this day, so Isaac's statement about it being in anything current holds water.

Although Barry has 'retired' from main-line Puppy development, he's still very active, and pursues his own interests in the 'Quirky' series (based on Puppy, but where he explores unusual, way-out ideas.)

You can contact him through his blog:-

http://bkhome.org/news/

Just my tuppence-worth.


Mike.

https://en.wikipedia.org/wiki/Aufs

September 18, 2017 — BarryK> I started thinking about updating the kernel, as I was considering trying the overlay filesystem again, instead of aufs.

The efforts of the individual contributor to experiment with OS work is commendable and I appald such efforts.
This forum is the consequence of such efforts rooted in work done thirty years ago by some folks that wer labeled zealots.

However, an experimetal system that passes data is not necessarily stable or well implemented.

It appears that AUFS is still under development and unstable.
Not quite ready to touch primary corporate data.

The OP was about how to mirror active and relatively large fielsysems.
It is unclear if EasyOS or AUFS are equal to such a task...

Aufs has been heavily used by pretty much all major linux distributions for many years now. Maybe they will transition to OverlayFS quickly over the next few years, but that doesn't change the fact that right now, aufs has been heavily used and HEAVILY tested for security vulnerabilities. It is used by specialized security and privacy oriented linux distributions as a critical part of their core functionality.

Can you say that security experts have examined your code? Why should we trust your code more than code that's used by things like Tails and Kali for core security functionality?

As for how well it scales... I don't see how it could have any scaling issues. There's no impact on the file servers at all, since all of the extra activity is done by the clients. And the clients don't store any sort of fancy data structure to keep track of what bits of data are where. It just dumbly does two directory reads instead of one, whenever doing a directory read. Writes only go to the read-write branch and not the read-only branch, so there's not an impact on writes.