All,

I've been asked by a company to create a clone of a co-located remote system running CentOS 4.x. That system is located in Virginia and, aside from some minor help from remote staff (e.g. hardware reboot), really isn't touchable. The remote system isn't RAID- or LVM-enabled. It runs an old copy of the Ensim control panel, which is no longer available for Linux, and for which the company no longer has the license scheme. Down time is not an option for this machine.

The remote system is loaded in its entirety on sda. A second, empty, disk (sdb) is also installed. Ideally, the clone would reside on sdb, so that if sda crashed, the system could be booted from sdb. That's not an absolute requirement, however.

I've looked at more possible solutions that you can imagine, but end up rejecting them as being unfeasible. For example, I've considered "dd", but that risks creating inconsistent filesystems. I've considered adding software RAID (RAID-1), but that risks data loss.

I've also considered (then rejected, then considered, then rejected, and am considering again :-) ) SystemImager. If I could get a true clone image of the Virgina system on a master server in the company's New York headquarters, that would work. SystemImager seems ideal for such a task. There are a few how-to's on the Web, but all of them pertain to master server and golden client on the same subnet. I'd ask for help on a SystemImager forum or mailing list, but there aren't any. Maybe I'm being a chicken here, but I would like to get this right.

I'm hoping that some bright person or persons there might be available to help me figure this out, especially if someone has already done something like this.

Apologies for such a long post, but I wanted to present as much information as I could.

Thanks.

Diggy

dd is indeed capable of making a serious mess if the command is delivered carelessly!

dd is also the one absolutely sure way of making a clone, which can include even the Master boot Record of a drive (like sda), and putting a copy of the whole record somewhere else (like in a file on a bigger partition).

Probably the best way is to use dd to make the copy to a backup file.
Also, calculate the MD5 sum before sending it.

Then, transport that file copy to your local machine. Use ftp, the post, whatever.
Once in your possession, you can use dd again to make a clone on a new drive - or just on a new partition is you have the space. The filesystem and all will be replicated.

You can also then mount the copy to any folder you make up, and take your pick of the contents.
This is where you have to use the appropriate -t options for the mount command, to suit the filesystem in use. There is lots of advice on the internet about how to use the dd command. If the clone is itself comprised of several partitions, as opposed to just folders, with those partitions possibly having different filesystems on them, then I am unsure how to get at them, but I am sure it is possible. At the very least, you will have the clone with you, even if temporarily, we do not yet know how to mess with it.

I once made a clone of a Windows drive this way, and despite my inexperience, it just worked. I will admit I was nervous, but it booted up as if it was the original.

check out mondo or cloanzilla.

both would work in your situation

Thanks, GTrax.

Believe me, I've read and asked and read and asked, but just can't get any consensus on how to do this. Whenever I consider using dd, someone throws cold water on the idea by pointing out that there will be filesystem inconsistencies where open files are concerned. Oh, did I mention that there are active MySQL databases running on the remote system? I guess I could somewhat mitigate that problem by doing database dumps, then restoring them.

I should note that sda holds the entire OS and several filesystem (well, the usual suspects, anyway - /usr, /var, /tmp, /home. etc).

I've thought of jus doing a simple "dd if=/dev/sda of=/dev/sdb bs=4096
conv=notrunc,noerror". But, I've also been told that I should do a dd of each filesystem individually, then run fsck on each filesystem indvidually. So, I'm still confused.

Diggy

Thanks, watcher69b.

As I understand it, using clonezilla or mondo requires that the target system be rebooted with a clonezilla or mondo-created CD. I think mondo can be used with NFS, but I'm not sure. At any rate, again, system down time is not an option. Nor is physical access to the system possible.

Diggy

OK Diggy
It is possible that watcher69b's suggestion is OK. Explore the tools. They may indeed allow a clone to be made and shifted offsite without downtime.

But back to dd.
So long as you use make the input file (dd if=sda of=<another_place> without any regard for record length, or many of the options dd offers, you will get every byte, including the unused space, with all the deleted files, random writes, fragmentation, address tables and formatting.

It is a harmless way of getting all those bytes stashed where you can have downtime, or make mistakes. You can even make another copy from it, for backup, so you can mess with it at will. In short,you have nothing to lose in attempting to copy the data, all the bytes to a safe place using dd. Just make sure that where it lands has enough space for it to land. BUT - there is a downside!

I learned a lot about using dd when trying to find the MD5 checksum of a CD that had already been burned from an iso file. Once I knew the record length and blocksize, I used the pipe (|) method to feed the dd output to the md5sum.

In your case, you will need at least enough time to access the filestore and get the copy without it changing while you are trying to do it!

For this reason, it may be better to look at the partitions on sda, and the content setup. Discover the filesystems on the partitions (there may be more than one) Then make a plan to copy off the content in separated chunks, keeping a record, so you can put it all together again. To ensure the bit you are snatching is not altering, you can make it unmountable by the system while you copy.

When taking a snapshot of a live system, you need the assurance that a part you copy out near the beginning does not have fragments further in. Just imagine if the file table entry for some chunk was marked DELETED, while your dd command is blissfully unaware, and carries on regardless, assuming that space yet to come was still assigned to the file. This is why you can end up with corrupted file systems by trying to use dd in a inappropriate manner. It only works if the data stays good (ie. unchanging) until you are done.

This is also why you should consider FileZilla or other tools.
Also, post the question among the server admin experts. I am sure there are some in this forum. These guys have had years of doing exactly what you are after. It is a common requirement. There will be safe, developed tricks to getting this right. I am not the guy, but I know there are some right here.
Good luck with it.

GTrax,

Thanks for taking the time to post a very informative reply. Appreciated (along w/ reponse from watcher69n).

I actually posted the same question to Dell's linux-poweredge mailing list, which is dominated by sysadmins. Alas, I got responses all over the board. Some recommended dd, others put the kibosh on it. Some recommended other techniques, others threw cold water on those ideas. Arrgh!

What makes this project tough, in part, is the fact that the box can't be physically touched (as it is both a colo'd box, and is in Virginia [while I'm in Massachusetts, and the company which owns it is in New York]. The other issue is no possibility of down time. Take those two things away, and the project's a snap. But, who said life was fair :-) .

I've looked at both clonezilla and mondo. Clonezilla seems not to be an option, but I've posted my question in one of its forums (fora?). It doesn't appear too active; mailing list even less so. But, we'll see.

I've also previously posted on the mondo mailing list, but haven't yet received a definitive answer. I'll try again.

SystemImager looks like it might work, but its mailing list is pretty quiet. I'm trying to explore that as best I can.

So, I'm back to square. I might have to suck it up and just try something - if it works, great, if it doesn't, ... . As long as there's no harm to the original data.

Diggy

To clone smth you need to unmount it first otherwise you risk getting errors in your copy.
I use partimage client to make a clone of a partition and later partimaged to serve those images.

rsync. I used it to mirror my system hourly to another drive in the same box. One day I lost the main drive and I saved alot of grief by being able to boot right into the mirror. You will have to partition and create filesystems and do a little tweaking but if you can't run RAID then rsync with cron is the next best thing.

You can use rsync across the network as well.

This is a great forum, and I truly appreciate everyone's feedback.

randux, I've used rsync several times to make backups of files and directories (I'd be happy to share a shell script I wrote to back up four weeks of a cvs repository to an internal and external HDD using rsync), but never to create a bootable disk. Might I ask, what tweaking?

Diggy

My preference is to use tar and untar for such a task, but essentially it is equal to rsync.

However, with copying you did not automatically create a fail-over or redundant situation. It is only when sda dies and sdb becomes sda at the next boot. How would you point the boot loader to sdb when sda is visible but only doesn't read?

My advice is to try this first at home. Take a running system, and make a clone using tar or sync. Install the boot loader on sdb using chroot and grub-install. Edit fstab, using UUID's. Edit menu.lst (or grub.conf), use there UUID's as well for booting from sda or sdb. Finally, edit menu.lst so that the next time the system is rebooted, it does so from sdb. There is also an option to change the config when a boot fails. Google for "boot grub remote" and you'll find the docs.

After having done this, make a copy at home from the remote system, exclude most of the data if that is much. Try the same.

Only when you have performed both steps do the same on the real machine. BTW how do you keep the data updated between the two disks?

It is extremely dangerous to trust us here to get a step-by-step instruction. If you follow all our steps, and there is just a tiny little thingy different on your colocated system, you have a disaster. Therefor, trying it at home is even more important than deciding on dd/tar/rsync.

I cloned a live system more than once, mostly at home, but also at 8000 km distance. Always rsync/tar, never dd. Be sure to exclude /proc, /sys and /dev in your copy, but do create the directories on the target system.

jlinkels

Welcome to the forums, Diggy. The majority of people on here are really good folks helpful and friendly.

jlinkels did a good job of covering the high points and said pretty much what I was going to say.

There's all the stuff you need to do to account for differences on the two boxes most of it has to do with setting up things once and then doing excludes in the rsync command.

For example, as mentioned, unless you have the 2nd box partitioned the same as the first box you will need to have a different fstab on the second box. Also as mentioned you will also need to have a different menu.lst (GRUB) or lilo.conf (LILO).

If you have X you will probably want to have unique xorg.conf files on both boxes unless you have the same card, etc. Maybe you have unique network configs or local scripts or anything else to deal with.

It's important to try it and tune it until it does what you want.

I haven't worked with Linux that much but I have done some volunteer work and had some experience with SME server 7.3 which uses centOS 4. The current project that I am working on is migrating SME to new hardware. I believe you can use what I'm trying to do and make it work for you. It's just an idea and that's my 2¢. Here's the link
http://wiki.contribs.org/Moving_SME_to_new_Hardware