Hello,

I need your advices regarding a backup strategy !.
I read a lot of method regarding the way to make a full system backup of Linux Servers. I need to look for a general strategy about it !. Currently i guess the person in charge of it just make a "tar" file of the important filesystems, and next use TSM to save it on tape !. In case of major crash, he has to re-install a new server, retreive the TSM back and overwrite the Tar to get back the server as it was originally configured !.
I don't think it is best and efficent solution.
Do you have any others solutions ?, advices ?,
I am also open to commercial product...How are you dealing with this major question in your side ?

Regards,

Maybe you could provide more information on your server's filesystem. Also is this the only server you backup. Another thing to consider is if you need to backup security contexts as well. For example, if you use acls you may want to use star instead of tar.

If you use XFS for example, there is an xfsdump program that can be used to produce backups including incremental backups. This backup could be done on a live system.

If it is an LVM volume, you can produce an LVM mirror of the initial installation.

You could also use "dd" to create an image of the filesystems. This wouldn't be a general backup solution, but could allow you to get the server up to the point of the initial installation and allow you to apply subsequents restores. You probably would want the server offline to perform a "dd" image backup.

If you use tar, you can produce a periodic full backup perhaps once a month, and then use incremental backups. You could also use the original timestamp file once a week to get a weekly backup of cumulative changes once a week and then apply daily backups. Look at the tar info manual. You want to read the section on incremental dumps.

If the server is a database, you will want to perform database dumps as well. Also look at off-site storage in case of a disaster. This is of particular importance for financial records that you need to preserve and protect by law.

Using a raid-5 array will protect you against a single drive failure. It won't get you accidentally deleted files back, but can reduce the frequency failures. I just had to replace a disk on a server at work today. We were getting daily ecc errors warnings. Even though the raid array was still healthy it was replaced just in case. On the other hand, I once had a server where lightning hit near by and the entire raid subsystem was burnt out (literally, holes where the drives plugged into the backplane). While we had a UPS and a surge protector on the modem, this server had other equipment connected to it via video & audio lines. Raid 5 won't help then.

more reading at http://www.linuxquestions.org/bookmarks/tags/backup

also the O'Reilly "Backup & Recovery" book by W. Curtis Preston, and the corresponding website http://www.backupcentral.com/

if you have multiple linux servers that you need to cover, and you need to implement a backup strategy without having to script all the separate pieces individually, then an open source backup solution like Amanda might do it for you (covered in the backup bookmarks linked above).

good information all that !

Commercial offering for imaging Linux systems.. you might want to look at Acronis http://www.acronis.com/
http://www.acronis.com/enterprise/products/ATIES/
Acronis has several different solutions available

Depending on the value of the data and your budget there are a few solutions out there that make 'instant' backups of data as it changes on your server.

Actually, sounds like he's doing a Bare Metal Recovery type backup, which actually is very efficient and can be a wise solution. Things to consider when backing up servers, data, etc with a Bare Metal Recovery type backup are if you backup the whole system, including the OS that comes default, you might just be wasting valuable space on the backup hardware or appliance you are using, if it's either disk based or tape based. This type of backup you described can be very essential in backing up only what you need, saving space, backup time, etc.

dd is only really effective if you know no changes are being made on the filesystem, not very effective on a live production system and really only good if you need to move an image to another server. Has no integrity checks, no validation of data, not a good backup solution if you value your data in my opinion. If you have a large setup, with a lot of servers, a commercial product or something that can handle the backups like Bacula or Amanda would probably be your best choice. Go with commercial if you like support and need that warm fuzzy feeling that some company is going to guarantee your backups are safe with their product, if it fails, you got a finger to point with.

Now maybe in your setup using tar just isn't manageable any longer, but it can work with what you described but if you are using home brew scripts on a dozen or more servers, it's probably time to move on to another product that makes your backup solution easier to manage.

It is actually very common to only backup the actual data that is not native to the OS. Reinstalling an OS can be as easy as 15 minutes if you have the right setup and determining how badly the crash was. In most cases, if you lose data and need to restore, you don't need to reinstall the OS but recover the lost files. Almost all data loss on a server if it's not hardware failure is caused by humans.

The book mentioned is actually a very good book. I've used or have touched just about all of the major commercial and open source backup software out there and prefer Bacula if I'm on a budget and Veritas Netbackup for commercial.

Or if you like the "warm fuzzy feeling" of someone sucking cash out of your pocket. Oops, sorry, that feature is extra. Cost you $ $ $. Oh, you're over your client limit. You can add more for $ $ $. Hmm, you've got "that" tape library? You'll have to add the special driver kit for $ $ $.

I always thought the give and take and friendships of the open source community were closer to what you might call a "warm fuzzy feeling."

On the other hand, if you have a situation where your corporate accountants and lawyers require indemnification or something like that, then most open source projects aren't prepared to deal with that. One exception is Amanda. You can choose open source and also get a support contract and indemnification from Zmanda.

While spending lots of $ $ $ as you put it can be off putting and expensive, losing critical data because your backup solution didn't work can be far more costly to a company, even possibly putting you out of business. I certainly wouldn't write off a commercial solution out of hand just due to cost if the data is critical.

With backups, as any other product, there are hundreds of ways to skin that cat.. find a solution that works, TEST IT, be comfortable with your choice and the data recovery and by all means continue to do test restores regularly to ensure your data integrity.

Just all depends on the situation, who you're working for and if the executives or even manager wants some type of support. I don't write off commercial applications ever. At times they may have a product that just works with very minimal interaction. The part about being a good sysadmin is to have most of your job automated. I learned to become a lazy sysadmin, if you have to do it once, automate it if you can so you never have to do it again.

Commercial applications, especially when it comes to backups are several steps above and beyond what most of the open source applications are currently at. Trust me, I don't write off open source either, I currently implemented a Bacula plus MySQL-ZRM solution at my present employer cause well, it's a startup and we don't have the money to pay for a commercial solution and secondly, the types of backups we have really don't demand anything special that a commercial solution would provide.

But commercial applications still have their place when you're dealing with a mostly open source workshop. I had a previous employer paying Red Hat for all 100 server licenses we had deployed in our production environment. When we soon realized that we had only used their support 2 times in 3 years, once sending an email and finding the solution from a google search before they even responded, we took this information back to the Operations VP to prove that we didn't need to spend 100k a year on support for Red Hat.

Most executives don't understand Open Source and how it works. Most don't realize or have 100% faith in the employees they hire to run their operations, that's the warm and fuzzy feeling I speak of mostly, it's to ease the minds of the nimkapoops that shouldn't have survived birth but somehow run our corrupt corporate world.

Not sure how or whether to respond to all that. Some of it is basically in agreement, and some of it is a matter of opinion through which all sysadmins have to find their own way.

I have a few situations where I resort to commercial software. For example, personally, I like Timbuktu over all the competition. Since it's just for me and a few computers, it's inexpensive, and I ask the department to buy it. Also, because our sysadmins are stretched thin (we really need one or two more), we can't support the research labs in the several departments we serve. So, we consult with them, but tell them they have to be responsible for their own backups. We recommend EMC Retrospect, because it is so easy for them to install and run themselves. They buy a disk for our servers and point Retrospect at it to do a disk archive. Then we periodically put their disk archive on AIT5 tapes using Amanda, but we don't have to deal with their lab computers.

Personally, my main reaction was to the specific use of the term "warm fuzzy." I often get that feeling from the support communities around open source. The term I would use with reference to the bureaucrats, lawyers and bean counters is more like "cold calculating." Yup, you have to deal with both, and you have to balance a lot of different demands and interests. And it depends on the internal support and management that you have. I happen to have a very supportive IT Manager and department heads. So I lean heavily to open source. I put in the setup time, and then it typically just runs. This applies to my Amanda backup server and all the other applications on my various servers, including Amanda, Apache, MySQL, php, Mediawiki, Sendmail, milter-greylist, mimedefang, majordomo, bind, dhcpd, perl, gcc, mon, OpenBSD, etc., etc., etc.

trickykid's original use of "warm fuzzy" followed with "if it fails, you got a finger to point with." And farslayer's response to my mention of "$ $ $" included "losing critical data because your backup solution didn't work can be far more costly to a company." So, the implication is that the open source solution may fail, and the choice of commercial software is in part so that when it fails you have someone to blame. In my experience, failures of either open source or commercial backup software are either a failure in the configuration and running of the software or a hardware failure that wasn't properly anticipated in the setup and configuration of the backup system. In my case, the only significant loss I've ever had was years ago with commercial software. The loss was my own fault, even if I would argue that it was a poorly defined option in the software. I've never had a loss with open source software.

Serious software bugs resulting in data loss are rare. There was recently a longstanding bug identified and repaired in Bacula, but it didn't result in actual data loss. You could still get the data off the tape, it just wasn't recorded in the sql database, so the normal procedures for recovering data from the tapes didn't "see it." But, I've also seen serious flaws reported in commercial backup software in the last year. One basically allowed any computer with the client installed to recover sensitive data from your server, even if they weren't explicitly a member of your backup set.

On the issue of "nimkapoop" managers, it's absolutely true that some managers just don't understand a lot of things. That can be true even if they have the title "IT Manger." Luckily, I got out of "that" job a few years ago, but I recognize perfectly well that some sysadmins have to deal with less than ideal managers. It's also true that there are some situations where the manager could spend their money better by "buying" a better sysadmin, and that blowing that money on a commercial backup application wouldn't solve "the problem." Hey, it can't always be their fault.

Now, with full assurances that we have gotten completely off track from the OP's immediate interests, I'll close my missive for the day.

Don't get me wrong though, I get warm and fuzzy feelings from dealing with the community with open source software. But I've worked many places and have come to realize that no matter how hard you try to explain whatever situation, some managers or execs will never get it. They think like a business person does, they don't understand the term "free".

I say you can point the finger cause I have gotten to do that before, with a commercial vendor, it was quite fun. I will admit it is rare and if you do lose data, it's either human error or hardware failure when it comes to backups. I guess when it comes down to it, if you have mission critical services, sometimes you just can't wait to rely on the open source community. When you pay someone and depending on the service you paid for, they have to respond according to the support contract.

It would be nice to only use all open source applications but in the real world, that's just not always entirely possible.

And I did follow that bug with Bacula as it didn't affect me. I only run one job at a time, well, I did at my previous employer where I setup Bacula. Hopefully they didn't make any changes or have upgraded since then, I doubt it. None of them knew anything about backups. For all I know, they're probably failing for some unknown reason and they don't even realize it cause the guy who took over was that dumb. He was the type who'd rather have some vendor come in and do the work for him, that's how they got a new monitoring system setup, he learned the basics while watching the vendors do all the work.