Need a Backup Solution
 

Hi


I have couple of servers, which have data like 200+ GBto Backup. I was planning to take Differential Backups on hard drives not tape drives.

Can someone please suggest me any open source software which can do this job for me. I had a chance to look at Amanda but it look too complicated to install.

Any easy to install and go software available or any scripts which can do this work?

Thanks

Try these
 

Try a couple from this site:

http://www.linux.org/apps/all/Admini...on/Backup.html

See if any of them float your boat. And if they don't... Sounds like a new open source project to me.:D

rsync sounds like what you want.

Or you can use tar - on it's own for complete backups or in conjunction with find for incremental. Something like

backs up last full day plus partial modified files

There are lots and lots of open source backup projects. No need for new ones. If you really wanted to work on something like this, you would find a product that you could contribute to and add the features you want.

Personally, I would use Amanda.

But, if you want something else, more basic, you could try this rsync recipe: http://www.mikerubel.org/computers/rsync_snapshots/, which is really quite snazzy.

Thanks for your reply people.

New Project doesn't seems to be a good idea even for me. Thanks for those links.

I wanted the differential backups and rsync doesn't seem to do that job, so its out of question. Although personally my first choice was rsync.

I was going with Bacula somehow I think it has better docs than AMANDA. But it would be kind of you if anyone have a tutorial or tips to install Bacula or AMANDA. Truly this stuff is very complicated lol.


Is there any other Backup Software which has good interface??

Did you check out the link I gave you? (http://www.mikerubel.org/computers/rsync_snapshots/)

That describes how to do snapshot style incremental backups with rsync. The only difference between what people typically call incrementals and differentials is the reference point. I wrote a piece that explains backup levels here http://wiki.zmanda.com/index.php/FAQ...da_use_them%3F (other parts of that were contributed by other people). So, using the techniques described in that first link with rsync, it certainly can do differentials, if that is what you want. Personally, I would be inclined to ask you why and to recommend following the technique described rather than modifiying it to do differentials.

Backup can be complicated. But it doesn't have to be that bad. My Amanda system just runs without me ever having to do anything. I check logs regularly to confirm that things are working, and I test recoveries, again to make sure things are working.

When I first set it up, I followed the quickstart guide here: http://wiki.zmanda.com/index.php/Quick_start, jumping off into other parts of the wiki for details and clarification as needed, since my setup differed a bit from that one.

Bacula has docs. That's for sure. Some find them overwhelming. But there is good documentation on the zmanda wiki and great help on the amanda users list.

Bacula is suitable and actually quite easy to use, install and configure. It's daunting at first but quite easy once you understand all the components involved. It sort of reminds me of Veritas NetBackup Enterprise without all the bells and fancy GUI's.

It's very flexible and robust. I like how you can have the Director and Storage daemons running on different hosts or have more than one Storage Daemon if you wanted to have two locations for backups. Runs well with disk based and tape. I've run both and prefer Bacula over Amanda, just seems better and easier to use. Although I've been a backup administrator in my previous last 3 jobs usually dealing with commercial vendors, when I had little or no budget, I went with Bacula. I actually replaced Amanda with Bacula which scaled better and was more suitable in the environment I was working with.

Bacula also scales very well when you have to deal with Windows clients as well, not requiring Samba in the mix.

Also the bconsole seems to be more advanced than the several different commands you have to use with Amanda, making it very easy to launch from the Director or client even to perform backups, list backups, recover and just about everything.

One other annoyance is Amanda still uses cron for scheduling jobs. Bacula actually runs as a daemon, which is easy to monitor if it crashes, shutdown, etc with just about any monitoring program like Nagios or OpenNMS, etc.

two superb replies with 2 superb softwares.

choogendyk the Amanda Quick Start links looks pleasing to me but then Trickykid replies looks satisfying to me lol I'm confused.

I will go ahead with Bacula first and then will install Amanda if that won't work.

You being great guys Thanks for help.

choogendyk I still can't find how to take differentail backups with rysnc. I understood your point of talking about reference points when dealing with incremental and differential but in that link you gave I am still searching how to go head with differentail backup using rsync. Can you give me an example to make things little easy please.

Actually differential and or incremental is kind of already built into rsync. Check the man pages for all the available options and power of rsync.

And as for choosing Amanda or Bacula, it really just boils down to what you prefer. I'm not making a case for Bacula, I just simply pointed out I feel it's more robust than Amanda depending on what you fully want out of your backups and how to administer them.

Personally I don't like how Amanda still uses cron jobs to schedule backups and the several different commands to launch to recover, backup, etc kind of turns me away. That's why I mentioned the bconsole in bacula, it's very robust, easy to navigate and use once you get everything setup.

Plus I'm usually in environments where I need or want to monitor processes, Bacula running as a daemon makes it that much easier for me. Also these environments had Windows clients. Bacula working without Samba makes life that much easier, since I don't want to run Samba, it's nasty. Only run it when I really need to or forced to with no other alternatives.

As Trickykid says, choosing Amanda or Bacula really just boils down to what you prefer. Tricky obviously prefers Bacula, and I obviously prefer Amanda: http://blogs.umass.edu/choogend/2007...-about-amanda/

I'm not quite sure what that says about Tricky and me, but I think my blog post spells out where I'm coming from and what my reasoning is. If that connects, then perhaps it will help in making a choice (or it won't connect and that may help as well ;) ). I also responded to a thread on the forums at zmanda some time ago where someone was asking for a comparison of Bacula and Amanda: http://forums.zmanda.com/showthread....=bacula+amanda . I happen to like the command line and Unix native approaches for systems administration tasks.

I use mon (http://mon.wiki.kernel.org/index.php/Main_Page) for monitoring, because it fits my minimalist unix approach to systems administration. I would point out that knowing a daemon is running doesn't necessarily mean that it is working or that it is doing its job the way it should (although, since it has to run as a daemon, then I suppose you have to make sure it is up). When mon monitors my DNS server, for example, it actually issues some basic DNS commands and looks at the responses it gets back. For MySQL, it does a table lookup. For Apache, it requests a page and looks to see that it got it.

I know that Amanda runs because it sends me emails with the results. Amstatus fires off in the mid afternoon and checks everything to make sure it is ready for the night's backup. If everything is fine, I hear nothing. If there is a problem, amstatus sends me an email with the specific details for what I need to straighten out before I leave work. Typically, nothing ever comes up. I check log files periodically to calm my natural sysadmin paranoia. Same thing with mon. Gee, is mon really working? I should take something down and see if it reports it. Yup. It's working.

Anyway, I should also point out that development in the Amanda/Zmanda world has been very strong over the last couple of years. Amanda is very stable and solid, there is a native Windows client now, the enterprise version has a management console, there are some powerful new API's, an Amazon S3 interface, and lots more. So if anyone's experience with, or memory of, Amanda goes back a few years, it may be time to get reacquainted.

Well with Bacula, they have created a fancy graphical interface but I do all my administration from the command line as well, which is easily done with Bacula.

As for monitoring, well, mon works but it might not scale with everything else you might have running.

Nagios even has a plugin for Bacula to notify you if a backup fails. Using SNMP, you can get a lot of information to monitor using OpenNMS, etc.

Bacula also has email notifications, tells you if a backup failed or succeeded with a ton of other info about the backup, how many files.

Another nice thing I do like and others might not like is that Bacula uses a database for it's catalog server. Makes it easy to backup and customize your own scripts and or programs to query whatever you need for your backups. It supports MySQL, PostgreSQL and SQLite for the catalog backend. From my experience, querying the database to look for jobs, list jobs, completed jobs and everything else seems to be faster than the reports from Amanda.

It's also utlized python, so creating custom scripts to interact with it is also a plus.

The only thing I do like about Amanda is that is uses native tools that comes with about any Linux install. But then again, with Bacula, you don't need a catalog or it running to extract data in a crisis, they have tools to extract and import the data into a new catalog for disaster recovery planning, etc.

Also using Bacula, I never saw load or performance hits on the clients or server. Just like Amanda, they actually both scale well on I/O performance.

It just really boils down to, what you want and need from your backups. Amanda to me is great for a small server farm but Bacula seems to be really good on a large scale, easier administration once you get it up and running. At a previous employer where I setup Bacula, I had it doing backups on 200 plus servers and around 60 MySQL database servers. Once it was setup, I never touched it for months unless I got the occasional server fail, usually for stupid reasons or human reasons, only to kick off another backup manually if it was important enough or I'd leave it to pick up the next day for it's incremental, etc.

A certain amount of this is a matter of preference and style. After all, these are the two most popular open source enterprise backup tools. However, I do have a couple of counterpoints.

Hmm. Kern Sibbald (author of Bacula) itemized 7 different reasons why he was probably going to drop support for python: http://www.mail-archive.com/bacula-u.../msg27416.html , perhaps the most serious of which was that python is single threaded and if multiple Bacula threads try to use it, it has to be blocked.

The Amanda front end (2.6.0 or greater) is now perl based.

Yup. That's one of my major things. If I have a researcher who leaves and goes to another university, I can send all his data tapes. Any moderately savvy sysadmin who comes along and dd's the first file off the tape will find that it is a text file with the exact unix/linux instructions required to read the tape. Don't have to learn about or download or build any tools you don't already know and have.

Amanda also has installations on that scale.

And mon has installations scaled to thousands of machines.

Simple, elegant software can scale very nicely.

Well, any competent sysadmin should also be able to research and find out how to extract the data by downloading the proper tools as well. Also, any competent company or IT department or other department he was in should also ensure he has proper documentation so if he just decides to leave or drop dead, anyone should be able to take over by reading the prepared documentation for their processes in place. :)

So do many others, it was a simple acknowledgment that I made to make the OP that Bacula is very scalable and robust.


True and I agree but looking at it, it doesn't have all the bells and whistles that some corporations need and or want out of monitoring. I mean, OpenNMS uses SNMP which can tell you just about anything about any system. It also has autodiscovery and plenty of other things like graphing, managers and execs love graphs.

When you have a company that depends on SLA's, mon and other simple elegant software just don't cut it when you have to present a board of directors of any downtime, etc, especially to clients and customers.

Down time? What's that? ;)

Oh, I guess that must be what hit Amazon yesterday.

Haven't seen any myself.

Actually, I did have a tape drive failure just over a year ago. Amanda notified me of it, and then proceeded to drop the fulls it had scheduled back to incremental (to save holding disk space) and did incrementals only of all scheduled disk list entries to the holding disk. Ultimately, I ended up having to remove the tape drive from the library, manually wind the tape out of it, reset it, re-install it in the library, test it, etc. Back and forth with manufacturer's tech support, it took a day and a half to get to that point. Amanda continued on, doing incrementals only to the holding disk. When the drive came back online, Amanda saw that it was available, rescheduled a bunch of full backups, flushed everything from the holding disk out to the tape, and was off and running.

So, technically, that was some sort of outage. But, was it down time? Amanda, and therefore backups, just kept running. I never had to intervene in the backups. While I was focused on fixing the tape drive, Amanda just did the right thing; and, if anyone had lost a file or a drive, I would have been able to recover it.

Most other backup software I know would have choked when its storage daemon or whatever failed. There would have been no backups during that period, and any files lost by end users would have just been lost.

I'm into maximizing redundancy and minimizing dependencies (and complexity) when setting up systems. Those are among the primary criteria that guide my decisions in choosing and setting up software and hardware (with the caveat that I don't have a pot of gold, so I generally can't have what I really want in the way of hardware).