Hi everyone,

About every 6 months I find myself buying a new hard drive to replace one that has crashed or become unreliable (constant superblock corruption, read/write errors etc). Its now become quite a chore for me to backup my drives because i have numerous external usb drives, internal drives, and 2.5" drives.

It used to be easy using rsync to backup files, but now i find myself wanting to back up the back ups so can i get away from the yearly hard drive failures that result in missing files. DVD's are not an option due to the amount of data and rather not go the tape route. Of course $$ is a limiting factor too.

My current setup has my webserver regularly backing up to my home server and while it uses rsync, the amount of data is quite large and will only grow as my client base/usage grows. My web development files, pvr, music, etc, are somewhat backed up on a regular bases to external usb drives that are only powered up when i backup the files. However i would like to remove the human error factor from the process and automate the task so i dont have to issue mount and backup commands.

One product that i have heard much about, and like, is Drobo. There are a few things i dont like about it besides the cost which leads me to think i could build something from what i already have around the house and with less cash and more importantly from the numerous drives i already have!

So here is what i think my ideal file server should be able to do:

• can grow in size as the amount of data i need to store grows. Write speed is of little importance to me as i mostly store and retrieve data in little chunks at a time -- website files, config files, music, archived movies from the pvr etc.
• can be housed in a single case, containing many hard drives (self contained box sitting somewhere on the network)
• can be mounted as an NFS drive, samba for the few windows computers on the network which need to be backed up
• run a debian based OS
• look like one big hard drive when mounted
• any sized hard drive could be plugged into the system (80gb+120gb+60gb+300 etc etc)

Im new to raid and have researched the topic somewhat but as i understand it, raid requires the hard drives to be of the same size. Am i asking to much?

It would also be interesting to know how you guys have set up your file/backup servers. Looking forward to any feedback you guys might have!

Okay, first of all, whatever you read, RAID is not a backup solution so get that out of your head now. It's for hardware failure but not backups. Good for systems where you can't be down for long, if a disk fails, system keeps going on remaining disks without losing data until you can replace the bad one.

Now to backups. Sounds like this is just a personal setup, not in a corporate environment. For just a few systems, what it sounds like you need is a good script that utilizes rsync, tar and so on.

It also sounds like your backups just keep growing cause you don't have a retention policy in place.

Things to consider so you don't just keep adding new drive space to your backups on data that is not growing as fast:

1. Think of a retention policy. Not sure how long you want to keep your data, a good month rotation is probably a good place to start.
2. Start with a full backup at the beginning of the month, incrementals afterwards. Or do a full dump weekly and incrementals daily inbetween the full dumps. Keep them for only a month or two months.
3. If you want to make things easier to automate, look into using something like Bacula, BackupPC or Amanda. In your case, BackupPC is probably ideal as it's targeted towards desktops, etc.

Search around for backup scripts, I know I've posted some on LQ for others to use, others have done the same. Customize it to your needs, that's if you end up going with some custom scripts. If it's just a couple of machines, might be the way to go, if it's 4 or more, go with Bacula, Amanda or BackupPC.

My own servers I have, I use rsync. I have a script that does daily incrementals and keep them locally. I do a weekly dump and have an offsite server pull those down. I also have daily mysql dumps for the databases that get pulled off daily as well. The actual data on the filesystem changes rarely, most of it is in the databases, so that's what gets priority. If it crashes and I have to restore from a dump from a week ago, chances are, I won't lose much that changed.

Backups are like snowflakes, design and implement them to your needs cause everyone's snowflake is different. Yeah, that sounded a little gay, it was my feminine love of snowflakes creeping out..

I understand raid is not a backup solution, but the reason i am interested in raid is due to hard drive failures. My data usage like you said doesnt grow very fast--meaning 10s of GBs a month, but having a file server that can withstand hard drive failure and help prevent data loss and can grow in size as my needs increase, is something I am interested in.

Also, most of my files are client projects and server backups, but i would like to eventually archive my pvr recordings and music to the same file server. Most of my clients are photographers and take a lot of pictures during the summer months and upload it to their websites for selling purposes, so in the case of hard drive usage on their part, it spikes during the summer months. And while this is not a corporate setup, I work from home and my network includes many users locally and remotely.

I use rsync via ssh and with cron on the server to my current file server and also custom rysnc scripts for local computers. But im looking into what kinds of options i have that can remove the hardware issues from the equation.

Is it impossible to have a system that can take many hard drives of different sizes, mash them together into a large virtual drive, and raid it so that i have a copy of the same data in case of hard drive failure? It would be a dream to just throw out the busted hard drive and insert a new one and have the system rebuild itself.

Well what kind of drives are you using that seem to fail so often? Setting up a RAID, striped with parity will prevent a lot of downtime when a fail does drive so I would recommend it but what type of drives are you buying that fail so often? I still have drives that are 5 and 6 years old grinding away without problems.

In my experience though, if you really want a good reliable drive, go SCSI. Get rid of the IDE, SATA and whatever else your using and buying on sale locally for $100 or whatever and spend the money on some good SCSI drives. They'll pay for themselves cause you won't be replacing them every 6 months or year like the other drives.

You can mash a bunch of drives together using LVM to create a large volume but adding all these drives in some type of RAID setup probably won't work like that.

Also for increases reliability and you go with a RAID setup, look into getting a hardware RAID card to handle the RAID. Software is ok but still sucks and will never beat Hardware RAID.

forget the old scsi - go for SAS

True also. Anything SCSI really is what I meant by suggesting SCSI drives.