Hello everybody,

I am a junior IT-support employee and I have been tasked with setting up our new internal Linux Backup server to replace a failing ReadyNAS device.
I would like some advice on setting this up.

Hardware and storage

For the hardware an HP server has been provided with four 12 Terabyte disks.
I have configured a RAID6 hardware raid allowing me to have two disks of fault tolerance.

I assume I can not do a lot wrong with this setup?

Software

This is where I have the problem. I would like to use a stable enterprise Linux desktop with NFS and Samba mount capabilities, but I am unsure of what to use. We are currently using Scientific Linux 6 on our servers, and Scientific Linux 7 has just come out. (A CERN Red Hat clone)

AFAIK it's usually not a good idea for critical systems to use the latest versions until all of the bugs are ironed out. Should I stick with SL6 or use SL7 anyway? Or are there any other suggestions that would better suit my needs?

I would like to stay away from FreeNAS/ZFS/Btrfs for now. I don't want to take any chances with our company backups, this will be the only backup server we use, and I will try to regularly create a full backup on external disks.

Also, what filesystem should I use? SL7 defaults to XFS (which should be better for smaller files)

Any advice would be great!

Thanks in advance,
Pascal

Why are you looking to install a "Linux deskto" if this is going to be used as a backup server?

Are you only backing up other Linux boxes to this or do you have Windows machines too? What is your thinking behind using NFS and Samba mount capabilities?

What have you considered for versioning / incremental backups as well as off-site storage?

Thanks for your reply TenTenths,

Why are you looking to install a "Linux deskto" if this is going to be used as a backup server?
I would be installing a Minimal Install type installation, with the bare minimum of services running and no GUI.

Are you only backing up other Linux boxes to this or do you have Windows machines too? What is your thinking behind using NFS and Samba mount capabilities?
The primary goal is to backup data stored on our Linux machines.
- LDAP Database
- SVN / GIT Server servers
- Multiple Bugzilla's
- Internal Twiki
- Other internal websites and MySQL databases
- Basic server config files

What have you considered for versioning / incremental backups as well as off-site storage?
The Linux environment used to be managed by a consultant, he had written his own script to perform the backups.
I will probably be using this script in the future or creating my own. It's a very simple script.

Versioning:

1) Rename all directories from daily_N to daily_N+1
2) Create a hardlink copy of daily_1 to daily_0 (cp -al)
3) Rsync the current state to to daily_0 (rsync -auv)


Off site backup:

We are a small company and off-site backup is done via USB disks in a bank vault, so live off-site backup is not an option.

Raid level is the ratio of money to security. Raid 6 is pretty good but it would depend on how valuable the data is. We have to assume it is real hardware raid and not anything less.

RedHat and clones seem to have fallen back in love with XFS. Not a great reason to ignore that. Many many improvements have been added to XFS. I've used it since IRIX and still using it on IRIX and that is a very long time.

Once those based distro's release a version you can consider it pretty safe as a start to build. The issue with just released tends to have issues with upgrades more from apps and bug. You are a clean slate so start off with the most up to date.

Your choice of nfs and samba may also have some security issues that need to be looked at. Not sure what would be better either of a choice like NFS or iscsi.

Thanks for the advice, Jeffro.

Your post already makes me feel a lot more secure about my setup and starting with SL7.

The server is in fact using a built in Hardware RAID controller from HP.
I guess I will have do do some reading on XFS to make my decision since I have no experience with this filesystem (pretty new to this world).

Our server VLAN is pretty secure and NFS/samba will only be allowed between servers, outside access is not possible without VPN access and I don't think we are a priority target for hackers to get in anyway. It's just a SMO with not too much money but a little data they want to securely backup.

For now the biggest concern is to have all of the user and logging data secure. (with a regular backup to external HDDs.)

I think the backup server is the most valuable asset system administrators have, so I need to get it right.
If data or service information is lost, you can be pretty sure you get axed.

Pascal

Sounds like you've got a good plan.

Regarding your offsite backup, I'm still not convinced that USB hard disks are reliable enough for extended use/transport/storage (too many moving parts!!!)

Call me old-fashioned but I still prefer offsite rotatable tape.

Our backup strategy consists of:

• A central backup server that takes nightly backups using dervish (this gives us differential versioning with history). We keep a rolling 3 months on this server.
• On a daily basis we backup the full 3 months worth to LTO4 tape (actually 4 tapes in an HP autoloader) using AES encryption.
• We use Mon/Tue/Wed/Thu/Fri1/Fri2/Fri3/Fri4 tape sets.
• First Friday of every month we take a new set of 4 tapes and use these.
• Tapes are collected/delivered but a third party data storage company.
• This company stores the tapes and the monthly tapes will be retained up to our required retention period.

Overkill?

Maybe, but it does mean we can go back 3 months "instantly" and obviously go back further at sort notice by having the relevant tape delivered. Having 3 months written to the "monthly" tapes means that there is a duplicate backup in case there are any issues with a specific set of monthly tapes.

Our backup policy also requires that on a quarterly basis we request a tape set from the vault and test the restore procedure from the tape to the central backup server.

(We are also a small company with around 50 staff)

Wow, you must feel pretty safe! It's not overkill, but I don't think we will be going that far yet.
Seems like a great setup though!

I do like the duplicate backups of 3 months every month, so I think I'll try to implement that, or at least 2 months every month.

How do you make sure none of the files in your backup are corrupted?
For instances:
- Files that are rarely used and got corrupted on the production environment would overwrite previous good backups of that file. Do you keep a yearly rotation too to make sure you can always go back far enough? Or are there some other things you do to prevent this mess?

Thanks! :-)

As per above, we take a random sample and compare the tape to the production server. This ensures that what's on the tape matches what's on the server.

Or the files can be changed by a user legitimately, "file integrity" isn't a function of backup, the function of backup is to ensure you've a copy of what is on the server. If you're looking for "corruption" or file change detection then make use of something like "AIDE (Advance Intrusion Detection Environment)".


We have this run daily on each of the individual servers and report on any file changes found. We exclude things that we "know" will change, for example user content or log files. The reports from AIDE are checked daily and anything unexpected is investigated.


Monthly off-site tapes will be retained up to our retention period (7 years), although it's only within the last 2 years that I've been employed to implement this strategy so we've a few years left before we have to deal with disposal (which will be actioned by having the tapes physically shredded).


Also be smart when deciding what you need to back up from each server. For example if you have load balanced web servers where the website code is the same across them all then you only need to back up the content files from one of them and back up the log/config files from the others.


If you're backing up log files then make sure these rotate on a daily basis (especially if you're using any form of "differential" backup), for example if you have a log file that grows by 1Gb a day then ensure that it's rotated each day and that it's not a single file that grows from 1Gb to 2Gb to 3Gb etc. By the end of the week you'll have backed that file up once a day when it increments, so you'll have backed up 1+2+3+4+5+6+7 = 28Gb instead of just the "new" 1Gb per day parts (=7Gb). Saving 21Gb a week doesn't sound much, but when you're dealing with 50 servers that's 1Tb saved!