Hi everybody,

I work in an SMO as a sysadmin and I am revising our backup strategy.
The first thing I am working on is our Git repositories.

I am looking for some second/third/... opinios to validate my thinking.

The history
We have a centralized backup server that copies backups over SSH using a pub/private key pair.
The backups used to be done from our Linux servers over NFS.
This caused immediate danger because every Linux server would have write access on our backup share (with no_root_squash).
Anyone with root/sudo access on the server could completely wipe/overwrite/encrypt the entire backup share.

(Note: We have sepparate VLANs for our server and pc network, only the server network would be allowed over NFS.)

I plan to do all backups from the backup server over SSH and have the backup server on a separate VLAN which is not accessible from our internal network.
Access from the new VLAN to our internal network would be allowed to take the backups

I take a backups of our Git repositories every night over SSH (using git clone --mirror).

The problem is that we could lose a full day of work or in the worst case scenario, two days (e.g. disks failing during backup).
Losing that much data is unacceptable to management because there are a lot of commits happening on any given day.

The solution
The solution I have come up with is to create a git post-receive hook that pushes changes to a mirrored backup on our backup server on every commit.
That way we would have a full copy of the latest state on our backup server.
The nightly full backup will still be triggered every night over SSH and kept for 7 days, 4 weeks and 3 months via rsnapshot.

The questions
If I were to create a post-receive hook, the git server will try to push changes to our backup server.
This means I would need to give git write-access on our backup server in order to push the changes.
Because it is automated, I would probably need to setup a passwordless private/public key pair on the git server.

- Are there any better alternatives I could use (e.g. polling from the backup server?)
(For Git this would work, but SVN backups take hours over night, so it might not be possible for SVN)

- If set up the backups with a pub/private key pair, this would open up holes for a potential attacker, is this an acceptable risk?
* If it is setup this way, should I create a sepparate account for every user in LDAP with write-access only to it's backup directory (instead of root).

Thanks for any input!

EDIT:
I forgot to mention, there are weekly off-site backups, so we will lose one week op data tops if our entire backup server is trashed.

Regards

My feeling is that a nightly backup should be good enough. In order to prevent losing a full day of work, it's better to use RAID redundancy than try to do backups in near realtime. Of course, that still leaves the risk of user error doing something stupid and wanting to take back the last five minutes of rm -fr. But that's why you're using version control rather than just directly editing production, right?

But that's just my opinion, not the opinion of your management, so ... shrug.

Getting back to your actual questions...hmm. I feel like you're running up against some of the unfortunate limitations of using ssh. Creating a bunch of accounts to limit access is a pain, compared to nfs security (just create different export entries for different IP addresses).

To avoid these issues, I think it would be better to have the backup server pull everything, rather than let anything be pushed to it. If you don't like regular polling, you could set up an extremely limited ssh account which acts as a trigger to tell it to poll for updates. Set up the ssh account to only be able to run a command that touches the trigger file. The backup server regularly polls the trigger file to see if the timestamp has changed. If so, it polls the other servers for updates.

Since this extremely limited ssh account can't directly modify any files on the backup server, it's safe for all of the other servers to use just that one ssh account. So, each server has a post-receive hook to touch the trigger file.

Or, you could just set up extremely limited read-only nfs shares on each server. Each server has its own trigger file that gets touched after each update. The backup server regularly polls those nfs shared trigger files to see if it should pull a larger update. This requires some bytes of network bandwidth per minute, but...come on. It's practically nothing. There's practically no security risk from a handful of read only nfs shares with ludicrously little data available (just the timestamp of a handful of trigger files, really).

Thanks for your reply Isaac!

Our servers are all running RAID arrays. The GIT/SVN server was running RAID 5.
About two weeks ago, two disks died around the same time causing all data to be lost.
There was about 1,5 day of data missing from som SVN repositories, causing the decision to be made about near realtime backups.

The new disk set I have ordered will be setup in RAID 6, but a raid controller or more disks can still die at the same time, so unfortunately, this will not be good enough (RAID is not a backup after all).

I was thinking of perhaps setting up a mirror server that is not the backup server, but that can still be compromised if an ssh account is used to push updates.

The file-touch method seems like a good alternative to polling. I will have a look into that, thanks!
I will have to do some research on how to limit ssh accounts and nfs.

The places I have worked at have been doing tape backups for many decades. Currently we place each day's backup or each weeks backup in an off site location or in a fire resistant box in a secure area. We might keep 60 days backup or a full years backup. There is almost zero chance that we'd get a ransomware attack.

I was foolishly installing usb drives on a not-for-profit that I volunteer at to prevent data issues. I had it set up for idle backups till I realized that these ransom attacks can encrypt any attached storage. Now I'm back to going over there and doing backups.

You can use authorized keys to have the backup server contact the primary server and run a preset command and allow no other access

Eg from the primary server to the backupserver
What the below does is backupserver says to primary server, "All I'm authorized to do is tell you to run your predefined command for this key" and ends the connection
Primary server then runs a rsync command copying its local directory to the backupserver at location /backup_location/primary_serverA

Then, configure (/etc/rsnapshot.conf) rsnapshot on /backup_location.

You can have hourly, daily, weekly and monthly backups. I personally take 8 bi-hourly backups (16 hours coverage), then misc daily, weekly, etc.
Once setup, there's a lot you can do from here. rsnapshot on the primary server for easily accessible restore (won't help with disc failure!) Setup backup Server to contact (backup from) multiple primary servers?
It's also good because rsync can minimize your bandwidth used, rsnapshot minimizes the amount of disk spaced used to only changes made.
I know rsnapshot can handle remote backups and it may be better then my method, look through the docs.

To clarify my suggestion - I'm actually suggesting the use of polling still, but polling that requires practically no network bandwidth overhead.

My first suggestion, which actually I think is a bit overkill now that I think about it, was to use polling on a local file - that requires no network bandwidth. But the method of creating an ssh user to run "touch /PATH/FILE" seems excessive, I think.

My second suggestion, which I like a lot better when I think about it, was to create a read-only nfs shared file on each server. This requires a negligible amount of network bandwidth for the backup server to poll, but it's simpler to set up and maintain. No need to set up any new users. Just set up /etc/exports on each server for the poll directory (set to read only), and create an entry for each nfs share in /etc/fstab on the backup server. Since there's a separate trigger file for each source server, the backup server knows specifically which server to pull updates from.

jefro: We are using a set of USB drives that are not connected to the machine. When a randsomeware would try to encrypt the server, we would (at least) have 3 sets of disks with weekly backups available. Granted, there would be a lot of data loss (possibly two weeks), but I hope I can mitigate this by isolating our backup servers via a firewall.

Sefyir: The limited ssh commands via authorized keys seems like a good plan too. This would still require the primary servers to have access to the backup server however. That is something I think I would like to avoid altogether. Perhaps I can still use this to setup a secondary mirror server for our services so we can have an immediate spare server of our critical applications.

Isaac: I will really take the read-only nfs shares into account, it requires some manual work, but I'm sure the bandwith saved will be worth it.

Thanks for your input guys!

I suggest that usable backups must occur much(!) more frequently than "once a day."

There are many online articles which discuss the use of rsync to create on-line backups similar to Apple's Time Machine tool.

The main thing that must occur is that the backups must be multi-generational, and that they must be stored in a directory that is not accessible to the general public. The directory should be owned by a user-id that does not permit login, and it should be readable and writeable only by that user.

I agree with multiple rsyncs per day, and have separate target dirs for each day of the week etc as per your overall backup strategy.
I also think having the backup server pull from the dev servers (rsync works over ssh) instead of the other way around is more secure.

Not difficult, just switch it. When PrimaryServerA contacts the backup server, it can only do "Please backup from me"
Regardless, I have found it to be extremely effective since it is incremental (only changes take up space) and pre/post hooks allow you to run scripts to allow you to capture databases / etc as well.

Eg here is output of rsnapshot du. As you can see, there were a lot of changes about 2 months ago. You can change the frequency in /etc/rsnapshot.conf and in /etc/cron.d/rsnapshot.
This has saved me many times as it allows me to reach back a few hours and grab a single unedited / deleted file