Recently I've added a small disk array to my home server.
Now I have plenty of disk space and it's still extendable.

So, the new idea is to build automatic backup solution, instead of copying manually from time to time some files into flashdisks, which are then placed "somewhere" (of course - "somewhere" means "nowhere" when I need it and "close at hand", when don't needed anymore).

May I ask for advice, what software should I check?
I see it as central backup server placed on ubuntu server, and some clients for devices as below.
I don't want another server (virtual or real)

What I want to backup (number on the list means priority, points 1 and 2 are necessary, 3-5 will be nice, but I can live without them)
1) home server (Ubuntu server built on Dell thin client) + disk array
/etc, /var, /log, or maybe entire / filesystem - all data storage is placed outside of root device and secured as much as I need. Maybe entire image of system disk is ok, but is it possible to create image of working system?

2) PCs - linux (mint) and windows
I have few users, everyone wants to backup own "something". Now we need to copy files to home server, I want to automate it.

3) two OpenWRT routers (primary - WAN interface, and secondary - extender for ethernet + access point for better wifi range)
As far as I know they can create theit own config backups, so maybe this is correct way if I can automate it and send to server. Not checked yet how to achieve it.

4) android devices (no iphone, for now, but may be - in the future)
I have nextcloud in my network, it servers caldav and carddav, so main need it to store photos/videos. Now I'm using external app and nextcloud. It works, but maybe real backup client will be better

5) Libreelec device
Homemade device, I didn't backup it yet. Will try internal backup things. However in case of disaster reconfig takes one hour, so very low priority here.

I would like to add two more points as to what types of backup you need:

You also need off site backup. About eight years ago my house burned down. I had two generations of backup in a detached garage from which I recreated my system. Since you do not want to mess with removable media you could set up off-site backup somewhere else across the network.

Ransomware will attempt to encrypt any attached backup that it can find. If you set up part of your backup on a remote network site make sure that ransomware cannot detect its existence and location.

As to backup software I just wrote by own backup scripts. Because I use detachable devices to place in my garage I can't automate the start of the backups. Also I could write the scripts in less time than I could find and set up a "universal" solution.

You're right, however, as I didn't have any automated solution yet, I need to start with my points first. I think about a place 200 kilometers from my home, so it is really off-site , but not in the first step.

Due to ransomware - it is a real problem. Is there any solution to keep fully automated backup safe?

whether local or offsite, rsync is my friend -- my solution was:

1. running openvpn server on offsite host, and openvpn client on local device, creating an encrypted tunnel interface on both devices
   
2. using rsync to manage the backup (--bwlimit flag very useful to prevent rsync from hogging bandwidth)--backup routes through tunnel
   
3. creating cronjob to manage when rsync runs

Securing a system is a long conversation.
Take a good look at the 'Sticky' posts at the top of the Security Forum.

One tip is to have the backup system login to the systems to be backed up, instead of the other way around.
IE You don't want to allow logins into the backup server, (except locally from the console) if possible.

Ideally only backup non-executables too ....
Ofc you could store hashes for them (executables) and only backup if no changes .
NB : exe = anything that can be 'run', so not just binaries but also shell, Perl, Python etc.

1 members found this post helpful.

Backula used to be one to suggest. Not sure what is out there now. Secure? Well, time locked file systems maybe. I tend to mount iscsi disks. I usually make an golden image of the working install or I just copy data files and rebuild the OS as needed.

I concur with rsync used for backup. To that end I have been using rsnapshot for years. I currently use rsnapshot to backup my entire system (Slackware) to another hard drive. I also backup a selection of files. I use cron jobs to start the backups. I do daily, weekly and monthly backups.

Are you sure that this is good idea? I mean direction of VPN?
If you establish server offsite, you have to store information about connection somewhere on local device. It looks unsafe in case of ransomware attack.
VPN server on-site and undescribed remote device which connects from time to time looks more safe?
Does rsync manage backup, or synchronization? I understand that backup means copying files and leaving them in safe place, not just synchronizing. It doesn't prevent you from ransomware or any other kind of data corruption/removal.
Of course you can script rsync usage and change sync places from time to time or sth.

I will check, thanks.
In such scenario - is it allowed to keep some kind of web interface for backup system? especially restricted to my network only (I don't know exact solution, just thinking)?
I have possibility to establish remote site. What about a script in the remote site which once per day mounts one of two backup disks, then connect via VPN to my local server and backup what I want to backup and then unmount?
Unfortunately automatic backup will be done even in case of ransomware attack (other scenarios will be covered).
Is there any good way to recognize infected system? And if it is infected, remote system may be disconnected, may unmount all disks or even turn itself off just to protect data (time of restoration has much lower priority than data existence).

I can create some set of data with well known checksum and recount it and compare each time, when backup system asks for connection. But infection may be in progress and my control data set may be not touch yet. Is there any part of OS which is usually attacked first?

Sounds smart. If executables really need backup, it may be zipped with password for example.

rsync does not manage backup. You can use rsync to copy the files that are new or have changed to the backup device. Files that are still the same as they were in the previous backup are not copied. You can set up a rsync command giving the path for the "from" file system to the "to" file system. You need some sort of control program to call the correct rsync command(s) at the appropriate time.

That is a good idea.

In a ransomware attack the infected computer freezes up. I don't think that a remote site could connect to an infected computer.

On second thought, once a ransom is paid the thieves give the victim a key which unlocks the encrypted data so the infected computer probably has some functionality.

I see I have much to read about rsync.... May I copy new and changed files to completely different location? (I mean compare files from "live" system with /BACKUP directory and if there are new/changed files - copy them to /NEW_BACKUP dir, not touching /BACKUP)
It will be perfect to make incremental backups...

It may be not so easy during the attack process. And - how infected system will inform user about the attack if it's frozen? Such message is important part of attack.

I didn't spot any ransomware attack (yet?), I just don't know.

The way that I handle that is to have generations of backup. The current rsync backup is run against the oldest backup generation. Thus I perpetually cycle through the backup generations. On one computer I have four generations of detached backup and on the other computer I have three generations of detached backup. I also have one copy of the current backup for each computer in the garage.

I would have to dig through the rsync documentation to see if backup could be done the way that you describe.

1 members found this post helpful.

Of course I can do it by myself. As you have experience with rsync it was possible, that you just know it

Thank you.

Any remote interface can be hacked, so no, not a good idea (imho)....
Really securing a webserver is very hard.

If you really want a remote login for management purposes, you could use eg ssh+key, but have sshd down on the backup server and get someone local to that, that you trust(!) to start if needed ...
You could ofc go to the trouble of using a vpn and only have you (& your wingman) actually have the vpn SW.
You'd need to keep that updated ofc ...

As mentioned, Security is very deep rabbit hole.
You have to balance security vs time/effort to setup/maintain/use vs 'likely' threat levels.

I have two 100% trusted persons on the other side, but with zero technical skills. Asking them to type a command is a challenge. But maybe its a good opportunity to use spare RasberryPi and create some buttons with led indicators - to toggle and show status of ssh, webserver, networking, power (?), disk mounts. I like such things, so it's a relax, not wasted time

I think this RaspberryPi may be also whole backup server even.

In case of ransomware... well, I have a backup on the remote, in a worse case scenario--I wouldn't pay, and I'd just restore my backup to new device...

But I also am very careful about what I download and click on... I pretty much stick to 3rd party apps that are at slackbuilds, and the maintainers of the slackbuids.org repository are selected based on skill and reputation; and I've never had any trojans from a slackbuild.

But I work for a school that sends random documents to fill out and sign, and so far so good, but the school is an attack vector, because I feel obligated to click on files sent by my employer.

But I don't think it makes a difference whether I run the server locally and have the remote connecting, or vice versa, because openssl certificates are stored on both sides. But I suppose it is true that the remote device is not clicking on anything, and so is less likely than a daily driver to download and click on any malware.

I do encrypt entire file system, so losing a local device doesn't yield anthing to anyone who finds or stole it.

In case of the certificates issued for openvpn, I have a root CA that never went online or had network interface connected to anything at all--it issued certs that were then transferred manually on usbstick to an online intermediate CA that signs certificates for use by openvpn. I followed the basic setup described here.


As far as rsync, it took a long time to intially backup... but using "--exclude=" flags, I can control what I want backed up, and don't waste time on the OS,or browser caches, and focus on just backing up essential data. It can checksum files, and only backup ones that have different sums... it can also be set to keep the old and give the backup an incremental suffix--this was helpful when I had multiple portible devices, and working on the same project, did one task on one and another on the other, and didn't want rsync to replace either task... since then, I just connect to the remote device and work directly on it, so that data is actually on the remote, and I don't even have local copy--that cleaned up things considerably, as far as papers or code I write... but when it comes to family media, like videos of my children... that is what is getting rsynced around... It's mainly sentimental, and not nation state secrets, so I am more worried about drive failures and loss of devices than being a target.

I think if I had more valuable data, there's probably all kinds of holes in my security that I haven't considered--like quantum computers just deciphering all conventional cryptography.