I used to consider myself a linux newbie. But now it's 3 years later and I find myself in charge of 4 dedicated linux web server machines in full production use, plus my own development box.

Managing all the patching, security monitoring, configuration changes, log files, logwatch reports, crontabs etc is starting to become a bit of a chore.

Does anyone have any tips as to how I can more effectively manage multiple linux machines? These machines are all on different networks and in different locations around the world.

EXAMPLE 1: Are there any tools for combined reporting?

EXAMPLE 2: Is it wise to set up systems that rsync's configuration file changes around and deploy software automatically across all machines, or is this more likely to make a maintenance headache?

Any suggestions appreciated.

Bash scripting is your friend of friends. It will combine your log files, do checksumming and global updates, and clean your shoes.
Look at everything you would want to do automatically. Then break it down into logical steps. Then convert those steps into bash scripts.
A little "hard work" is always necessary, but the more you can script, the more free time you'll have to kick back and relax.

Actually, it was to solve almost exactly this kind of prob that Perl was invented
If the systems are pretty std/common layout, then yes, having a central server push out (via rsync) new configs does work. Seen it myself at a major intl merchant bank.

Thanks. All sounds reasonable.

So what kind of thing is it common to automate, or is this really bespoke to every individual setup? What are the prime candidates? Things I'm thinking of include:

• Software updating via package management (my systems are Fedora boxes so cron'ing yum would be a start) - or is it best to do these manually so you know exactly what is changing on your system? Or perhaps to exclude certain packages from yum update's and be forced to do them manually as required ??
• Backup routines - creating tarballs of each user's home dir and saving it off to a central backup location, plus backing up all the databases schemas and data
• Rsyncing backup data off to other servers for safe keeping

For server to server communication, I presume rsync is the tool of choice? Main benefits of using rsync as far as I know are that it tunnels through a secure connection such as SSH so there's no need to open additional ports on your firewalls etc, and that it only transfers the data that has changed etc, so that it massively conserves bandwidth. However, are there any security related or other concerns I should worry about if taking this approach?

Also, any ideas as to how it's possible to securely backup user information (and preferably passwords too) to another server without creating a big security hole, or is trying to do this a big no no ?? And what are the risks of storing backup data from server A on server B, if any?

Thanks