We are using RedHat Linux on lots of computers with about ... 15 different configurations. (Configuration means some run FTP/some done, some have only 1 user account, some have 3 user accounts, different mount-points on each, etc)

Is there software out there that I can use to test a new machine to make sure it is setup/provisioned properly?

The goal would be some software that can run as root on a new machine and I can feed it a configuration file for the "fubar" machines, and it checks and confirms/warns when something expected is not found.

It would also be used if/when the machine is given a new IP address or the network card changes. (Existing SSH entries do not like it when you change a IP or network card).

Is there an off-the-shelf solution to do this check? If not, I have to write my own in Perl (not a bad idea as I am a fairly good programmer.)

Before you say it, Yes - we know there is provisioning software and blade-management software that helps create new systems. We use some of these for the basic disk image.

The problem comes in with things like user-accounts and SSH keys that need to be created on the new system and pushed to 10 existing systems, (and vice-versa), re-compling Perl DBI, DBI::Sybase/Oracle to match the current version of our database, setting mountpoints, insuring minimum disk space allocation in some important dirs, that the tape-backup account has sudo rights, etc.

Get the idea - its the stuff that comes AFTER the gross Linux installation that needs to be checked.

We currently spend about a week after each new system goes on line fixing these types of issues.

Is there an off-the-shelf provision-checker or do I need to roll my own?

I'd take a slightly different approach and look at using a configuration management system to push the setups you want to the servers, and building packages rather than compiling etc. directly on the systems.

cfengine, lcfg and radmind are all designed for config management, but many people use source control systems like subversion to distribute config files from a central repository. For quick packaging, the checkinstall utility will monitor a source compile and then create a package that performs the same job.

I think that the main thing is to treat any manual configuration as a issue to be fixed by automating the task or even removing non-standard configurations, rather than doing it by hand and trying to check for all the possible errors afterward. It can be a tough discipline to stick to initially, but it's worth it.

your use of 'configuration' is a bit vague and considering that you have 15 different configs, there will not be much off the shelf stuff.

I look after lots of linux boxes and just tar up the key files and push them back again on log-off and log-on... allows some freedom, but keeps the core profile in tact with no input from me after the initial setup.

You can skip files that need to be dynamic... ssh etc and just pack those you need to manage the systems... oh, and it is free

Regards
Richard

Thanks for the reply hob.

I have already suggested we try and 'standardize' the config of our systems but we cannot dictate the procedures in another group (yes, we have another group in charge of buying/installing/maintaining the computers). We also have the problem of a team of engineers constantly updating software and creating new dir structures or jobs that require a shifting amount of services. (I do this weekly).

My group is the one that gets pages at 1-3 am when some nightly script errors out because someone forgot to start FTP on a new machine, or the SSH keys were generated, but only pushed to 3 of the 5 other machines that need password-less access.

(Oh - and permissions. Don't get me started on how often a new install created dirs as "root root" when they should have been "fubar fubar")

So it looks like I get to roll up my sleeves and write my own tool.

The side effect is that I will be forced to learn the config of all our different machines. That alone could help our network group come up with "standard configs" for each.

FWIW, you may be able to make your life a bit easier by building a sandbox or contained environment so that what the engineers do with their apps is separated from the main host OS. They might like having greater direct control of a chroot or virtual machine as well.

Admins not checking the configurations that they set up is really a non-technical problem, so rather than trying to write a tool I'd look at documenting the problems as they occur and using that as ammo to push for a change in the situation. My experience is that admins are careless when assigned to jobs to that they don't want, and the only fix is to move the job to someone else .

We get some machines built on a regular basis and these are fine. But some of our other boxen were hand-crafted 6 months ago by the department manager (and then my boss added customizations for it's new mission). Now that we need some more of these, the task is assigned to one of his staff with instructions like:

"It's just like the Benthoven box's, but with 2 network cards. Go look at benthoven01 for a sample."

Our admins do an honest job with the base system. But the nuance of the "/data" directory needs to be owned by "tom dba" and the /archive/table_dump directory needs to be owned by "harry root" are what get missed.

Since we write the applications that depend on things setup by "us" months ago, we have nobody to blame but ourselves when things fail at 1 am.

So the scenerio might be more accurate this way: We created a prototype system months ago, and suddenly it has become "production". Network gave us 4 more boxes (1 development, 3 production) that they claim is setup similar to the one box, but we have to sanity-check things for the 20-800 scripts we plan to run on the thing.

I suspect a new box is an opportunity to install all the latest versions of Linux, admin tools, NIC's, mount points, versions of perl, Oracle/Sybase drivers, network appliances, etc. The prototype box also has dev tools/test/experiment stuff that they dont want on the new box's.

Looks like I get to write more code.

Classic Example:

This just cost me about 4 hours of work.

We have 2 Linux servers that I have written scripts to go and do things with using SSH.

They recently created 3 new box's that "should" be identical, 2 for production and 1 for testing. I tried to get our software to work on the test box and it throws funny errors. I tried to get our software to work on the 2 new production box's, same error.

The software works great when reaching out to the production box's.

Turns out - the production box's are configured to only use SSH2. None of the new box's specify the protocol for SSH, so it defaults to SSH1.

I spent lots of time making sure SSH was setup, creating no-password access, etc., and nothing worked. Finally, I found that my software would let me specify which protocol to use. Playing with these options showed me the config difference.

I have added the SSH Protocol check to my list of things to test.

Note: Kudos to linuxquestions.org! It's search feature led me to an article that explained where the SSH protocols are specified and I was able to prove the config difference.