How do you folks working in the big enterprise control configuration management?

By configuration management control I mean a process of ensuring equipment and devices are in a known state and condition rather than relying on implied or perceived knowledge.

I am not looking at configuration software, such as Ansible or Puppet. I am looking at documentation and control.

I am referring to changes with servers, routers, firewalls, APs, power controllers, etc. For example, how do you have confidence that the device listed in the web browser interface as connected to outlet No.1 on a remote power controller is actually the device connected? Or, how do have confidence that the server connected to port 8 on a router is the actual server? How do you know that all routers have had firmware updated? How do you ensure device configurations are backed up some place to avoid Keystone Kops when the device fails?

I work for a really small but successful company. I get to tinker with Linux systems. Yet just about any employee can change things in the infrastructure. All without documentation or informing others. Seat of the pants kind of thing.

This seat of the pants approach worked well enough for the previous owner who played the role of chief cook and bottle washer for many years. The new owner wants to expand the business. We both foresee the lack of configuration management control and documentation eventually haunting us. I have installed DokuWiki and that has been well received, but we lack any means of knowing what we think is configured in the field is actually what is configured. We do not have a ticket tracking system, but the new owner has me looking into that.

I am NOT asking for judgment of business operations or any soap box wisdom. Researching around the web indicates this style of management is rather typical for small companies. Profits, convenience, and customer satisfaction come before security and documentation. "Just fix it."

I'm just looking for ideas with how you folks in the big shops handle this. I realize the operations of a big company differ from a small business, but looking and asking nonetheless.

Thanks much!

You might want to research the terms "continuation document" and "change control". If you are not using an automated tool to set and document configurations, then a manual document (or digital with a paper printout) is important. Changes in configuration should be justified and recorded for risk control as a matter of course.

1 members found this post helpful.

There are systems that do such things for larger enterprises (Tivoli from IBM is a good example...TSM is one small part), but they cost $$$, and have the associated data-security departments behind them, along with resources to monitor such things. Typically, it's structured like:

• Data security designates admin-level users based on hires/department managers, along with associated paper trails
• Those users are added to systems locally/LDAP/whatever sign-on system is used
• Changes to systems are monitored with things like Tripwire, and compared against the written change-control forms, that have to be filled out by admins/users, submitted up through managers for approval, before work can be done. At least, done without getting yelled at or fired.

They're of course, more complicated than this, but that's the gist of it.

But I've been where you are, and for small-but-successful its best to get buy-in from the company owner, and have him put the LART in your hands as the administrator, with you designating others to make changes too as you see fit. Shut everyone else out from being able to do anything, and get SOME process in place for folks to submit work to be done. Could be as simple as an email to you, so you have a paper trail. Forward to boss/whoever for approval, archive reply, and do work.

And I'd restrict SUDO shell to only the most trusted, so if someone DOES cowboy something in, you have a history, and can apply the LART as needed. Nothing like a head on a pike outside your office door to set the tone about "let's not cowboy stuff in anymore, huh?".

Thanks for the replies. I don't feel too bad about the thread as I have been accumulating notes along the same lines as the replies. Eventually I'll massage the notes into a short proposal to the owner.

Part of the proposal is we simply embrace a work culture of configuration management and abandon the old seat-of-the-pants approach.

The good news is the new owner wants lots of documentation. I should be able to sell the idea of configuration management as a tangent. The goal is not to smack knuckles but to know what is actually configured.

This is a really small company and the owner is hands-on. My goal is to provide configuration management. Doesn't really matter who makes a change, just that we know and the change is documented.

I need to script grabbing nightly backups of Mikrotik and Ubiquiti devices. Then script a way to compare backups and send an email report of changes. When there are changes send everybody a reminder note to document changes.

A ticket system will help, but will not be foolproof.

Our server backups are primarily rsnapshot. I plan to script something to compare rotations and send similar reminder emails.

I might send email alerts whenever anybody logs in to devices or servers. Much of the time the login's do not include changes -- just checking things, but the intent is to provide a head's up for the next day if I see changes in the email reports.

I think that is about the best I can do and hope for. That's probably good enough too.

Look into Tripwire...they have an open source version and a pay-for version, and I swear by it (and sometimes AT it ). Configurable, and it can email you a list of files that have changed either permission wise, size-wise, modified, etc., etc. Good to reference, since if something DOES change, you'll be able to recover a file from the previous night, and compare.

Coupled with a decent sudo'ers file, you can find out who modified it too. And if it was ME putting this proposal together, I'd make sure to mirror the syslogs to a machine that only you and the owner know about. Yes, a bit paranoid maybe...but if someone edits sudoers logs or tries to cover their tracks, you'll see it and be able to deploy the LART.

I looked quickly at Tripwire. Packages are available for the distros we use. Seems there is also a fork called Aide. I need to look at other tools too, such as logwatch, etckeeper, auditd.... All new territory to me but I suppose just walk one step at a time.

The focus is not the trustworthiness of employees but not knowing what is actually configured. Without any kind of program, policy, or monitoring we have no way of knowing and remembering when things change. In this case the challenge is simply a lack of configuration management awareness. My challenge is introducing configuration management at palatable level for a really small company. And I'm included -- for example, I want to remember when I change things on servers and why.

Yes, but none of those are hard nuts to crack. I'd shove in a centralized system just to capture all other syslogs, and run logwatch/splunk on it. Make things easy for yourself.
Understand, totally, and that program fits those needs. It'll let you know what changed, when...you can track down the "who" from there.

What do you use to mirror logs? Or do you use remote logging?

Syslog-ng, nothing fancy. I typically configure a central log server to have a tree like:

/var/log/remote/machine1 (machine2, 3, etc.), and have the logs separated out into separate
directories, with file names related to the levels, such as .warn, .info, .mail, etc.

Configure Splunk to look in those directories and you've got a one-stop shop.

Thank you.

I was going to start a new thread but I will first try here for some results.

Other than ticket systems, is there software available to help a sys admin document work performed?

Where I work we do not have a ticket system installed. I proposed as much to the owner. I believe that is going to happen eventually but not soon. Until then I need something to document server changes. Yes, a text file will suffice, and a LibreOffice Writer "form" would be better than a raw text file. Just wondering whether something else exists that provides more helpful information.

I need something, not just for configuration management but a little CYA too. Perhaps I should just install my own local ticket system.

I'd suggest OTS, since it has a built-in knowledgebase feature. Document what's done there, edit as needed, and it gives you a ticketing system too. Personally, I'd also print those FAQ's and put them in a binder. Online systems are great...right up until THAT system crashes, with all the docs. A three-ring binder or two is ALWAYS online at dark-O-thirty...

For the first ten years of my IT career I kept two kinds of long-term documents: 1. on each server a "continuity document" that specified the hardware and major software products and configuration, with each change performed dated and documented, and 2. a project book with each project defined, start dated, and progress notes that summarized the daily or weekly progress and most important details. I had field engineers borrow my continuity documents for understanding, documenting, replacing, or repairing a server. No one actually asked for my project notes, but they enabled me to answer questions no one else could answer about what was done when and why.

Although that system served me well, it was very manual and I really recommend keeping your documents and notes in cloud storage and well backed up and secured. A single hard copy with your offsite backups of the matching critical systems Disaster Recovery backup images can have value, otherwise avoid paper.

A proper change control, documentation, and configuration management system is really a better answer, but may be overkill for a small company. The bottom line is that any documentation is better than none, and documentation you can search, update, and use when your job or business is on the line is better than the bets answer in the world when you cannot get to it.

Do your own risk analysis, and work with what you have to protect the business and your career.

With respect to "continuity documents," one of my first projects on this job was creating a "system-audit" script. The script is specific to our systems -- meaning there is a smidgen amount of hard-coding for our use. I run that script in a nightly cron job to create a DokuWiki compatible txt file. I have another script and weekly cron job that grabs and pushes those txt dumps to our server running DokuWiki. I use the "include" plugin to create the Dokuwiki pages using those pushed txt files. Thus the wiki is always current with respect to bare metal server servers.

Did you mean OTRS?

My short list of ticket systems proposed to the owner included 1) OTRS, 2) osTicket, 3) Request Tracker, and 4) Zammad. I am not excluding proprietary systems but I thought looking at free/libre projects first was just common sense.

As I have to research and test those free/libre ticket systems, perhaps the answer to my question is forego an interim solution. Install one of the ticket systems and start documenting the work I perform on each server.

Yep...been there.
Hehehe...actually, I *MEANT* osTicket, but apparently couldn't make my fingers type that out.