Infrastructure versioning
 

Howdy cowboys,

I asked this question on StackOverflow for a programming perspective, and the responses I got didn't seem to appreciate the situation, which I think was becuase they were only aware of a programmers view of the world, so maybe some fellow sysadmins can be more helpful...

I'm architecting an infrastructure for a web service, covering network design, CentOS build configuration, bespoke application deployment, KVM builds, iptables scripting, Windows images, Nagios and all sorts of other things. These naturally seldom know about the other parts of the system, and really have nothing in common, however we need to be able to "version" the majority of the constituent parts under a single release number automatically. How can I do this? All scripts and bespoke software are to be held within SVN, as will (most likely) place holder projects for other things which are not scripts - e.g. CentOS repositories.

I was looking at using the post-commit triggers on SVN to say that when an project is committed to, then a script will find what wrapper packages it is declared to be relevant to (e.g. "this iptables script is used on VM1 and VM2") and then force a commit of this parent package, updating the current SVN versions of each package it holds into that package. This could then happen over a few more layers up until a point at which a master package is updated. This master package version would then be made available to developers and testers so they can request an infrastructure environment of an exact age / specification to test with.

Does this make sense to anyone here? What should I actually be doing instead of the daft idea I've come up with so far? The other variant I'm immediately aware of would be to just use a single SVN project and just use a static folder hierarchy, which whilst being simpler doesn't seem quite what I'm after. But it *might* be, depending on my understanding of the SVN model iself. The closest thing I'd had mentioned to me was SVN Externals, which I'm sure is basically the opposite of what I want.

Thanks guys, now gimme the warm fuzzies...

I'll start from the wrong point...but this is mostly pointless waffle, so I don't see how that can hurt:

Yes, absolutely. I'm a bit unclear on how many times you would want to run this (I was about to write 'how many boxes you want to run this on' but maybe some of the boxes are virtual). If you were only going to do it three or four times, putting a lot of effort into reducing the human involvement at each run wouldn't make sense; three or four thousand and it would be essential.

Given that I'm a simple sort, I'd try to work out what you couldn't do with a simple approach (and I don't understand SVN...to someone who does, the landscape probably looks different)...BTW, I've a suspicion that you can probably 'make' everything that you want to do, but then I'm not good enough with that, either, but 'making' a kickstart script sounds quite possible.

• Networking is always an issue...if your boxes can pick up IP addresses from dhcp, that can make networking easy to solve, but you probably don't want to do that (to handle the 'everything has to be exactly the same apart from the bits that don't' issue). Otherwise you have to arrange for the boxes to get stuff absolutely predictably, except for the networking set up.
• Can you (could you) use kickstart to do the server provisioning and grab a build version file at build time? what if someone then updated a package selectively? And would writing your own file in, say, /etc with your version number be enough, or do you have to do more than that? (Maybe adding the identifying details of the box and the version that it got to your master database?)
• Have you considered caching the stuff that you use from the repos? Not only do you save bandwidth, but once something is available, it will (should) stay available. And what if there is a non-availibility issue when you run your provisioning script (I guess that comes down to being careful with exit codes)?
• I'm probably overlooking stuff, but the iptables script seems like one of the less challenging bits to do (less challenging, particularly if you already have a somewhat usable 'one shot' iptables script, not being the same as trivial or zero effort,)...largely because it is a bash (...other shells exist..) scripts, and once you've got the script running, it can do all of the querying and conditional testing stuff that you'd like

Oh, bugger! there was something about windows in your original post and I can't possibly comment on that, except to say that the I've found that the further I stay from that, the happier I am, particularly for anything serious, that I want to work.

This is on a large scale, about 1000 VM's at any one time, rebuilt weekly etc... It's actually a web service you'll know of, maybe even use regularly.

In terms of how each individual piece is done, there is less of a concern. I'm currently thinking about how you would handle storing kickstart scripts in subversion and have some php script to whip out a specific version of a script dynamically, and how you get a VM to boot using that kickstart url via cobbler... not sure how that'd all work, but fundamentally the issues I have are how to control the versioning of these disparate pieces, not the pieces themselves, yeah?

as an aside, we would be naturally caching repos, an angle to further that is that we will need to install from a point in time snapshot of the repos too, for consistency. Not sure how to do that, but I'll suss it out somehow.

I now better understand where you are starting from...

Do you really need to do that dynamically? Can't it be static within each build version?

well things like "each build version" make it dynamic. At any point in time a non technical staff member can request a deployment of VM number xyz of build abc, so there's always a need for some level of dynamism, but it's a case of dynamically pulling on a static file, not creating it dynamically.