Creating a custom CentOS VM image for KVM

PeterSteele · 09-10-2012, 03:36 PM

I'm using the CentOS kickstart facility to create a base image for additional VMs I intend to create. I'm not completely satisfied with the approach I am taking and I was wondering if anyone might have some suggestions.

I create my base-vm.img using virt-install in conjunction with kickstart. I use a single 4GB partition and after the install is done, I make a copy of the partition using dd:

dd if=/dev/sdc1 of=base-vm.img bs=16k

I can then use this image to pre-install on drives (again using dd) and create VMs very quickly without having to go through an explicit OS install phase. I just tell virt-install that it can boot directly off a drive where I've pre-installed my base VM image.

The problem is that I want to tweak this base image a bit for the VMs I am creating. The base image is a small 4GB img file providing space for the entire file system. I want my VMs to in fact have /var mounted on a separate drive with a ten GB partition. Plus I want to define a 4GB swap partition.

The trick I am using is to do some prep work when a VM boots for the first time. The VM is configured to know it is booting the first time, and it goes out and configures a swap partition on /dev/vdb1 and a /var partition on /dev/vdc1. It then copies the existing /var contents to the /etc/vdc1 partition (after mounting it) and updates /etc/fstab to add entries for the new swap and var mounts.

Once this is done, the VM reboots itself and on reboot it will have a new /var partition mounted on /dev/vdc1 and swap space mounted on /dev/vdb1. This all works well enough but I can't help but think there is a better way. Can anyone suggest an alternative approach to accomplish what I need to do? Ultimately what I want is a base VM image as small as I can get, while at the same time providing a means to expand the size of the VMs that are based on this image.

acid_kewpie · 09-11-2012, 08:22 AM

I'm not a fan of imaging if it can be avoided. I built a system somewhat similar to this a few years aog for a major client, and stuck with kickstarts throughout.

With a little love and affection you can craft a nice small kickstart to build a consistent base install which is always easy to update and modify. I then used puppet to perform personalisation of each host as part of the %post section of the kickstart.

The build phase of the system was mainly rooted in cobbler, and I only needed to use a specific name format to identify everything onward for what kvm config was required, what subnets the vm was in, and what build type it should have. Nice. Well, i thought so.

These sorts of solutions are always a little bit homebrew, as everyone wants something different but I think I had a happy middle ground.

PeterSteele · 09-11-2012, 09:10 AM

I considered going with kickstart for the whole thing. My main issue is that I have to automatically create a lot of identical VMs and I want to do it as quickly as possible. I can do this with img files, since the majority of the time in bringing up a VM is the OS installation phase and with img files I can better control how long the OS install takes. I can blast the identical img file out to multiple partitions very quickly, and can have a new VM up and running in under two minutes.

In the end I guess we all have our requirements.

acid_kewpie · 09-11-2012, 09:15 AM

I was getting times of around 3.5 minutes to fully deploy an ESX based VM from a few wrapper scripts, which I'd say was pretty jolly good.

Possibly interestingly, one thing we did do with images was a windows server, but still via kickstart. By putting a wget in the %pre section and bailing out as soon as that was done, we used exactly the same deployment method for linux and windows. hacky I know, but very simple and robust.

PeterSteele · 09-11-2012, 11:37 AM

Where did you wget the file from? The host OS?

PeterSteele · 09-11-2012, 12:57 PM

Actually, this idea intrigues me. Can you elaborate any? A kickstart would normally of course be used for installing CentOS/Redhat, but I assume that instead of installing CentOS, you grabbed a pre-created Windows image file and copied that image directly onto a partition that you create in the %pre script?

The virt-install command requires that you specify the --location option and point it to your installation iso, if you want to use a kickstart file. I assume you just specified a dummy iso for this since in fact you were intending to install a Windows OS, not CentOS?

acid_kewpie · 09-11-2012, 03:37 PM

ahh well you need an iso to boot the stage one in the first place, but you abort abort abort before you start the install real. This is the templated kickstart I wrote...

Code:

# Kickstart file for DRM server. Clearly NOT actually installing Linux here...
$SNIPPET('set_revision')

url --url=$tree
lang en_GB.UTF-8
keyboard uk
text

%pre
$SNIPPET('pre_anamon')
echo `date +%H:%M:%S` "CUSTOM  : Installing DRM image to /dev/sda" | tee -a /tmp/anaconda.log

wget http://build/infrastructure/$revision/drm/D1-disk0.gz -O - | gzip -dc > /dev/sda

echo `date +%H:%M:%S` "CUSTOM  : Imaging completed, reboot!" | tee -a /tmp/anaconda.log

# Last chance for anamon to forward log
sleep 10 
poweroff

as I said already, I had an overall wrapper script watching these builds and polling for the vm power status, doing reboots etc. (which was one of the uses of the anamon (anaconda monitor) tool that's out there with Cobbler. It actually built a 4 vm replicated environment simultaneously. if it wasn't me that wrote it, i'd have been impressed!

virt-install was abstracted using func, which has a direct module for kvm. So once the physical machine was built, we could happily call python libraries for func on the build server and know the magic was happening on one of a 100 target boxes.

PeterSteele · 09-11-2012, 04:21 PM

Thanks for the post. This general approach might work for me. I've got some experimenting to do...

Peter

Lirbank · 09-18-2012, 05:09 AM

Hi Peter,

We're working on a product where one of our goals is to simplify templating and VM management and CentOS is among the pre-configured machine templates. It's quite a different approach to virtualization deployment and management (goal is to reduce complexity). We're still in an early phase feature-wise but it might be worth checking it out, see http://witsbits.com

Love to hear back from you if you find it interesting.

/Mikael

PeterSteele · 09-18-2012, 08:40 AM

Thanks for the post. Our use of virtualization is somewhat different than the norm and from what I gather from the info on your home page this isn't something that would suit our needs. That said, I'm going to check your site out more thoroughly to see if there is something there for us.

Peter

szboardstretcher · 09-18-2012, 09:40 AM

Quote:

Originally Posted by PeterSteele

I considered going with kickstart for the whole thing. My main issue is that I have to automatically create a lot of identical VMs and I want to do it as quickly as possible. I can do this with img files, since the majority of the time in bringing up a VM is the OS installation phase and with img files I can better control how long the OS install takes. I can blast the identical img file out to multiple partitions very quickly, and can have a new VM up and running in under two minutes.

In the end I guess we all have our requirements.

In the windows world you would use Windows Deployment Services and PXE boot. In the Linux world, you would set up a PXE Boot Server, along with Kickstart for a 100% unattended deployment -- aside from making sure that PXE is the first boot choice. There are plenty of guides out there to do this -- have fun.

PeterSteele · 09-18-2012, 09:50 AM

A PXE boot isn't an option for us. We want to give our customers a bootable CD-ROM or USB stick that provides a fully self-contained boot environment.

Lirbank · 09-18-2012, 11:37 AM

Quote:

Originally Posted by PeterSteele

a bootable CD-ROM or USB stick that provides a fully self-contained boot environment.

Peter, this sounds awfully similar to what Witsbits does out of the box. You can download the ISO and burn a USB sticks or CD-ROM, and then send it to your customers. You can then restrict your customers to only be able to access certain servers from the management console. Let me know what your requirements are and we might be able to work out a solution for it.

PeterSteele · 09-18-2012, 12:34 PM

Send me a personal email and we can talk.

Peter

Lirbank · 09-18-2012, 01:07 PM

Sure, please contact me on mikael@witsbits.com or http://witsbits.com/contact.php
/Mikael