I have somewhere between 8 and 12 servers, spread amongst different webhosts, all running CentOS 5.3

Everything on the servers has been compiled from source:
> EngineX, PHP, MySQL, Munin, etc

I compile from source because I usually run ./configure to modify the source before installing.

What would be the easiest way to keep the server software updated?

I am considering creating custom packages from a server which I would keep up to date manually. Then use Capistrano or Puppet to install those packages on all of my servers.
Can anyone vouch for this approach, or is there a more elegant solution?

You could setup a custom repository and just update your servers normally using it, but you'd likely need to start from the base source rpms and work from there to get it to work well.