Patch that kernel:
http://grsecurity.net
PaX combined with grsecurity acts as an incredible safety net. Security is about layers. Don't let intruders get through them all.
Also check out libsafe:
http://freshmeat.net/projects/libsafe/
Libsafe is pretty much the only missing piece in grsecurity. This will assist in protection from return-to-libc attacks.
Those are some basics from the tech side of things.
As for certifications, the CISSP is most respected by managers. However I know CISSPs that are not that bright, and I definitely wouldn't trust them to secure any of my machines or networks (like managers!). The GSEC certification by SANS is a certification that is respected amongst your peers. It doesn't necessarily prove your skills/credibility, but it is helpful to have.