heh, it's not easy to search for this in google, so many random man pages show up.
I did find a page describing how to view coloured man pages in vim:
Tip #167: Using vim as a man-page viewer under Unix
This page about color in xterm for BSD (you'll have to scroll down a bit), no reason why it shouldn't work in Linux, but maybe I have to restart X before it will read .Xdefaults
Edit: This thing doesn't seem to be working for me, maybe you have to actually all the colours they say are in the rest of the config file that comes with BSD...
I also found a manpage/info page viewer called
pinfo. If you alias man to pinfo -m then you'd get man pages in colour. Might be a bit overkill though...