I personally use the Grsecurity patch. I think that building the patched kernel is the easy part. Slackware has great documentation discussing how to build a kernel.
http://docs.slackware.com/slackbook:linux_kernel
The difficult part, which isn't skill related, is being detailed enough to configure PaX permissions and RBAC roles. On a laptop or desktop it is time consuming. On a server running httpd, bind, etc, it takes less effort. This is due to the fact that the server doesn't need to use many different software processes. The Grsecurity documentation is very easy to follow so it's mostly just time consuming.
Check it out and see for yourself.
I did come across a github project that aids in the process of building the kernel and such. I found some discrepancies in the kernel configuration because I wanted different kernel features.
This project was helpful to me because it documents part of the process for Slackware.
https://github.com/pyllyukko/grsec_confnbuild
Anyway, I hope someone else will find this info useful.