I had always thought that there were no issues with the kernel.org kernels until very recently. It's not that they're unstable. In fact, they are probably better kernels than what your distro has; at the very least, they have updates for security and drivers. The issue is one of compatibility. For instance, if you switch to a kernel somewhere past 2.6.28 (don't know exactly which one) you will need to modify /etc/init.d/nfs-kernel-server, as the check for nfsd has changed. I was also getting a random screen freeze up to about 184.108.40.206. Earlier kernel issues included such things as a change to the config strings for SATA drives.
So, if you have a sufficient level of competence (and time), then go for the kernel.org kernels and have the latest stuff. OTOH, if you are unable or don't have the time to track down things like the nfs-kernel-server script problem, then it's better to stay with what's provided by your distro. Most users will probably want to stay with the distro kernel.