Generally speaking, yes, convention suggests you keep any source trees inside an src directory (/usr/src, /usr/local/src, etc.) In fact, the convention goes so far as to suggest placing source trees for basic system programs in /usr/src (the kernel, core utility packages, and anything else needed to boot the system into a rescue mode), and that everything else should be installed in /usr/local/src. Part of that reasoning is, distributions are not supposed to touch anything in the /usr/local tree. So you could (in theory) reinstall a distro or install an entirely new one and everything in /usr/local would remain intact. It's not quite that simple, but that's the idea.
Also, you don't have to keep source tree around if you don't want to. You can re-tar them, and stick them somewhere (like /usr/local/src/tarballs) or you could just delete them entriely. I only keep the kernel source and anything I've hacked on. I re-tar everything else.