deleting the source/build folder is standard practice. the binaries which are copied or installed should be self-sufficent.
Also, I (and most people I know) usually just build things in their home directory, make install as root, and then delete the build folder. This makes it difficult to uninstall; but you can either
1) keep the tarball, unpack, configure, and "make uninstall" (if you use the same installation folder both time, it should work)
2) use a package management system like GNU stow.
or, like me, never bother uninstalling anything