I create a /usr/local/src/.tarballs directory. It keeps the tarballs there, and removes the directory from sight when doing listings. Similarly, patches either go in the .tarball directory or a .patches directory; whichever suits your method of organization.
I "cd" to /usr local/src and issue command like: tar xvjf .tarballs/tarball_name.tar.bz2
I build within the directory itself unless specified to do otherwise in the docs.
I store a file with the commands used to compile and install the source. This is usually just a list of the patch commands used and the options to configure; make and make install rarely change.
I run two installs. The first is a compile and install with a non-privileged user. and to a non-standard location. For instance, installing to this user's home directory (something like /home/testinst/local_install). Then I run a find command on the directory to get a list of all the files created/copied, and then make uninstall or just simply delete the whole local_install subtree. The second install is the "real" install, with the location being set to /usr/local or wherever.
After installing, I remove the source tree and save the two files I created: the one with commands to patch/configure the source and the one listing the files installed in some other location (again, wherever your organizational preferences tell you). This way, I can always re-extract the source and get the source tree back to exactly the state it was when I compiled and installed without the need for keeping the source tree around forever. It also allows for manual uninstall if the developers left out that target in the makefile.
Pitfalls of this method: If the install script is hard-wired to copy something to a system location (such as /usr/lib), then its possible the file listing isn't complete. That is, if the make install on the /home/testint/local_install fails because it doesn't have root privileges to make the copy, then commands following that copy may not get executed. Similarly, if make install modified any configuration files, this method will not show it. There might be more, but I haven't noticed them yet.