If you are on Slackware, look into an application called checkinstall at
www.freshmeat.net
However, for your question:
If you want to uninstall something, and a "make uninstall" is included in the Makefile, then yes, you should at least keep the original tarball to untar and uninstall. If uninstall isn't included in the Makefile, then you will need to manually remove each file that it has "installed" from make install.
HTH
Cool