In general: no good way. That is why we have package management software (deb, rpm) in the first place.
Occassionally there might be an uninstall target in makefile, that is, running command 'make uninstall' in the source directory might do the job.
Also using a tool called checkinstall can do quite a good job trackking the changes made to the filesystem in the installation phase. Thus this have to be done before installation.
Hopefully the program is installed in /usr/local, that contains only programs installed by the system administrator from the source, so it should be easy to locate the files installed by the software, so removing them manually shouldn't be impossible, either.