How did you install it? With 'make install'? If so, you can hope there is a "make uninstall" that the programmer added to the Makefile. If not, you can go through the Makefile and find the location for "all" and go to those locations and manually remove all the files that were copied to those locations.
If it made a single directory tree, and nothing else is inside that tree, then yes, go ahead and remove it and it's subdirectories it created.
For things like this in the future, look into getting something called checkinstall from
www.freshmeat.net
Cool