Most well written makefiles have also a "uninstall" rule. Try this out in the source dir (same where you did "./configure ... etc"
If that does not work, I suppose you will have to check from the Makefile where the install-rule copies the files and remove them by hand.
You should definitely use your distribution's package manager, if possible.
RPM-based distros (not a specialist with these)