Yeah, what titopoquito said
You need to realize that not every makefile will include an "uninstall" target. In fact, they don't even have to include an "install" target either.
It's only a convention that source tarballs will include a configure script and a "make install" target, but there's nothing stopping developers from ignoring these conventions.
Thus every time you use "make install" you run the risk of not being able to remove the app cleanly.
Unless of course, you use checkinstall