If you still have the source tree and haven't changed anything like executing make clean, you might get lucky and there will be an uninstall target. You would then just execute make uninstall. Otherwise, rm -f individually on every file that was installed.
That's the advantage of package managers, they keep track of every file and directory installed by a package so they can easily be removed. Different ones do it different ways, but they "all" do it.