depends on the program. Sometimes it comes whit a make uninstall, but sometimes not.
I used to have this problem all the time, until I figured out how to make my own packages.
Now I always make a package from the source code (dunno how to do that on debian, I use slackware).
That way i can always see what software is installed and can easily remove them.
But that of course doesn't fix the problem you have now...