Go into the directory where you built the source origingally then do
If you deleted your originaly build directory just unpack the source and run configure (with the same prefix option if you specified any) then try running make uninstall (you may need to run make first, I'm not sure).