When compiling from source many packages will have a
target, which will cleanly remove the installed software.
There is an even better approach available on many distros: Rather tan directly installing with "make install" you can build a package and install it through the package manager.
The advantage of that approach is that the package manager "knows" about the package, so that it can
a) include it into dependency calculations
b) remove it cleanly
... and probably more.
The exact procedure depends on the distro / package manager you are using.
Which distro are you on?