Yes, yes, I think so, and no.
The Add/Remove Software tool is a Gnome app that manages yum and apt repositories. So it and yum will mirror each other.
As for those that you install yourself, I am 90% sure RPMs you install yourself don't show up in the tool or yum, as I can't find them in there. So, yes, you do have to keep track of them.
Also, if you uninstall a package that had dependencies specifically for it, you have to track those other packages down and uninstall them manually.
Hope I helped.