Every package built with a correctly written SlackBuild script and installed with installpkg will be listed in /var/log/packages and therefore taken into account in the Slackware package management system. For those, you do not need to keep the .tgz packages. If you, however, build/install something via simple (not recommended):
or executing a binary package (eg. nvidia binary driver, VirtualBox, etc), you should keep the installation directory or executable binary file to make it easier for you to uninstall it in the future. They are not included in the slackware package management system.