Every package built with a correctly written SlackBuild script and installed with installpkg will be listed in /var/log/packages and therefore taken into account in the Slackware package management system. For those, you do not need to keep the .tgz packages. If you, however, build/install something via simple (not recommended):
Code:
./configure
make
make install
or executing a binary package (eg. nvidia binary driver, VirtualBox, etc), you should keep the installation directory or executable binary file to make it easier for you to uninstall it in the future. They are not included in the slackware package management system.