Yeah, it makes sense to keep at least the last stable kernel you built around, or just the stock huge Slackware kernel. The idea is that if you have a problem in the new kernel, you can fall back on a known-good kernel. Otherwise you would have to break out the install disc and boot up your system that way, which is sort of a hassle (especially if you don't have an optical drive installed normally...).
Of course this mainly applies during the time period right after a kernel upgrade. If you have been using your newly compiled kernel for awhile and there are no problems then you probably wouldn't need to drop back down to the stock one. Of course, for the few MBs it takes up, it is nice to have a backup.
But you don't say if the source package is the same as the kernel you have built, or if you downloaded the vanilla sources for a newer kernel release and built that. You should always keep the source tree you used to build your current kernel, as that can be required for building out of kernel modules (like binary video drivers, WiFi drivers not yet included in the kernel, etc).
Also keep in mind that you need to have the exact same source tree as the one you built the kernel for, not just the same release. In other words, if you built a custom kernel from the 184.108.40.206 source package, just reinstalling that source package in the future won't let you build kernel modules for your kernel.