If you compiled or otherwise added kernel modules after upgrading the kernel, the package manager will leave those changes in place even after it deletes everything in the original kernel package. For example, the nvidia driver, vmware driver, etc, may have been added after the fact.
du -hs /lib/modules/*
will show the size of each directory, and you will find that most of them are small. You can safely delete the old unused directories, if you are careful not to delete the current ones.