UNIX/Linux is more concerned with "system time" than "hardware clock time". While running the OS does not periodically adjust itself to hwclock unless you've told it to do so. It simply runs and tries to keep rack of clock ticks. Network Time Protocol is used to try to keep running system time in sync with reliable time sources external to your OS and hardware.
However, when you shutdown the system ONLY the hardware clock is running. On boot up the hardware clock is read to set the initial system time (otherwise it would default back to January 1, 1970). Therefore it becomes important that on shutdown you set the hardware clock to have the current system time (which is presumably accurate before the shutdown) so that system time is as close to accurate on reboot. If you then have ntp setup your system time will deal with any drift the hardware clock had during the downtime.
Since only the hardware clock is running during downtime and there is no way for the system to know how long it was down without relying on the hardware clock's time at boot there is now way to keep track of how much time it may have actually drifted while down. You seem to be assuming that hardware clock drift will always be the same but that isn't necessarily the case. It might be affected by a weak battery or strong magnetic fields from adjacent servers or even by the dreaded cosmic ray that vendors love to blame for many things.