The goal with hwclock is to adjust the hardware clock either at boot time or regularly by taking the system clock as a reference since its drift is usually in the order of 1 µs from what is said in the man page of hwclock.
What I don't understand is why is it written that the user should regularly adjust the hardware clock for hwclock to calculate its drift?
When you have two clocks, one of which being used as a reference, a periodic checking should be enough to measure the drift of the inaccurate clock. And when the drift amount is memorised along with the time at which the whole system is shut down, this should be enough to adjust the inaccurate clock to the right time at boot time.

UNIX/Linux is more concerned with "system time" than "hardware clock time". While running the OS does not periodically adjust itself to hwclock unless you've told it to do so. It simply runs and tries to keep rack of clock ticks. Network Time Protocol is used to try to keep running system time in sync with reliable time sources external to your OS and hardware.

However, when you shutdown the system ONLY the hardware clock is running. On boot up the hardware clock is read to set the initial system time (otherwise it would default back to January 1, 1970). Therefore it becomes important that on shutdown you set the hardware clock to have the current system time (which is presumably accurate before the shutdown) so that system time is as close to accurate on reboot. If you then have ntp setup your system time will deal with any drift the hardware clock had during the downtime.

Since only the hardware clock is running during downtime and there is no way for the system to know how long it was down without relying on the hardware clock's time at boot there is now way to keep track of how much time it may have actually drifted while down. You seem to be assuming that hardware clock drift will always be the same but that isn't necessarily the case. It might be affected by a weak battery or strong magnetic fields from adjacent servers or even by the dreaded cosmic ray that vendors love to blame for many things.

If your system has ntp, then the adjtime file is useless unless the network is down for long time and you have to use the computer anyway during that time.

I agree the hardware clock might drift due to some external influence, but the hwclock man page clearly shows that the linux clock system wants to tackle with the "systematic drift" of the clock.