Running on the CentOS 7.2 with the base kernel we are experiencing severe drift between the hwclock and software clock. NTP and Chrony are not running. I synchronized the software and hardware clock a little over 12 hours ago and now the systems are about and hour and 37 minutes apart:

[root@atd08 ~]# date
Thu Apr 20 07:30:58 CDT 2017
[root@atd08 ~]# hwclock
Thu 20 Apr 2017 06:13:54 AM CDT -1.053936 seconds

[root@atd09 ~]# date
Thu Apr 20 07:34:35 CDT 2017
[root@atd09 ~]# hwclock
Thu 20 Apr 2017 06:17:13 AM CDT -0.400729 seconds

[root@atd07 ~]# date
Thu Apr 20 07:36:40 CDT 2017
[root@atd07 ~]# hwclock
Thu 20 Apr 2017 06:19:25 AM CDT -0.808477 seconds

[root@atd11 ~]# date
Thu Apr 20 07:36:50 CDT 2017
[root@atd11 ~]# hwclock
Thu 20 Apr 2017 06:20:20 AM CDT -0.928686 seconds

What would cause a drift of this nature?

Assuming this is a physical and not a virtual machine. The two clocks are totally independent of each other. The hardware clock which sole purpose is to set the system clock at boot time is an IC chip and as configured is only updated when you shutdown the computer or manually set. It can drift due to voltage and temperature.

The system clock i.e. what you see with the output of the date command is a software counter based upon a timer interrupt. The timer interrupt used may not be exact so it will either be slow or fast. There is a drift factor that can be adjusted which is stored in the /etc/adjtime file.

The hardware clock can be referenced to local or UTC time and the output always displays its actual time i.e. does not convert to local time despite the fact that it outputs a timezone. linux defaults to UTC as the hardware clock reference which is also stored in the /etc/adjtime file.

So this is more or less normal which is why most run ntp or connect to a stable time source.

Most frequently, one machine on the network will very-regularly sync its clock using ntpd to some public atomic-time source, and then every other computer will sync to it. Absent this, the system clock can – as you saw – drift badly.

We have to run PTP for our time protocol. In most cases it runs great but at one site I am getting the following error. And from what I have read this usually means something else is affecting the system clock and I thought this extreme drift was an indication that something was affecting the clock. I just can't tell what program or driver.

systemctl status ptp4l phc2sys
● ptp4l.service - Precision Time Protocol (PTP) service
Loaded: loaded (/usr/lib/systemd/system/ptp4l.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-04-20 18:40:26 CDT; 16min ago
Process: 16134 ExecStartPre=/usr/local/sbin/hwstamp_ctl -i ens2 -r 1 -t 1 (code=exited, status=0/SUCCESS)
Main PID: 16139 (ptp4l)
CGroup: /system.slice/ptp4l.service
└─16139 /usr/local/sbin/ptp4l -f /etc/ptp4l.conf -i ens2.172 -p /dev/ptp2

Apr 20 18:56:39 atd07 ptp4l[16139]: [19955.797] clockcheck: clock jumped forward or running faster than expected!
Apr 20 18:56:39 atd07 ptp4l[16139]: [19955.798] master offset 273815247557 s0 freq +100000000 path delay -121130418
Apr 20 18:56:41 atd07 ptp4l[16139]: [19957.019] clockcheck: clock jumped forward or running faster than expected!
Apr 20 18:56:41 atd07 ptp4l[16139]: [19957.021] master offset 274162356305 s0 freq +100000000 path delay -121130418
Apr 20 18:56:42 atd07 ptp4l[16139]: [19958.242] clockcheck: clock jumped forward or running faster than expected!
Apr 20 18:56:42 atd07 ptp4l[16139]: [19958.243] master offset 274506842666 s0 freq +100000000 path delay -118544105
Apr 20 18:56:43 atd07 ptp4l[16139]: [19959.464] clockcheck: clock jumped forward or running faster than expected!
Apr 20 18:56:43 atd07 ptp4l[16139]: [19959.465] master offset 274853933589 s0 freq +100000000 path delay -118544105
Apr 20 18:56:44 atd07 ptp4l[16139]: [19960.686] clockcheck: clock jumped forward or running faster than expected!
Apr 20 18:56:44 atd07 ptp4l[16139]: [19960.687] master offset 275201024661 s0 freq +100000000 path delay -118544105

Not all that familiar with PTP but s0 means the clock is unlocked. Has this one computer ever been able to sync correctly? Is the hardware the same on all computers?

I'm only guessing but are you using the correct phc device i.e. /dev/ptp2?