I'm running ntpd on my servers to keep their times synced with my 2 ntp servers (both windows 2003 server domain controllers).

The problem is that the ntpd is not keeping the time accurate and the clocks are drifting. I'm sure that when ntp is working properly the time is supposed to be second accurate, but it's not happening here.

The clocks are drifting 15-180 seconds away from the time of the master ntp servers within a few days. I then have to ntpdate to adjust the clocks back to the right time manually which is silly (in fact I have to stop ntpd, then ntpdate, then start ntpd as they use the same socket)

Below is my ntpd.conf without the comments:
Ntpd is running as root so there is no lack of permissions.

Both ntp servers are second accurate and their time is identical down to the split second. So why aren't the unix boxes keeping time using ntpd?

You are restricting it to 127.0.0.1 (loopback) so you're telling it to only to allow it access to itself. You should have restrict entries for each of your time servers:

e.g.
restrict xx.xx.xx.xx mask 255.255.255.255 nomodify notrap noquery
restrict xx.xx.xx.xx mask 255.255.255.255 nomodify notrap noquery
restrict 192.5.41.40 mask 255.255.255.255 nomodify notrap noquery
restrict 192.5.41.41 mask 255.255.255.255 nomodify notrap noquery

(The latter two entries are tick and tock at the US Naval Observatory - nice fall backs to have - you'd have to add server entries for those as well).

Also once you've set your system time I'd suggest running:
hwclock --systohc

That will insure your hardware clock is in sync and doesn't cause issues - I've seen other users here have issues that seemed to have been because the hardware clock was overriding things.

I suspected this might be the case, ok I'll try that. Thanks.