Here is something that might work.
Make sure the ntp daemon is not running.
/etc/init.d/ntpd stop ( or what ever you use on your distro )
check that your system is reasonably close by manually setting the date using the date command. That way there won't be a big difference between timeserver and your machine.
Try this configuration /etc/ntp.conf file:
## Default rules for all connections
restrict default nomodify notrap noquery
## Allow full access to the local host
## Your Client subnet
restrict <my internal IP network> mask <Internal Mask to Match> nomodify notrap
## Your time servers
Now that you have the above configuration and ntpd is not running lets run the following command.
ntpdate will not run if ntpd is running.
Now start ntpd
/etc/init.d/ntpd start ( or what ever you use on your distro )
Let me know if that worked?