NTP Synchronization Failed

antriksh · 03-18-2015, 05:51 AM

I have two clients : exa02db01 & exa02db02

While querying for NTP server from exa02db01, we see that it is syncing to LOCAL time on the node. However, on exa02db02 it points to the NTP server on the network.

Code:

[root@exa02db01 ~]# ntpq -pn
    remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*127.127.1.0     .LOCL.          10 l    7   64  377    0.000    0.000   0.001
192.168.1.82    .INIT.          16 u    -   16    0    0.000    0.000   0.000
-------------------------
Node-2:
---------------------
[root@exa02db02 ~]# ntpq -pn
    remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
127.127.1.0     .LOCL.          10 l   42   64  377    0.000    0.000   0.001
*192.168.1.82    66.228.59.187    3 u    6   16  377    3.145    5.747   3.683

Both the clients have exact same configuration but why its is failing on db01.

[root@exa02db01 ~]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@exa02db01 ~]# ntpdate -b -v 192.168.1.82
17 Mar 10:12:01 ntpdate[84316]: ntpdate 4.2.2p1@1.1570-o Mon Dec 9 14:55:56 UTC 2013 (1)
17 Mar 10:12:05 ntpdate[84316]: no server suitable for synchronization found
[root@exa02db01 ~]# service ntpd start
ntpd: Synchronizing with time server: .....................[FAILED]
Starting ntpd: [ OK ]

wpeckham · 03-18-2015, 06:07 AM

Try one: check your networking. Can both servers ping the network ntp server? If not, you need to fix that first.

Try two: once you are sure networking is fine, does ntp server repond equally to both? (Test using ntpdate with ntpd locally stopped.) I can set an ntpd server to respond to ONLY one subnet, and if a client from a different subnet polls the connection is quietly dropped. It is worth checking to make sure this is not part of the problem.

NOTE: nethoer of these tests indicate a solution, they only tell you if the fault is in certain areas. Once you know where to look, it should be easier to find and fix the cause.

antriksh · 03-18-2015, 06:17 AM

Both the servers can ping the NTP server. More details below:

192.168.1.82 is the NTP Server IP

Quote:

[root@exa02db01 ~]# ntpdate -u -v 192.168.1.82
18 Mar 05:35:20 ntpdate[50298]: ntpdate 4.2.2p1@1.1570-o Mon Dec 9 14:55:56 UTC 2013 (1)
18 Mar 05:35:23 ntpdate[50298]: step time server 192.168.1.82 offset 2.573212 sec

No firewall is running on both the servers.

Quote:

[root@exa02db01 ~]# ntpdate -q -d -d -d -d 192.168.1.82
17 Mar 12:27:45 ntpdate[90680]: ntpdate 4.2.2p1@1.1570-o Mon Dec 9 14:55:56 UTC 2013 (1)
Looking for host 192.168.1.82 and service ntp
host found : rscvm-it03.com
transmit(192.168.1.82)
transmit to 192.168.1.82
receive(192.168.1.82)
offset: 1.925157, delay 0.00394
transmit(192.168.1.82)
transmit to 192.168.1.82
receive(192.168.1.82)
transmit(192.168.1.82)
transmit to 192.168.1.82
receive(192.168.1.82)
offset: 1.925262, delay 0.00307
transmit(192.168.1.82)
transmit to 192.168.1.82
receive(192.168.1.82)
offset: 1.925247, delay 0.00308
transmit(192.168.1.82)
server 192.168.1.82, port 123
stratum 3, precision -17, leap 00, trust 000
refid [192.168.1.82], delay 0.02869, dispersion 8.00002
transmitted 4, in filter 4
reference time: d8b2d269.50017a60 Tue, Mar 17 2015 12:18:49.312
originate timestamp: d8b2d483.42a4fc59 Tue, Mar 17 2015 12:27:47.260
transmit timestamp: d8b2d481.555f7835 Tue, Mar 17 2015 12:27:45.333
filter delay: 0.02956 0.00000 0.02869 0.02870
0.00000 0.00000 0.00000 0.00000
filter offset: 1.925157 0.000000 1.925262 1.925247
0.000000 0.000000 0.000000 0.000000
delay 0.02869, dispersion 8.00002
offset 1.925262

17 Mar 12:27:45 ntpdate[90680]: step time server 192.168.1.82 offset 1.925262 sec

pwalden · 03-18-2015, 11:46 AM

I ran into a situation wherein the HW clock on an instance was bad. NTP at boot sets the HW clock, but then relies on the HW clock to track to real time with only periodic (every few hours) syncs by ntp.

If the HW clock is too slow or fast and slews by many seconds over those periodic timeframes, the ntp daemon just gives up. You should see a ntp "giving up" entry in the message log.

In my case, the HW clock was a a few minutes fast per day.

antriksh · 03-20-2015, 03:43 AM

Can someone tell me why i am getting below error though NTP server is configured in ntp.conf:

Quote:

[root@exa02db01 ~]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@exa02db01 ~]# ntpdate -b -v 192.168.1.82
17 Mar 10:12:01 ntpdate[84316]: ntpdate 4.2.2p1@1.1570-o Mon Dec 9 14:55:56 UTC 2013 (1)
17 Mar 10:12:05 ntpdate[84316]: no server suitable for synchronization found
[root@exa02db01 ~]# service ntpd start
ntpd: Synchronizing with time server: .....................[FAILED]
Starting ntpd: [ OK ]

ntp.conf file:

Quote:

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default ignore
restrict -6 default ignore

# restrict default kod nomodify notrap nopeer noquery ignore
# restrict -6 default kod nomodify notrap nopeer noquery ignore

# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).

server 127.127.1.0 # local clock iburst burst minpoll 4 maxpoll 4
fudge 127.127.1.0 stratum 10

# Drift file. Put this in a directory which the daemon can write to.
# No symbolic links allowed, either, since the daemon updates the file
# by creating a temporary in the same directory and then rename()'ing
# it to the file.
driftfile /var/lib/ntp/drift

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8
#### BEGIN Generated by Exadata. DO NOT MODIFY ####
# 12650539
restrict default mask 0.0.0.0 noquery nomodify notrap
restrict 192.168.1.82 mask 255.255.255.255 nomodify notrap noquery
server 192.168.1.82 prefer iburst burst minpoll 4 maxpoll 4
#### END Generated by Exadata ####

wpeckham · 03-20-2015, 05:53 AM

Perhaps I am full of it here, but it APPEARS you have restricted acccess to the NTPD service to loopback and the local server IP. In other words, the NTPD daemon on 82 can only talk to itself and is not being allowed to server time to any client. Open up that restrict to the local subnet and see if it all suddenly works.

antriksh · 03-20-2015, 07:21 AM

I am not sure if i understand it correctly... But there is no firewall running on the node. Is there any other way to check if there is any restriction?

Output of netstat -tulnp | grep ntp:

Quote:

udp 0 0 169.254.253.67:123 0.0.0.0:* 86542/ntpd
udp 0 0 192.168.12.1:123 0.0.0.0:* 86542/ntpd
udp 0 0 10.128.197.33:123 0.0.0.0:* 86542/ntpd
udp 0 0 10.128.197.32:123 0.0.0.0:* 86542/ntpd
udp 0 0 10.128.197.29:123 0.0.0.0:* 86542/ntpd
udp 0 0 10.128.197.27:123 0.0.0.0:* 86542/ntpd
udp 0 0 10.128.196.10:123 0.0.0.0:* 86542/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 86542/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 86542/ntpd
udp 0 0 ::1:123 :::* 86542/ntpd
udp 0 0 fe80::221:28ff:fee8:104e:123 :::* 86542/ntpd
udp 0 0 fe80::221:2800:1cf:437f:123 :::* 86542/ntpd
udp 0 0 fe80::221:28ff:fee8:104f:123 :::* 86542/ntpd
udp 0 0 :::123 :::* 86542/ntpd

wpeckham · 03-21-2015, 07:15 AM

I am not talking about firewall or network settings, look at the 'restrict' lines in your ntpd settings on your ntopd server and see what networks they allow to be clients. If those lines do not specifically allow a client IP or subnet, the server will not answer BY DESIGN! Since we have checked nearly everything else without finding an issue, this is the next logical place to check.

antriksh · 03-31-2015, 03:38 AM

ntp server conf:

Quote:

driftfile /var/lib/ntp/ntp.drift
logile /var/log/ntp.log

statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
filegen protostats file protostats type day enable

server 0.north-america.pool.ntp.org burst iburst
server 1.north-america.pool.ntp.org burst iburst
server 2.north-america.pool.ntp.org burst iburst
server 3.north-america.pool.ntp.org brust iburst

restrict -4 default kod notrap nomodify nopeer noquery

restrict 127.0.0.1
restrict ::1

I don't see any issue from NTP server side. After restarting ntp service on the server still getting below error:

Quote:

[root@exa02db01 ~]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@exa02db01 ~]# ntpdate -b -v 192.168.1.82
30 Mar 23:54:02 ntpdate[2923]: ntpdate 4.2.2p1@1.1570-o Mon Dec 9 14:55:56 UTC 2013 (1)
30 Mar 23:54:06 ntpdate[2923]: no server suitable for synchronization found
[root@exa02db01 ~]#

antriksh · 04-06-2015, 05:06 AM

Can anyone help me on this? I am clueless what to check now.

zafar_dandoti · 04-07-2015, 11:12 PM

As said by wpeckham ur ntp server is restricted to localhost
U need to add another restriction for ur subnet
Ex: restrict 192.168.1.0/24 nomodify nopeer notrap