linux server time change in a sudden
 

Hi Guys,

I just find the time on our server could change in a sudden and restored.

this could happen hundreds of times during a day, and ruined our service. do you know why this is happening ?

Server info:

• ubuntu 14.04
• kernel: 4.9.0

we are using ntp to sync time:
ntpd 4.2.6p5

ntp config

I don't think the system time actually changed. The log file might have been parsed incorrectly, or there may have been a long delay in reporting. I'm not clear as to whether this "could" happen hundreds of times, or if it "did" happen hundreds of times.

Ntp will ignore servers that are off by over a certain amount. It would not suddenly change the time by 48 hours.

Hi, Thanks for the reply. But it did happen lots of times. Please reference to the below:
the log file is /var/log/syslog. And the /var/log/messages file on our centos 7 servers do not have this phenomenon.

If it's not a time change issue, what's the cause of this? how to prove it?

Thanks a lot.

a) where's this output coming from? dmesg?
b) i presume those are PIDs at the end; i'd start by investigating those.

Hi,

output is coming from /var/log/syslog
Those are CRON pids.

Anyone has any new ideas ? Need your help :(

Is there any remote logging to that server from elsewhere? Besides the date/time issue, I find the wide variation in PIDs very strange. What does your syslog.conf/rsyslog.conf look like?

I don't like this kind of answer, but at this moment I can only say: you need to wait. I found this also, and I will go into it, but you need to wait a few days, because at this moment I'm busy with other things/tasks.

i'd still start investigating those PIDs.
like:not saying that's the straight way to a solution, just the first thing i can think of to get more insight into the matter.
many more troubleshooting steps are possible, i suppose.
it is up to you to perform them.

@ondoho

I will try this and let you known. Thx.

I have figured this out. It has nothing to do with the server time. It's because of the chaotic log of crond.

I think it would be very helpful if you could please provide a more-detailed explanation of what you just said there. "Don't be cryptic!" :) You're writing to the next poor soul ... :banghead: ... who is trying to wrap his/her head around the very same problem, and they just stumbled-upon this thread. (Five years from now.) Write to them.

(Then, please mark the thread as [SOLVED].)

I'm assuming that you have a server that is snycronized to an external NTP timee server and is providing time to other machines on your LAN?; that would be the normal way to keep everybody on time.

NTP will not synchronize if the system time is more than 600 seconds off. It also will not slew time by hours or days. Once your local NTP is synchroized to an external time source it will slowly walk you system time into synchronization (over days, not instantly).

I notice that nobody else has asked so I will: what are those crontabs doing? Do you have statistics enabled and are you comparing logs against one another across servers? (Typically, you would keep peerstats and statistics on the time server on your LAN, not on servers that are synchronized to your local time seerver.)

first of all, sorry for my late reply.

take the log below for example:

Apr 29 08:30:01 localhost crontab[30444]:
Apr 29 08:30:01 localhost CRON[30206]:
Apr 27 19:48:01 localhost CRON[30929]:

The last line looks like the server time changed in a sudden, but actually it means on Apr 27 19:48:01, a cronjob ran with pid 30929, and for some reason the log printed to syslog after this line (Apr 29 08:30:01 localhost CRON[30206]:).

how to prove this:
Let's suppose the cronjob of the last line executed every minutes. Then you could only find one timestamp: Apr 27 19:48:01 in the whole log.

The point is: Time of server is fine. But the log to syslog is not fine. We expect syslog to log order by time, but it didn't. but I don't know what caused this.

Hope I have clarified this out. Enjoy !

I'm not really sure I understand that.