LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-28-2017, 11:42 PM   #1
hilou
Member
 
Registered: May 2013
Posts: 93

Rep: Reputation: Disabled
linux server time change in a sudden


Hi Guys,

I just find the time on our server could change in a sudden and restored.

Quote:
Apr 29 08:30:01 localhost crontab[30419]:
Apr 29 08:30:01 localhost crontab[30430]:
Apr 29 08:30:01 localhost crontab[30444]:
Apr 29 08:30:01 localhost CRON[30206]:
Apr 27 19:48:01 localhost CRON[30929]:
Apr 29 08:31:01 localhost cron[1469]:
Apr 29 08:31:01 localhost cron[1469]:
Apr 29 08:31:01 localhost CRON[30962]:
this could happen hundreds of times during a day, and ruined our service. do you know why this is happening ?

Server info:
  • ubuntu 14.04
  • kernel: 4.9.0

we are using ntp to sync time:
ntpd 4.2.6p5

Quote:
remote refid st t when poll reach delay offset jitter
==============================================================================
+61.216.153.105 118.163.81.63 3 u 902 1024 321 32.320 9.831 4.736
+46.31.185.18 77.40.226.114 2 u 1031 1024 367 311.879 -30.550 27.890
176.9.118.9 192.53.103.108 2 u 41h 1024 0 288.055 -17.100 0.000
*173.255.215.209 127.67.113.92 2 u 636 1024 377 189.785 11.539 4.749
123.59.63.20 .INIT. 16 u - 1024 0 0.000 0.000 0.000
ntp config
Quote:
driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
server 0.ubuntu.pool.ntp.org
server 1.ubuntu.pool.ntp.org
server 2.ubuntu.pool.ntp.org
server 3.ubuntu.pool.ntp.org
server ntp.ubuntu.com
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1

Last edited by hilou; 04-28-2017 at 11:58 PM. Reason: ubuntu,ntp,time
 
Old 04-29-2017, 02:34 AM   #2
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009Reputation: 1009
I don't think the system time actually changed. The log file might have been parsed incorrectly, or there may have been a long delay in reporting. I'm not clear as to whether this "could" happen hundreds of times, or if it "did" happen hundreds of times.

Ntp will ignore servers that are off by over a certain amount. It would not suddenly change the time by 48 hours.

Last edited by AwesomeMachine; 04-29-2017 at 02:36 AM.
 
Old 04-29-2017, 05:46 AM   #3
hilou
Member
 
Registered: May 2013
Posts: 93

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by AwesomeMachine View Post
I don't think the system time actually changed. The log file might have been parsed incorrectly, or there may have been a long delay in reporting. I'm not clear as to whether this "could" happen hundreds of times, or if it "did" happen hundreds of times.

Ntp will ignore servers that are off by over a certain amount. It would not suddenly change the time by 48 hours.
Hi, Thanks for the reply. But it did happen lots of times. Please reference to the below:
Quote:
Apr 29 08:30:01 localhost crontab[30444]:
Apr 29 08:30:01 localhost CRON[30206]:
Apr 27 19:48:01 localhost CRON[30929]:
--
Apr 29 08:38:01 localhost CRON[32724]:
Apr 29 08:38:01 localhost CRON[32725]:
Apr 28 09:24:01 localhost CRON[336]:
--
Apr 29 09:09:01 localhost CRON[9855]:
Apr 29 09:09:02 localhost CRON[9850]:
Apr 27 22:08:01 localhost CRON[10024]:
--
Apr 29 09:11:01 localhost cron[1469]:
Apr 29 09:11:01 localhost CRON[10813]:
Apr 28 20:41:01 localhost CRON[10812]:
--
Apr 29 09:13:01 localhost CRON[11339]:
Apr 29 09:13:01 localhost CRON[11341]:
Apr 27 15:14:01 localhost CRON[11469]: message repeated 2 times: [
Apr 25 00:27:01 localhost CRON[11498]:
--
Apr 29 09:41:01 localhost cron[1469]:
Apr 29 09:41:01 localhost cron[1469]:
Apr 24 12:27:01 localhost CRON[19829]:
--
Apr 29 09:44:01 localhost CRON[20455]:
Apr 29 09:44:01 localhost CRON[20453]:
Apr 26 23:30:01 localhost CRON[20742]:
--
Apr 29 09:56:01 localhost CRON[24113]:
Apr 29 09:57:01 localhost CRON[24404]:
Apr 28 21:27:01 localhost CRON[24403]:
--
Apr 29 09:58:01 localhost CRON[24590]:
Apr 29 09:58:01 localhost CRON[24591]:
Apr 28 00:17:01 localhost CRON[24838]: message repeated 3 times: [
--
Apr 29 10:06:01 localhost CRON[27196]:
Apr 29 10:06:01 localhost CRON[27198]:
Apr 28 19:48:01 localhost CRON[27193]:
--
Apr 29 10:10:01 localhost crontab[28368]:
Apr 29 10:10:01 localhost crontab[28369]:
Apr 27 21:25:01 localhost CRON[28881]:
--
Apr 29 10:11:01 localhost CRON[28904]:
Apr 29 10:11:01 localhost CRON[28906]:
Apr 25 10:35:01 localhost CRON[29085]:
--
Apr 29 10:14:01 localhost CRON[29557]:
Apr 29 10:15:01 localhost CRON[29852]:
Apr 27 23:09:01 localhost CRON[29854]:
--
Apr 29 10:22:01 localhost CRON[32023]:
Apr 29 10:22:01 localhost CRON[32026]:
Apr 28 18:17:01 localhost CRON[32307]:
--
Apr 29 10:31:01 localhost CRON[2367]:
Apr 29 10:31:01 localhost CRON[2368]:
Apr 26 20:49:01 localhost CRON[2523]:
--
Apr 29 10:51:01 localhost CRON[8245]:
Apr 29 10:51:01 localhost CRON[8240]:
Apr 27 04:14:01 localhost CRON[8404]:
--
Apr 29 10:57:01 localhost CRON[9809]:
Apr 29 10:57:01 localhost CRON[9803]:
Apr 28 02:48:01 localhost CRON[9965]:
--
Apr 29 11:02:01 localhost CRON[11364]:
Apr 29 11:03:01 localhost CRON[11657]:
Apr 25 02:11:01 localhost CRON[11658]:
--
Apr 29 11:12:01 localhost CRON[14285]:
Apr 29 11:12:02 localhost CRON[14277]:
Apr 28 19:06:01 localhost CRON[14583]:
--
Apr 29 11:28:01 localhost CRON[18841]:
Apr 29 11:28:01 localhost CRON[18842]:
Apr 28 12:15:01 localhost CRON[18972]:
--
Apr 29 11:30:01 localhost CRON[19345]:
Apr 29 11:30:01 localhost CRON[19347]:
Apr 27 03:02:01 localhost CRON[19341]:
Apr 28 14:05:01 localhost CRON[19348]: message repeated 2 times: [
--
Apr 29 11:38:01 localhost CRON[21833]:
Apr 29 11:38:01 localhost CRON[21834]:
Apr 27 21:02:01 localhost CRON[22123]:
--
Apr 29 12:30:01 localhost CRON[7818]:
Apr 29 12:30:01 localhost CRON[7825]:
Apr 27 23:35:01 localhost CRON[7821]:
--
Apr 29 12:35:01 localhost CRON[10097]:
Apr 29 12:35:01 localhost CRON[10098]:
Apr 28 20:40:01 localhost CRON[10099]:
--
Apr 29 12:50:01 localhost CRON[16600]:
Apr 29 12:50:01 localhost CRON[16599]:
Apr 27 04:41:01 localhost CRON[16601]:
--
Apr 29 12:51:01 localhost cron[1469]:
Apr 29 12:51:01 localhost cron[1469]:
Apr 28 06:46:01 localhost CRON[17462]:
--
Apr 29 12:51:01 localhost CRON[17463]:
Apr 29 12:52:01 localhost CRON[17758]:
Apr 28 03:14:01 localhost CRON[17756]:
--
Apr 29 13:01:01 localhost CRON[21769]:
Apr 29 13:01:01 localhost CRON[21770]:
Apr 28 17:51:01 localhost CRON[22042]:
--
Apr 29 13:10:02 localhost crontab[25491]:
Apr 29 13:10:02 localhost crontab[25507]:
Apr 28 10:50:01 localhost CRON[25764]:
--
Apr 29 13:26:01 localhost CRON[30447]:
Apr 29 13:26:01 localhost CRON[30448]:
Apr 25 09:28:01 localhost CRON[8873]:
--
Apr 29 13:29:01 localhost CRON[30932]:
Apr 29 13:29:01 localhost CRON[30933]:
Apr 27 04:17:01 localhost CRON[9393]:
--
Apr 29 13:30:01 localhost CRON[9395]:
Apr 29 13:30:01 localhost CRON[9400]:
Apr 23 04:18:01 localhost CRON[9399]:
--
Apr 29 13:42:01 localhost CRON[12064]:
Apr 29 13:42:01 localhost CRON[12065]:
Apr 25 18:33:01 localhost CRON[12062]:
--
Apr 29 13:51:01 localhost cron[1469]:
Apr 29 13:51:01 localhost cron[1469]:
Apr 28 10:09:01 localhost CRON[13709]:
--
Apr 29 13:56:02 localhost CRON[3748]:
Apr 29 13:56:02 localhost CRON[3750]:
Apr 28 10:11:01 localhost CRON[14593]:
--
Apr 29 13:58:01 localhost CRON[25494]:
Apr 29 13:58:01 localhost CRON[25493]:
Apr 28 12:37:01 localhost CRON[25580]:
--
Apr 29 14:05:01 localhost CRON[5058]:
Apr 29 14:05:01 localhost CRON[5059]:
Apr 26 10:55:01 localhost CRON[16104]:
--
Apr 29 14:11:01 localhost CRON[6424]:
Apr 29 14:11:01 localhost CRON[6425]:
Apr 28 18:58:01 localhost CRON[12059]:
--
Apr 29 14:12:01 localhost CRON[17391]:
Apr 29 14:12:01 localhost CRON[17385]:
Apr 26 05:52:01 localhost CRON[22938]:
Apr 28 19:43:01 localhost CRON[25882]:
--
Apr 29 14:13:01 localhost CRON[28800]:
Apr 29 14:14:01 localhost CRON[7329]:
Apr 27 22:00:01 localhost CRON[7330]:
Apr 29 14:14:01 localhost CRON[7330]:
Apr 29 14:14:01 localhost CRON[7331]:
Apr 28 22:30:01 localhost CRON[10551]:
Apr 28 17:14:01 localhost CRON[10699]:
Apr 28 12:01:01 localhost CRON[15075]:
--
Apr 29 14:15:01 localhost CRON[18641]:
Apr 29 14:15:01 localhost CRON[18633]:
Apr 27 17:25:01 localhost CRON[20161]:
Apr 27 10:35:01 localhost CRON[25170]:
--
Apr 29 14:17:01 localhost CRON[8777]:
Apr 29 07:29:01 localhost CRON[11674]:
Apr 28 22:40:01 localhost CRON[13524]:
Apr 28 15:37:01 localhost CRON[14337]:
Apr 28 21:06:01 localhost CRON[18084]:
Apr 26 00:15:01 localhost CRON[19406]:
--
Apr 29 14:18:01 localhost CRON[19898]:
Apr 29 14:18:01 localhost CRON[19890]:
Apr 25 08:23:01 localhost CRON[21822]:
Apr 29 14:19:01 localhost CRON[31258]:
Apr 29 14:19:01 localhost CRON[31259]:
Apr 26 19:00:01 localhost CRON[321]:
Apr 28 13:04:01 localhost CRON[1046]:
Apr 28 13:05:01 localhost CRON[1358]:
Apr 29 00:02:01 localhost CRON[6177]:
Apr 29 03:36:01 localhost CRON[6252]:
Apr 28 04:30:01 localhost CRON[8089]:
--
Apr 29 14:20:01 localhost crontab[10095]:
Apr 29 14:20:01 localhost crontab[10096]:
Apr 23 15:10:01 localhost CRON[10397]:
--
Apr 29 14:45:01 localhost CRON[2560]:
Apr 29 14:45:01 localhost CRON[2561]:
Apr 27 09:19:01 localhost CRON[2563]:
--
Apr 29 14:50:01 localhost CRON[3894]:
Apr 29 14:50:01 localhost CRON[3896]:
Apr 27 18:19:01 localhost CRON[3891]:
--
Apr 29 14:51:01 localhost cron[1469]:
Apr 29 14:51:01 localhost cron[1469]:
Apr 27 20:08:01 localhost CRON[4612]:
--
Apr 29 14:51:01 localhost CRON[4617]:
Apr 29 14:51:01 localhost CRON[4613]:
Apr 27 11:15:01 localhost CRON[4763]:
--
Apr 29 15:04:01 localhost CRON[8365]:
Apr 29 15:04:01 localhost CRON[8367]:
Apr 27 09:40:01 localhost CRON[8671]:
--
Apr 29 15:16:01 localhost CRON[12010]:
Apr 29 15:16:01 localhost CRON[12012]:
Apr 26 16:04:01 localhost CRON[12302]:
--
Apr 29 15:20:02 localhost CRON[12935]:
Apr 29 15:20:02 localhost CRON[12936]:
Apr 27 04:30:01 localhost CRON[12934]:
--
Apr 29 15:38:01 localhost CRON[18445]:
Apr 29 15:38:01 localhost CRON[18447]:
Apr 28 03:17:01 localhost CRON[18738]:
--
Apr 29 15:41:01 localhost cron[1469]:
Apr 29 15:41:01 localhost CRON[19608]:
Apr 27 12:05:01 localhost CRON[19757]:
Apr 27 12:05:01 localhost CRON[19758]:
--
Apr 29 15:59:01 localhost CRON[24670]:
Apr 29 16:00:01 localhost CRON[24831]:
Apr 28 19:40:01 localhost crontab[24820]:
--
Apr 29 16:01:01 localhost CRON[25621]:
Apr 29 16:01:01 localhost CRON[25622]:
Apr 28 00:20:01 localhost CRON[25804]:
--
Apr 29 16:54:01 localhost CRON[8880]:
Apr 29 16:54:01 localhost CRON[8873]:
Apr 28 20:36:01 localhost CRON[9177]:
--
Apr 29 17:30:01 localhost CRON[19402]:
Apr 29 17:30:01 localhost CRON[19403]:
Apr 27 17:22:01 localhost CRON[19404]:
--
Apr 29 17:41:01 localhost CRON[23108]:
Apr 29 17:41:01 localhost CRON[23109]:
Apr 28 08:52:01 localhost CRON[23294]:
--
Apr 29 18:25:01 localhost CRON[3687]:
Apr 29 18:25:01 localhost CRON[3688]:
Apr 27 14:48:01 localhost CRON[3690]:
--
Apr 29 18:44:01 localhost CRON[9327]:
Apr 29 18:44:01 localhost CRON[9326]:
Apr 21 18:40:01 localhost CRON[9404]:
the log file is /var/log/syslog. And the /var/log/messages file on our centos 7 servers do not have this phenomenon.

If it's not a time change issue, what's the cause of this? how to prove it?

Thanks a lot.

Last edited by hilou; 04-29-2017 at 05:52 AM.
 
Old 04-29-2017, 06:08 AM   #4
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,413
Blog Entries: 10

Rep: Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226
a) where's this output coming from? dmesg?
b) i presume those are PIDs at the end; i'd start by investigating those.
 
Old 04-29-2017, 10:29 PM   #5
hilou
Member
 
Registered: May 2013
Posts: 93

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by ondoho View Post
a) where's this output coming from? dmesg?
b) i presume those are PIDs at the end; i'd start by investigating those.
Hi,

output is coming from /var/log/syslog
Those are CRON pids.
 
Old 05-09-2017, 05:00 AM   #6
hilou
Member
 
Registered: May 2013
Posts: 93

Original Poster
Rep: Reputation: Disabled
Anyone has any new ideas ? Need your help
 
Old 05-09-2017, 01:43 PM   #7
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.8.2003
Posts: 5,435

Rep: Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066Reputation: 2066
Is there any remote logging to that server from elsewhere? Besides the date/time issue, I find the wide variation in PIDs very strange. What does your syslog.conf/rsyslog.conf look like?
 
Old 05-09-2017, 01:54 PM   #8
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,604

Rep: Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604
I don't like this kind of answer, but at this moment I can only say: you need to wait. I found this also, and I will go into it, but you need to wait a few days, because at this moment I'm busy with other things/tasks.
 
Old 05-10-2017, 01:39 AM   #9
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,413
Blog Entries: 10

Rep: Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226Reputation: 5226
i'd still start investigating those PIDs.
like:
Code:
ps aux | grep PID
not saying that's the straight way to a solution, just the first thing i can think of to get more insight into the matter.
many more troubleshooting steps are possible, i suppose.
it is up to you to perform them.

Last edited by ondoho; 05-10-2017 at 01:40 AM.
 
Old 05-10-2017, 03:40 AM   #10
hilou
Member
 
Registered: May 2013
Posts: 93

Original Poster
Rep: Reputation: Disabled
@ondoho

I will try this and let you known. Thx.
 
Old 05-12-2017, 06:04 AM   #11
hilou
Member
 
Registered: May 2013
Posts: 93

Original Poster
Rep: Reputation: Disabled
I have figured this out. It has nothing to do with the server time. It's because of the chaotic log of crond.
 
Old 05-12-2017, 10:44 AM   #12
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,233
Blog Entries: 4

Rep: Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260Reputation: 3260
Quote:
Originally Posted by hilou View Post
I have figured this out. It has nothing to do with the server time. It's because of the chaotic log of crond.
I think it would be very helpful if you could please provide a more-detailed explanation of what you just said there. "Don't be cryptic!" You're writing to the next poor soul ... ... who is trying to wrap his/her head around the very same problem, and they just stumbled-upon this thread. (Five years from now.) Write to them.

(Then, please mark the thread as [SOLVED].)

Last edited by sundialsvcs; 05-12-2017 at 10:46 AM.
 
Old 05-12-2017, 01:42 PM   #13
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,541

Rep: Reputation: 1062Reputation: 1062Reputation: 1062Reputation: 1062Reputation: 1062Reputation: 1062Reputation: 1062Reputation: 1062
I'm assuming that you have a server that is snycronized to an external NTP timee server and is providing time to other machines on your LAN?; that would be the normal way to keep everybody on time.

NTP will not synchronize if the system time is more than 600 seconds off. It also will not slew time by hours or days. Once your local NTP is synchroized to an external time source it will slowly walk you system time into synchronization (over days, not instantly).

I notice that nobody else has asked so I will: what are those crontabs doing? Do you have statistics enabled and are you comparing logs against one another across servers? (Typically, you would keep peerstats and statistics on the time server on your LAN, not on servers that are synchronized to your local time seerver.)
 
Old 05-22-2017, 10:55 AM   #14
hilou
Member
 
Registered: May 2013
Posts: 93

Original Poster
Rep: Reputation: Disabled
first of all, sorry for my late reply.

take the log below for example:

Apr 29 08:30:01 localhost crontab[30444]:
Apr 29 08:30:01 localhost CRON[30206]:
Apr 27 19:48:01 localhost CRON[30929]:

The last line looks like the server time changed in a sudden, but actually it means on Apr 27 19:48:01, a cronjob ran with pid 30929, and for some reason the log printed to syslog after this line (Apr 29 08:30:01 localhost CRON[30206].

how to prove this:
Let's suppose the cronjob of the last line executed every minutes. Then you could only find one timestamp: Apr 27 19:48:01 in the whole log.

The point is: Time of server is fine. But the log to syslog is not fine. We expect syslog to log order by time, but it didn't. but I don't know what caused this.

Hope I have clarified this out. Enjoy !

Last edited by hilou; 05-23-2017 at 06:10 AM.
 
Old 05-22-2017, 11:03 AM   #15
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,604

Rep: Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604Reputation: 5604
I'm not really sure I understand that.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Sudden change in the size of the fonts: Xfce 4.10 stf92 Linux - Desktop 15 03-24-2016 03:37 PM
How to change MariaDb time, synchronize it with the server time (CentOS7)? AdultFoundry Linux - Newbie 1 01-09-2016 03:44 PM
Can't boot GNOME 3 in Fedora 15 -- Sudden Change JWilliamCupp Fedora 1 08-31-2011 08:15 AM
Sudden change in volume control behavior ugge Linux - Software 8 01-16-2003 08:48 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration