what we are seeing in the logs is the entries started around 1:28 AM CET and ended around 12:24 PM CET. When I add ~5 Hrs 30 Mins it becomes around 7 AM CET to 6 PM CET. This 7 AM – 6 PM is the actual time when the users did the activities. On this server we are seeing that the logs are written with ~5 Hrs 30 mins early then the activity performed time for the entire day. Can some help me analysis what could have cause the above issue, i did verify that no user log into the server on the day this happened and there was no changed to server time.

Which server/logs?
It is possible for the time being used by a daemon to be other than, and therefore different from, system time.

You've given us no real information. What kind of server is this (real or virtual)? What kind of hardware, running what services? How long has this server been in operation, and where are you syncing time to now (internal NTP server, or external?)

Thanks Sean for your response, logs referenced in above question was a java application installed on the server which is recording the audit trail of user transaction. We have observed server time didn't changed from CET since the provisioned of server.

But as mentioned above audit trail data captured by java application was 5+ hours early than user transaction time.

Raj.

Great...and you *STILL* don't provide any details that would let anyone even GUESS as to the cause of the problem.

AGAIN, you were asked:

• What kind of server is this (real or virtual)?
• What kind of hardware?
• Version/distro of Linux?
• Running what services? (saying "a java application" is meaningless)
• How long has this server been in operation (is this a new server, or one that's been around a while?)
• Where are you syncing time to now (internal NTP server, or external?)

Just saying you have a java application writing to a log is meaningless.

What kind of server is this (real or virtual)?
Virtual Server

What kind of hardware?
HOST: IBM System X , ESXi 6.5 U2 is installed
Virtual Server:VMWARE OS: Centos 7.4 , vCPU: 4, RAM: 32 GB , HDD1 : 60 GB and HDD2: 450 GB, Compatibility: VM Version 8

Version/distro of Linux?
Linux: CentOS 7.4 3.10.0-693.el7.x86_64

Running what services? (saying "a java application" is meaningless)
JDK 1.6.0_33
Jboss 4.0.5 GA
OpenOffice 3.3
SWFtools 0.9.1
activemq 5.3.2

How long has this server been in operation (is this a new server, or one that's been around a while?)
7 Months

Where are you syncing time to now (internal NTP server, or external?)
External

Thanks,
Raj

Ok...possible cause there.
Still haven't told us what application is creating these logs. And a further question; does the SYSTEM time drift, or the time in the logs that are generated by this mystery application??
And has it been doing this the whole time, or is this a new problem?
What is syncing?? The guest OS, or the VMWare host?

And have you contacted VMWare for assistance, since you're paying for support? Have you installed/configured the VMWare tools for time synchronization, and followed their guides on best practices for time sync? What service are you using on the CentOS system for time, and do you have the patches applied and the VMWare utilities loaded?

Quote:
Still haven't told us what application is creating these logs. And a further question; does the SYSTEM time drift, or the time in the logs that are generated by this mystery application??
- Inhouse developed web based java application, we have reviewed the application logs, application code and application audit trail.

Quote:
And has it been doing this the whole time, or is this a new problem?
- this happened 16 times from 7 Dec to 24 Jan, 2019, after 24 Jan we didn't see this happening again.

Quote:
What is syncing?? The guest OS, or the VMWare host?
- Guest OS

And have you contacted VMWare for assistance, since you're paying for support? Have you installed/configured the VMWare tools for time synchronization, and followed their guides on best practices for time sync? What service are you using on the CentOS system for time, and do you have the patches applied and the VMWare utilities loaded?
- Reached out to vmware but couldn't find the root cause, VMware Tools are installed on server with version: 10277(10.1.5), default standard services were enabled on OS while provisioning the server, vmware patches will be applied to host and vmtools on guest vm during maintenance activity last one was performed on 23 Mar 2019.

...and yet this is the application that 'drifts'. Again, does the system time drift, or just the application logs?
So its not happening any longer...so what is the problem with your system, then??
Ok...and is the guest OS drifting?? The VM Host time correct? Do you have the guest set to get time from host?
Ok, but if you're not having the issue any longer, it would seem there's not much point to your thread.

Quote:
...and yet this is the application that 'drifts'. Again, does the system time drift, or just the application logs?
--I didn't find anything which say there is a system time drift, and also didn't find anything in application log which could have cause this time drift issue.

Quote:
So its not happening any longer...so what is the problem with your system, then??
--This is an audit trail issue logged on application consider to be critical , i need to figure out what could have cause this issue. So i can mitigate and prevent such issues in future.

Quote:
Ok...and is the guest OS drifting?? The VM Host time correct? Do you have the guest set to get time from host?
--No, Guest OS is not drifting, VM HOST is in sync with few seconds acceptable drift. No we have not set the guest to get time from host.

Quote:
Ok, but if you're not having the issue any longer, it would seem there's not much point to your thread.
-- audit trail issue logged on application consider to be very critical , i need to figure out what could have cause this issue. So i can mitigate and prevent such issues in future. Request your inputs or any ideas on what could have cause this issue.

Thanks,
Raj

Given what you’ve said so far I suspect some glitch in whatever program wrote the log file in question.
Look at:
The change history of that application.
Look at changes before the issue occurred that may have caused the problem as well as changes afterwards that may have corrected it.

I don’t see how anyone here can help you with that unless you provide a LOT more information.

1 members found this post helpful.

The application log will not tell you anything more than the program was written to tell you, so what do you think you'll find there??? I bolded some sections above for emphasis only. I suggest you have your systems administrator read them, because:

• You say you're NOT having a problem now
• You haven't for months
• The OS is working fine and is in sync
• The guest is working fine, and is in sync
• The **ONLY** thing having a problem **IS YOUR APPLICATION**
• ...but you won't look at it, because it went through 'internal review'.

Sorry, but there is absolutely nothing we can help you with. The only thing broken is whatever java application that was written in-house. Since those programmers work for you, have the source, and actually know what the program is, what it does, and how....TELL THEM TO FIX IT.

1 members found this post helpful.