Hello,

There is a server which was installed 3 days back. It got Cent OS 5.4 64 Bit with 4 GB RAM. Apache, MySQL, PHP and vsFTPd were installed using yum.

Server was on the network and approx 100 visitors were on the website. All of sudden server got down. It stopping responding ping. Server guys told that server got down and after a request submission, they restarted server. However reason of the problem is still unknown.

I logged in to shell via SSH and checked /var/log/messages file, but didn't find any record which indicate cause of system shutdown. However dates in this file aren't in chronological order.

Is there any way or log file to check for the exact reason, why system got down. One reason could be hardware fault (for that I am going to ask hosting company to run diagnostics). Other reason could be a software/os bug. Is there any tool / log file to check and diagnose the cause.

Thanks.

I would be extremely suspicious of entries in any system log, being NOT in chronological order. What about kernel log file, and sulog if you have one? Have a look at root's shell history files as well. Anything strange in there? Any other log files showing missing entries, or entries being out of chronological order?

Have you got a rootkit checking tool on the machine (rkhunter, chkrootkits, etc..), that's been set up and run on the fresh install to have a baseline reference point, which you can compare to the current state of the machine?

Sasha

1 members found this post helpful.

What kind of request is it? HTTP?

Hi Sasha,

Thanks for your quick reply. I logged into server and checked history of root user. There were few command which weren't executed by me and those are appearing in last 30 commands. Those commands are for pinging yahoo, then modifying /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0 using nano (not sure if any change were made or not, but at least files were opened with nano). Then there is a command to restart network service, and then again 2 ping command for yahoo. These command may be executed by hosting company person after restarting server. Or other possibility may be that due to these commands, system was out of the network. Really clueless about these 6 commands.

I don't see /var/adm directory and also /etc/default/su file isn't there, so I think sulog isn't there. Following is content of /var/log folder:

@Red_Man:
Thanks, but actually its not just apache web server, but the server was out of network or physically down. And hosting provider rebooted it after a ticket submission. What I am trying to do is finding reason, why it was down. So that before opening my site for users I can resolve any possible fault.

As for the mystery commands: ASK the web hosting provider if they would have (if they DID) issue these commands, or if they did anything at all other than reboot the server. There must be a record of what happened: "no record" can be a record in itself.

And, examine all the logs, in particular: audit, (what is conman??), faillog, httpd, mail, mcelog, secure...

Asked web hosting provider about those commands and they told that after restarting server, data center guys may have executed these commands to check DNS and networking. Also checked audit log. That shows a lot of failed login attempts to ssh from various ip, which is fine for a public server, I think.

faillog file is empty. conman and conman.old are 2 directories which are empty. Under mail folder, there is a file named statistics, which that contain 2 binary lines. secure log file contains log in same manner as in audit and messages log file (dates aren't in chronological format). And it also contains entries for failed login attempts. mcelog file is empty.

So one thing I suspect that due to any reason server date setting went wrong which may have caused these date discrepancies (though any command for changing date wasn't issued).

httpd error_log is also showing date in strange order. Also it contains one error for MaxClient setting. Beside this, all other errors are for missing pages. And MaxClient thing is not supposed to power off or hand entire server (as its related to Apache only and it should affect only Apache server).

I find it very odd that the time changed backwards in the log by about 6 hours! Very weird. I have not seen that sort of behavior for any normal reason that I can think of. Could be caused by a very extreme change of the machine's clock, but something would have had to been wrong for quite some time for the machine's clock to get 6 hours ahead.

And yes, on a public IP, it's common to have a lot of bone-heads trying to get into SSH port. As long as there were no accesses (that you can tell) from unauthorized users. Make sure to have a VERY good password for publicly available SSH service, and it's also a good idea if possible to run SSH on a non-standard port, to make it harder to locate (a little bit anyhow)..

At this time, with the information you have provided so far, I do not know what to suggest now, except I will reiterate what I said about rkhunter and chrootkit -- you should have at least one of these, if not both, and maybe even something else in addition to these. I would also be monitoring the httpd logs from now forward, looking for weird traffic, particularly unsolicited outgoing traffic which should not exist on an inbound-serving webserver.

I don't want to get you excited about maybe the machine was hacked -- I do not know -- but it is possible it was hacked, DoS's or some other thing, but without any log record of this sort of thing, it's really hard to say. And, I am not a webserver forensics/security expert either. I am just going through the course of action that is prudent under the circumstances.

I have no problem moving this thread to the /Security forum if you would like; you may get quicker & more experienced input as to what may have happened, and/or where to look next.

Sasha

Hi Sasha,

Yes, this date/time issue is really strange. Because after having some entries for April 2, it got more entries for April 3.

I have asked hosting company guys to run a hardware diagnostic test on the server to ensure that hardware is in good state or even swap the server hardware. Once they complete the test, I am going to login to server again and will re-configure all services (MySQL, Apache, FTP, SSH etc). I don't think that it was due to a hack attempt or DoS attack. Because in both case log files were supposed to have some clue, like a lot of http requests.

You can move this topic to security category too. So that if anyone else have some idea about such event, he can suggest.

Thanks for your quick and great help.

Thanks

Moved to /"Linux - Security" for perhaps better exposure.

Sasha

I'm going to add my voice to the chorus. I may inadvertedly duplicate some questions slash answers but that will only be in an effort to be complete.

Not as root I hope?..


What was the exact message and when and how did they find out? Any details available? BTW, what HW are you running (specs)? (Para-)Virtualization guest?


Is the timestamp error exactly the same across log files? Does the "timestamp problem" show in all logs? Or only those written to by Syslog (see /etc/syslog.conf)?


Some HW problems manifest themselves on stderr and in /var/log/messages, goes the same for some SW problems ("BUG: time warp detected!"), shutdown and reboot commands are logged in /var/log/wtmp but there's no "one tool to rule all tools" when it comes to diagnosing problems.


Post the exact lines?


Parsing all those logs with Logwatch could show clues (or not).


In the event we may not be able to help there are some things you can do to help gather information:
- ask the service provider if they have additional (HW) monitoring to offer,
- enable local HW sensors if possible,
- install and run NTPd (just in case),
- run any form of SAR (Atsar, dstat, collectl, sar), and
- log everything Syslog logs to a remote server you (temporarily) re-purpose as syslog server.
There's more to do (check other users shell and access history, look for dumped cores, odd files in docroot and directories holding temporary files, verifying the installation) but (apart from what your service provider has on offer) these should have maximum effect and minimal impact.


I agree. While it may sound thrilling it is not in any crackers interest to bring a machine down. They may or may not care about detection but bringing a machine down doesn't let them (ab)use the machine and that is after all what they are interested in, right?..


Please let us know any outcome. Report slash details welcome.


Completely OT wrt the problem at hand but before you reconfigure the machine it might be a good idea to take stock of the complete machine. If you create a baseline before and apply versioning to configuring it will be easier to pinpoint performance losses slash gains while you reconfigure and test(!).