Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Apologies for my first post here being a question, but that's the whole point of the site so I'm sure you guys are used to it
OK. Here is my situation. I came home this evening to find my server not responding to HTTP connections. After a little digging, I discovered that the httpd service was stopped. Upon starting that, I tried to load a page and was greeted with a mysql error. It turns out that the mysqld was down too. At this point I rebooted the box, and all services are now up. I then started to look at logs.
I can't find any sign of a crash or anything unusual so I was wondering what this could be?
Here are a few commands/logs that may help this issue.
Code:
[root@groomi httpd]# who -b
system boot Nov 10 11:44
The history command shows no obvious signs of tinkering and none of the services were stopped from the command line as root.
Looking a /etc/passwd, there is only the user Root with the guid 0, and all other users' shell is /sbin/nologin (except the top 5-6 lines)
The last SSH login to root prior to today was 23rd Nov
Actually, as your last shows, its not Fedora anything, it's Centos 5.3 (free version of RHEL).
FYI, RHEL/Centos have been on 5.4 for a little while now; you could update...
As for your problem I'd look at /var/log/messages; you may need to go back to the prev file.
If you've got a /var/log/audit or /var/log/secure, check those too.
Actually, as your last shows, its not Fedora anything, it's Centos 5.3 (free version of RHEL).
FYI, RHEL/Centos have been on 5.4 for a little while now; you could update...
As for your problem I'd look at /var/log/messages; you may need to go back to the prev file.
If you've got a /var/log/audit or /var/log/secure, check those too.
Hey there,
Thanks for your reply. You are right, it is centOS but to me they're one and the same thing as they all use the same package manager, file structure, etc etc
Anyway, bizarrely there are none of the files you mentioned. There is, but they are empty, all 5 copies.
Code:
[root@groomi log]# ls -l
total 239048
-rw------- 1 root root 103558 Dec 3 10:30 apf_log
-rw------- 1 root root 131611 Nov 29 02:15 apf_log.1
-rw------- 1 root root 131595 Nov 22 02:15 apf_log.2
-rw------- 1 root root 148082 Nov 15 02:15 apf_log.3
-rw------- 1 root root 147609 Nov 8 02:15 apf_log.4
-rw------- 1 root root 129 Dec 3 07:30 bfd_log
-rw------- 1 root root 0 Nov 22 02:15 bfd_log.1
-rw------- 1 root root 0 Nov 15 02:15 bfd_log.2
-rw------- 1 root root 0 Nov 8 02:15 bfd_log.3
-rw------- 1 root root 1415 Nov 6 07:51 bfd_log.4
-rw------- 1 root utmp 223079808 Dec 3 09:33 btmp
-rw-r--r-- 1 root root 0 Apr 17 2009 dmesg
-rw------- 1 root root 16064 Apr 18 2009 faillog
drwx------ 2 root root 4096 Nov 29 02:15 httpd
-rw-r--r-- 1 root root 146584 Dec 3 10:34 lastlog
drwxr-xr-x 2 root root 4096 Mar 15 2007 mail
-rw------- 1 root root 0 Nov 29 02:15 maillog
-rw------- 1 root root 0 Nov 22 02:15 maillog.1
-rw------- 1 root root 0 Nov 15 02:15 maillog.2
-rw------- 1 root root 0 Nov 8 02:15 maillog.3
-rw------- 1 root root 0 Nov 1 02:15 maillog.4
-rw------- 1 root root 0 Nov 29 02:15 messages
-rw------- 1 root root 0 Nov 22 02:15 messages.1
-rw------- 1 root root 0 Nov 15 02:15 messages.2
-rw------- 1 root root 0 Nov 8 02:15 messages.3
-rw------- 1 root root 0 Nov 1 02:15 messages.4
-rw-r----- 1 mysql mysql 20330648 Dec 3 10:32 mysqld-slow.log
-rw-r----- 1 mysql mysql 30330 Dec 2 17:24 mysqld.log
drwxr-xr-x 2 root root 4096 Apr 18 2009 pm
-rw------- 1 root root 79234 Jun 17 14:08 rkhunter.log
drwx------ 2 root root 4096 Jan 21 2009 samba
-rw------- 1 root root 0 Nov 29 02:15 secure
-rw------- 1 root root 0 Nov 22 02:15 secure.1
-rw------- 1 root root 0 Nov 15 02:15 secure.2
-rw------- 1 root root 0 Nov 8 02:15 secure.3
-rw------- 1 root root 0 Nov 1 02:15 secure.4
-rw------- 1 root root 0 Nov 29 02:15 spooler
-rw------- 1 root root 0 Nov 22 02:15 spooler.1
-rw------- 1 root root 0 Nov 15 02:15 spooler.2
-rw------- 1 root root 0 Nov 8 02:15 spooler.3
-rw------- 1 root root 0 Nov 1 02:15 spooler.4
-rw------- 1 root root 0 Apr 18 2009 tallylog
-rw-rw-r-- 1 root utmp 134784 Dec 3 10:34 wtmp
-rw------- 1 root root 70418 Oct 19 10:08 xferlog
-rw-r--r-- 1 root root 13446 Oct 19 09:45 yum.log
[root@groomi ~]# last -n 7
root pts/1 x Thu Dec 3 11:12 still logged in
root pts/0 x Thu Dec 3 10:34 still logged in
root pts/0 x Wed Dec 2 17:28 - 18:08 (00:40)
reboot system boot 2.6.18-028stab06 Wed Dec 2 17:24 (17:48)
root pts/0 x Wed Dec 2 17:13 - down (00:10)
root pts/0 x Mon Nov 23 15:01 - 15:05 (00:03)
root pts/0 x Fri Nov 20 11:11 - 11:12 (00:01)
Hmm, empty messages file is almost unheard of; they are normally busy; anything that doesn't have its own log file and most logging system cmds log an entry there. The secure logfile logs logins etc, should also have some data.
Have you checked the syslog.conf file?
There's a small possibility you've been hacked, but don't panic yet, lets look at more evidence.
SOURCE ADDRESS: x
TARGET SERVICE: sshd
FAILED LOGINS: 30
EXECUTED COMMAND: /etc/apf/apf -d x {bfd.sshd}
and
Quote:
SOURCE ADDRESS: x
TARGET SERVICE: sshd
FAILED LOGINS: 66
EXECUTED COMMAND: /etc/apf/apf -d x {bfd.sshd}
Totally different looking IP ranges, don't know if that's normal or not..
Anyway, re: /var/log/messages, it seems that the syslogd wasn't running, another curious issue which I'd like to get to the bottom of. I couldn't find a syslogd in /etc/init.d, so I did a locate syslogd. Here is the relevant lines from the result:
So, should there be a rsyslogd in /etc/init.d? I have a clouded understanding of the purpose of rc.x and init.d, if someone could explain to me, and or tell me if my config above is correct, this it might help me somewhat .
More stuff:
Code:
[root@groomi ~]# cat /etc/rsyslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
I hope this helps
PS.
Forgot to add, I did an /etc/init.d/rsyslog start and the server is now running and logging messages in what I can only assume is a normal way.
Interesting; I thought that syslog was the default on Centos, not rsyslog. Did you install with defaults, or do you know who did/didn't.
Those are alternative services to do the same job (see also syslog-ng).
Can't get to my system to check atm.
Also, you only have shutdown scripts (Kxxx), there should be equiv startup scripts in those dirs, beginning with Sxxx.
That's definitely wrong; possibly suspicious.
Have you done any system alterations/updates recently?
The system is actually a Virtuozzo VZ using their default CentOS image.
The only system changes I've made is occassional package installs, and infrequent system updates. I have never had any direct interaction with those directories so this is definately suspicious.
There is NO syslog on this system that I can see.
Thanks for your reply, I'll delve into this a little more when I get the chance to access my system when I get home.
Woke up to the same situation today, this is getting desperate now...
HTTPD is not responding, all requests are being reset by the server. I go in and restart Apache, and now pages report "mysql: too many connections". I try to restart MySQLd but it will not work and seems to be hanging. I then reboot the box and it comes up all happy and normal.
Can anyone suggest something I can post up to get to the bottom of this?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
Apache and MySQL stopped, why?
