LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 01-26-2011, 12:17 PM   #1
jsp_1983
LQ Newbie
 
Registered: Dec 2010
Posts: 29

Rep: Reputation: 0
CentOS 5.5 VPS web server has 'fallen over' - how to troubleshoot?


I have a VPS that I'm using as a web/mail/DNS server (all-round server) and is repeatedly falling over every 12 hours or so.

It seems to be since I used a Wordpress plugin on one of my sites to send out an email newsletter - I had problems with the plugin itself, anyway.

I think it might be causing a memory issue, as I've run 'free' and 'top', which sometimes causes a "fork: Cannot allocate memory" message to appear. Other times it displays statistics and suggests that there's plenty of memory available.

What I'd like to know is, how can I start to isolate and troubleshoot this particular problem? Are there logs that I should look at and if so, what should I be looking for?

Also, how can I stop this rogue plugin from doing anymore damage, seeing as it still appears to be running?
 
Old 01-26-2011, 01:08 PM   #2
xeleema
Member
 
Registered: Aug 2005
Location: D.i.t.h.o, Texas
Distribution: Slackware 13.x, rhel3/5, Solaris 8-10(sparc), HP-UX 11.x (pa-risc)
Posts: 987
Blog Entries: 4

Rep: Reputation: 249Reputation: 249Reputation: 249
Greetingz!

The OOM (Out of Memory) Killer may be enabled in the kernel. Check the following;
cat /proc/sys/vm/oom-kill
0 = Off
1 = On

To turn it off, do this;
echo 0 > /proc/sys/vm/oom-kill
However, if the system panics, you might want it to auto-reboot.
echo number_of_seconds > /proc/sys/kernel/panic

Note that if the OOM Killer is the culprit, you can edit /etc/sysctl.conf to make the settings stick.
(If you don't have an /etc/sysctl.conf, find and update your rc.local script with the commands from above)
Code:
# reboot 9 secs after panic
kernel.panic = 9
# turn off oom-killer
vm.oom-kill = 0
Logs:
Well, for starters there's always the syslog log(s) (possibly /var/log/messages), and the output you get from "dmesg". You may also just have a process that's SEGFAULT'ing (look for a core file with a recent time/date stamp).

Hope this helps!

Last edited by xeleema; 01-26-2011 at 01:10 PM. Reason: added log info.
 
Old 01-27-2011, 08:23 AM   #3
jsp_1983
LQ Newbie
 
Registered: Dec 2010
Posts: 29

Original Poster
Rep: Reputation: 0
Okay, so I did this first: cat /proc/sys/vm/oom-kill

And nothing was returned, so I presume OOM isn't enabled in that case.

'dmesg' doesn't return anything on-screen, whilst cat'ing the messages file seems a bit unwieldy - is there a way to tidy up how it displays log messages?
 
Old 01-27-2011, 02:12 PM   #4
xeleema
Member
 
Registered: Aug 2005
Location: D.i.t.h.o, Texas
Distribution: Slackware 13.x, rhel3/5, Solaris 8-10(sparc), HP-UX 11.x (pa-risc)
Posts: 987
Blog Entries: 4

Rep: Reputation: 249Reputation: 249Reputation: 249
Quote:
Originally Posted by jsp_1983 View Post
Okay, so I did this first: cat /proc/sys/vm/oom-kill
And nothing was returned, so I presume OOM isn't enabled in that case.
Correct! That narrows it down a bit.
Quote:
Originally Posted by jsp_1983 View Post
'dmesg' doesn't return anything on-screen
Hm, okay. That's good, but a lil suspicious that nothing there indicates a problem...
Quote:
Originally Posted by jsp_1983 View Post
cat'ing the messages file seems a bit unwieldy - is there a way to tidy up how it displays log messages?
Yes, "tail" and "egrep" are your friends;

Code:
tail -100 /var/log/messages | egrep -v "exclude this|and that|^starting_with|or_ending_with$"
-100 = How many lines off the end of the file you want to look at.
| = We "pipe" the output to the egrep command (to help filter out stuff we're not interested in"
"exclude this|and that|^starting_with|or_ending_with$" = this is our filtering string. Since we used "-v", this is going to be stuff we exclude. Without the "-v", egrep would think this was stuff we were looking for, instead. (We don't know exactly what we're looking for, so we just identify and eliminate stuff that isn't useful.)

I'll break the syntax down here

"exclude this|and that|^starting_with|or_ending_with$"
"exclude this" = any line that contains this exact string will be excluded. You can do this multiple times, the trick is to pick a unique part of the line you don't want. (Bad Example: if you pick "sda" when cutting up the output from dmesg, a lot more than just hard drive info will not get displayed)

"exclude this|and that|^starting_with|or_ending_with$"
"and that" = This is an example of another 'unique' string we're excluding from the display. Note that both "exclude this" & "and that" are separated by the pipe character (|). All of our critera has to be separated by pipes, and the whole thing has to be wrapped in one pair of double-quotes.

"exclude this|and that|^starting_with|or_ending_with$"
"^starting_with" = This string starts with a caret (^). That means any line beginning with $STRING is going to be excluded.

"exclude this|and that|^starting_with|or_ending_with$"
"or_ending_with$" = Here we have a string that ends with a doller sign. Not to be confused with an environment varible (which has a dollar sign at the beginning of the string). This will search for any line that ends in the string we specified.

Note that in order to include "special characters" in your search (rather than trigger their special function), you will have to "escape" them with a backslash. Check the man page for egrep for more detail, or google-up "Regular Expressions" if you're really curious.
 
Old 01-29-2011, 08:35 PM   #5
jsp_1983
LQ Newbie
 
Registered: Dec 2010
Posts: 29

Original Poster
Rep: Reputation: 0
Wow... That's quite a detailed reply! Many thanks for taking the time to help

I'll have a play with egrep and become a bit more familiar with it, before I start looking through error logs.

Thanks :~)
 
Old 01-29-2011, 09:00 PM   #6
trist007
Member
 
Registered: May 2008
Distribution: Slackware
Posts: 983

Rep: Reputation: 56
That fork error may suggest that your plugin is leaving zombies and eventually your process table gets full to where you cannot create any more processes. Not sure how you can troubleshoot this, I'd like to know but if it is the case you need to setup some wait functions to clean those zombies up.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
CentOS 5.3 VPS and RealVNC TheN00B Linux - Server 1 03-11-2010 11:36 AM
Centos Server Troubleshoot Tips LGX Linux - Newbie 4 12-17-2009 10:47 PM
How to audit a CentOS VPS (Virtual Private Server) wek888 Linux - Newbie 2 09-09-2009 06:16 PM
web server setup VPS dujlinvik Linux - Server 2 05-07-2009 04:22 AM
Where can I download UML VPS or Xen VPS to make a virtual private server? abefroman Linux - Software 3 12-09-2005 10:00 AM


All times are GMT -5. The time now is 11:32 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration