hi,
My server crashes continously after some time and the system restarts.


The /var/log/messages file contains the following line:

indianboy logger : weblogs: (6671) failed to acquire lock - another weblogs running?

someone adviced me to change rotating weblogs from 10 to 30 min in /etc/crond.d/weblogs.d file
it now contains */30 * * * * root /usr/local/bin/weblogs
But the server is still crashing.

Please help me. Its very urgent.


SAM.

Can you please describe what weblogs does? Also, do you have console access to the web server and if so doesn't it just reboot or actually go through initlevel changes to reboot?

Hi,
"Weblogs" is a perl script which creates a log file for the web server and rotate it to some other place. but i didn't know much about that. And secondly when the web server crashes, the whole system reboots through initlevel changes.

Please help me.

Hi

Even if the webserver crashes, the wholse system shouldn't reboot. So perhaps its a hardware failure?

From the symtoms, it might be it crashes because of heat. A test for this is to open the box and start it while it's open. Does all the fans spin right, or are some filled with dust? Try to get some airflow into the box, and see if it still crashes.

Hi,

I tested my hardware and its working fine. Is there any kind of software problem.
Well i am posting the weblogs script file. see if there is any problem in it.

Thanks,
SAM.


#! /usr/bin/perl -w
#
# Copyright (C) Ensim Corporation 2002. All Rights Reserved.
#
# This software is furnished under a license and may be used and copied
# only in accordance with the terms of such license and with the inclusion
# of the above copyright notice. This software or any other copies thereof
# may not be provided or otherwise made available to any other person. No
# title to and ownership of the software is hereby transferred.
#
# The information in this software is subject to change without notice and
# should not be construed as a commitment by Ensim Corporation. Ensim
# assumes no responsibility for the use or reliability of its software on
# equipment which is not supplied by Ensim.
#

use strict;
# use warnings "all"; # debugging

my %SITE = ();
my $_PID = $$; # used for locking
my $LDIR = "/etc/logrotate"; # Ensim logrotate directory

# Python locking code
my $LOCK_CODE = <<_END_LOCK_CODE;
/usr/bin/ensim-python -c "
from vh3 import virthost
from vh3 import virtutil
import sys
err = virtutil.lock_file('weblogs', virtutil.LOCK_NB, pid = $_PID)
status = virthost.cli_display_status_list([(virthost.LOCK_MGMT, virthost.GET_LOCK, err)])
if status == virthost.ERROR:
sys.exit(1)
else:
sys.exit(0)
"
_END_LOCK_CODE

# Python unlocking code
my $UNLOCK_CODE = <<_END_UNLOCK_CODE;
/usr/bin/ensim-python -c "
from vh3 import virthost
from vh3 import virtutil
import sys
err = virtutil.unlock_file('weblogs', pid = $_PID)
status = virthost.cli_display_status_list([(virthost.LOCK_MGMT, virthost.RELEASE_LOCK, err)])
if status == virthost.ERROR:
sys.exit(1)
else:
sys.exit(0)
"
_END_UNLOCK_CODE

# Writes log messages to /var/log/messages
#
sub
syslog($)
{
my $msg = shift;
my $logcmd = "/usr/bin/logger -p warn \"weblogs: ($$) $msg\"";
system($logcmd);
}

# Exit code for problematic cases
#
sub
abort($)
{
my $msg = shift;
syslog($msg);
system($UNLOCK_CODE);
exit 1;
}

# Exit code for AOK cases
#
sub
done()
{
system($UNLOCK_CODE);
syslog("done.");
exit 0;
}

# HUPing Apache
#
sub
apHUP()
{
system("/etc/init.d/httpd reload");
system("/etc/init.d/httpd13 reload");
}

# Populates site hash table
#
sub
getsite($) # siteN
{
my $siteN = shift;
my $info = "/home/virtual/$siteN/info/current";
if ((! -e $info) || (! -d $info)) {
syslog("info filesystem missing for site $siteN !");
}

my $file;
my $rec = {};
my @files = qw(analog proftpd siteinfo webalizer weblogs);
foreach $file (@files) {

my $fn = "$info/$file";
open(S, "<$fn") or return undef;
my @S = <S>;
close(S);

foreach $_ (@S) {
chomp;
if (/^\s*([^#\s]+)\s*=\s*([^#\s]+)\s*$/) {
my $k = $file . "_" . $1;
$rec->{$k} = $2;
}
}
}

$SITE{$siteN} = $rec;

return 0;
}

# Rotates logs for a given site (chroot, as real root)
#
sub
do_logrotate($) # siteN
{
my $siteN = shift;

my $root = "/home/virtual/$siteN/fst";
if ((! -e $root) || (! -d $root)) {
syslog("root filesystem missing for site $siteN!");
}

my $c = "/usr/sbin/chroot $root /usr/sbin/logrotate -f /etc/logrotate.conf";
my $ret = system($c);

if ($ret != 0) {
syslog("logrotate failed for $siteN");
}

unlink("$LDIR/now/$siteN");

syslog("logrotate done for $siteN");

return $ret;
}

# Runs Webalizer for a given site (chroot, su to admin_user)
#
sub
do_webalizer($) # siteN
{
my $siteN = shift;
my $rec = $SITE{$siteN};

if (!$rec || ($rec->{webalizer_enabled} == 0)) { # nothing to do
return 0;
}

my $root = "/home/virtual/$siteN/fst";
if ((! -e $root) || (! -d $root)) {
syslog("root filesystem missing for site $siteN !");
}

my $admin = $rec->{siteinfo_admin_user};
if (!$admin) {
syslog("failed to determine admin user for site $siteN");
return 1;
}

my $chcmd = "/usr/sbin/chroot $root ";
my $sucmd = "/bin/su $admin - /bin/sh -c ";
my $wacmd = "/usr/bin/webalizer -c ";
my $web = "/etc/webalizer/web/webalizer.conf";
my $ftp = "/etc/webalizer/ftp/webalizer.conf";

my ($ret1, $ret2) = (0, 0);

if ($rec->{weblogs_enabled} == 1) {
$ret1 = system("$chcmd $sucmd \"$wacmd $web\"");
if ($ret1 != 0) {
syslog("webalizer returned non-zero exit status for $siteN (web)");
}
}

my $chmod_ret = 0;
if ($rec->{proftpd_enabled} == 1) {
if ( -f "$root/var/log/xferlog.1" ) {
$chmod_ret = chmod 0644, "$root/var/log/xferlog.1";
if ($chmod_ret != 1) {
syslog("weblogs could not chmod xferlog.1");
} else {
$ret2 = system("$chcmd $sucmd \"$wacmd $ftp\"");
if ($ret2 != 0) {
syslog("webalizer returned non-zero exit status for $siteN (ftp)");
}
}
}
}

if (!(($ret1 == 0) && ($ret2 == 0) && ($chmod_ret == 0))) {
return 1;
}

return 0;
}
# Runs Analog for a list of sites (chroot, real root)
#
sub
do_analog($) # [siteN, ...]
{
my $sites = shift;
my @sites = @{$sites};

if ($#sites < 0) {
return 1;
}

my @siteN = ();
my @ftp = ();
my @web = ();

foreach $_ (@sites) {
my $rec = $SITE{$_};
if ($rec && ($rec->{analog_enabled} == 1)) {
push @siteN, $_;
push @ftp, $rec->{proftpd_enabled};
push @web, $rec->{weblogs_enabled};
}
}

my $L_siteN = "'" . join('\',\'', @siteN) . "'";
my $L_ftp = join(',', @ftp);
my $L_web = join(',', @web);

my $ANALOG_CODE =<<_END_ANALOG_CODE;

/usr/bin/ensim-python -c "

from vh3 import virthost
import getopt
import sys
import cmdlnpopen
import ensimglobals
import os
import traceback

def invoke_analog(siteN, ftp, web):
prefix = '/home/virtual/%s/fst' % siteN
ana_web_cmd = '/usr/sbin/chroot ' + prefix + ' /usr/lib/opcenter/analog/updatestats.pl /var/usage/web /var/log/httpd/access
_log.1 '
ana_ftp_cmd = '/usr/sbin/chroot ' + prefix + ' /usr/lib/opcenter/analog/updatestats.pl /var/usage/ftp /var/log/xferlog.1'

analogsvcs = []

if web == 1:
out, err, ret = cmdlnpopen.cmd3(ana_web_cmd)
if ret == 0:
analogsvcs.append('web')
else:
ret = 0

if ftp == 1:
out1, err1, ret1 = cmdlnpopen.cmd3(ana_ftp_cmd)
if ret1 == 0:
analogsvcs.append('ftp')
else:
ret1 = 0

if len(analogsvcs) == 0:
return 0

if ret == 0 and ret1 == 0:
sys.path.append('%s/analog' % ensimglobals.OCW_HOME)
from genAnalogFunctions import genAnalogFunctions
sys.path.append('%s/published' % (ensimglobals.OCW_HOME))

genAnalogFunc = genAnalogFunctions()

try:
lfd =open('%s/fst/etc/analog.langpref' % virthost.domain_path(siteN),'r')
lpref =lfd.read()
lfd.close()
except:
lpref ='en_US'

genAnalogFunc.langpref =lpref

for analogtype in analogsvcs:
reportpath = '%s/fst/var/usage/%s' % (virthost.domain_path(siteN),analogtype)
if ( not os.path.exists(reportpath) ):
continue

try:
dirfiles = os.listdir(reportpath)
except OSError:
traceback.print_exc(file=sys.stderr)
sys.exit(1)

filterfiles = genAnalogFunc.filterfiles(dirfiles)
infp = None
outfp = None
for file in filterfiles:
try:
infp = open('%s/%s'%(reportpath,file),'r')
data = infp.readlines()
except IOError, err:
traceback.print_exc(file=sys.stderr)
sys.exit(1)
infp.close()

reports = genAnalogFunc.parseAnalogData(data)
try:
filename = '%s/%s.html' % (reportpath, file)
outfp = open(filename, 'w')
genAnalogFunc.display_reports(reports,outfp)
os.chmod(filename, 0640)
except IOError, err:
traceback.print_exc(file=sys.stderr)
sys.exit(1)
outfp.close()

return 0

map(invoke_analog, [$L_siteN], [$L_ftp], [$L_web])
"
_END_ANALOG_CODE

system($ANALOG_CODE);

return 0;
}

sub
do_webalizer_generate($) # siteN
{

my $siteN = shift;
my $rec = $SITE{$siteN};

if (!$rec || ($rec->{webalizer_enabled} == 0)) { # nothing to do
unlink("$LDIR/now/genreports/$siteN");
return 0;
}

my $root = "/home/virtual/$siteN/fst";
if ((! -e $root) || (! -d $root)) {
syslog("root filesystem missing for site $siteN !");
}

my $admin = $rec->{siteinfo_admin_user};
if (!$admin) {
syslog("failed to determine admin user for site $siteN");
unlink("$LDIR/now/genreports/$siteN");
return 1;
}

my $chcmd = "/usr/sbin/chroot $root ";
my $sucmd = "/bin/su $admin - /bin/sh -c ";
my $wacmd = "/usr/bin/webalizer -c ";
my $web = "/etc/webalizer/web/webalizer.conf";
my $ftp = "/etc/webalizer/ftp/webalizer.conf";
my $weblog= "/var/log/httpd/access_log"; # since only generation we pass this parameter on the command line
my $ftplog= "/var/log/xferlog";

my ($ret1, $ret2) = (0, 0);

if ($rec->{weblogs_enabled} == 1) {
$ret1 = system("$chcmd $sucmd \"$wacmd $web $weblog\"");
if ($ret1 != 0) {
syslog("webalizer returned non-zero exit status for $siteN (web)");
}
}

my $chmod_ret = 0;
if ($rec->{proftpd_enabled} == 1) {
if ( -f "$root/var/log/xferlog" ) {
$chmod_ret = chmod 0644, "$root/var/log/xferlog";
if ($chmod_ret != 1) {
syslog("weblogs could not chmod xferlog");
} else {
$ret2 = system("$chcmd $sucmd \"$wacmd $ftp $ftplog\"");
if ($ret2 != 0) {
syslog("webalizer returned non-zero exit status for $siteN (ftp)");
}
}
}
}
if (!(($ret1 == 0) && ($ret2 == 0) && ($chmod_ret == 0))) {
unlink("$LDIR/now/genreports/$siteN");
return 1;
}

unlink("$LDIR/now/genreports/$siteN");
return 0;
}

sub
do_fplogs($) # siteN
{
my $siteN = shift;
syslog("Generating fp logs.");
my $httpd_test_logdir="var/log/httpd-test";
my $httpd_prod_logdir="var/log/httpd";

open(MYSITE,"/home/virtual/$siteN/info/current/frontpage") || warn("Unable to open\n");
my @file = <MYSITE>;
close(MYSITE);
my ($line,$fpenabled);
foreach $line (@file){
if( $line=~ m/enabled/){
chomp($line);
$line=~ s/\s*enabled\s*=\s*(.*)/$1/g;
$fpenabled = $line;
}
}

if ( $fpenabled eq 1) {
# Test server
if ( -e "/home/virtual/$siteN/fst/$httpd_test_logdir/access_log" ) {
chdir("/home/virtual/$siteN/fst/$httpd_test_logdir");
# To make sure cronosplit is running on right file
my $ret = system("/usr/local/sbin/cronosplit --template='access_log.%m-%d-%y' access_log");
if ($ret == 0) { syslog("fp logs for $siteN generated for test server"); }
}

# Production server
if ( -e "/home/virtual/$siteN/fst/$httpd_prod_logdir/access_log" ) {
chdir("/home/virtual/$siteN/fst/$httpd_prod_logdir");
my $ret1 = system("/usr/local/sbin/cronosplit --template='access_log.%m-%d-%y' access_log");
if ($ret1 == 0) { syslog("fp logs for $siteN generated for production server"); }
}
}
syslog("Generated fp logs");
return 0;
}

sub
do_fp_usage_anal()
{
opendir(FPDIR, "/usr/local/frontpage") or die("Cannot open /usr/local/frontpage");

my @cnfFiles = grep /\.cnf$/, readdir FPDIR;

foreach $_ (@cnfFiles) {
if (/(we|.*\(\d+)\.cnf/) {
my $hostHeader = '';
if ($1 ne 'we') {
chop ($hostHeader = $1);
}
my $port = $2;
LaunchUsage($port, $hostHeader);
}
}
syslog("Done with the frontpage usage analysis");

closedir FPDIR;
}

sub LaunchUsage {
my ($port, $hostHeader) = @_;
my $cmdLine = "/usr/local/frontpage/version5.0/bin/owsadm.exe -o usage -p $port";
$cmdLine .= " -m $hostHeader" if ($hostHeader);

# print `$cmdLine`;
}

sub
main()
{
my $ret;

syslog("starting.");

$ret = system($LOCK_CODE);
if ($ret != 0) {
syslog("failed to acquire lock - another weblogs running?");
exit 1;
}

# code for daily generation of webalizer reports

opendir D, "$LDIR/now/genreports" or abort("failed to open $LDIR/now/genreports");
my @sites = grep { /^site\d+$/ } readdir(D);
closedir(D);

opendir DD, "$LDIR/now" or abort("failed to open $LDIR/now");;
my @sites1 = grep { /^site\d+$/ } readdir(DD);
closedir(DD);

# comparing if the day is reached for weekly rotation
if ($#sites != -1) {
foreach $a (@sites) {
foreach $b(@sites1) {
if ($a eq $b) {
unlink("$LDIR/now/genreports/$a");
}
}
}

opendir D, "$LDIR/now/genreports" or abort("failed to open $LDIR/now/genreports");
my @sites = grep { /^site\d+$/ } readdir(D);
closedir(D);

foreach $_ (@sites) {
getsite($_); # Pull in siteinfo
do_fplogs($_); # generate the frontpage access logs
do_webalizer_generate($_); # generate webalizer reports
}
}

opendir D, "$LDIR/now" or abort("failed to open $LDIR/now");;
@sites = grep { /^site\d+$/ } readdir(D);
closedir(D);

if ($#sites == -1) { # nothing to do: no sites in 'now'
done();
}

# Pull in siteinfo
foreach $_ (@sites) {
getsite($_);
}

# code for taking care of fplogs
foreach $_ (@sites) {
do_fplogs($_);
}
do_fp_usage_anal();

# Logrotate
foreach $_ (@sites) {
do_logrotate($_);
}

# HUP Apache
apHUP();

# Analog
do_analog(\@sites);

# Webalizer
foreach $_ (@sites) {
do_webalizer($_);
}

done();
}

main();

# __END__

This strikes me as a bit peculiar. If the problem related to heat and you have some form of shutdown process then the server should shutdown not reboot. Reboot is a particular command. I think you need to check the syslog to establish what is happening when this occurs.

TigerOC is on the right track. Any critical event that would cause the system to change initlevels, would also document its error in the syslog. Let us know what you find.

Hi,
Sorry to respond so late for your post but there was some personnel problem. Please don't mind. Back to the problem: I got wrong info from the adminstrator. Actually the system doesn't start itself. Only the web server shuts down.Then the system needs to be restarted again manually for working again.
By the way where is this syslog placed. Are you refering to /var/log/messages file?
After changes in the /etc/crond.d/weblogs.d file
( */30 * * * * root /usr/local/bin/weblog )
server was running quite ok for 4 to 5 days but then again started to crash.
Is it something to do with the log files that weblog create.

Please help...

You'll find it in /var/log/syslog. That should give you the reason for the server exiting.

There is no such file in /var/log/

If you found /var/log/messages as stated above then /var/log does exist. This is a standard file structure in all Linux systems.

I doubt you actually have to do a restart. just use service <webserver> start (if it's redhat) or /etc/init.d/<webserver_script> start. here's what i use as a service script on debian:

save that code in a file called service in /usr/local/bin and chmod +x it

I mean to say that /var/log/ directory exists but there is no 'syslog' file in there. It contain other log files like messages, boot.log, etc.

Secondly creating such a service file may not require me to restart the system, but the problem of server crashing still exists. What i want is that the server must not hang. It must run continously without any problem.

syslog is standard in every distro I have seen because it records all events that occur and enable diagnostics to be performed. It is only accessible by root.

the file messages also contains a lot of useful info. on my debian box it contains everything that syslog contains.

although i understand that you need continous service, using that service thing is a lot quicker than rebooting