newbie needs help for down webserver

natcire · 05-09-2005, 07:31 PM

Hello:

Am a Linux newbie still learning...Am looking for any kind of help to restart my personal Redhat 4 ES webserver. Installed RH about 3 months ago. Everything was configured and running smoothly: web server and email server. Updated patches several times whenever they were available.

This weekend, webserver went down...
Tried restarting services:
In Service configuration, httpd status says "httpd is stopped'
When I try to restart httpd service the Error message states:
httpd failed. The error was: Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]

Tried
netstat -tlnp 'egrep ":80 |:443"
as stated in RH knowledge base with no replies of any app running on port 80 or 443.

In command prompt I tried another Redhat knowledge base solution:
service httpd restart
Feedback:
Stopping httpd [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
No listening sockets available, shutting down
Unable to open logs.
[FAILED]

Need help...

Natcire

zeos · 05-09-2005, 09:05 PM

Are you sure with the netstat syntax you posted? It should be netstat -tlnp | egrep ":80 |:443" or you can use lsof -i:80 .

It definatly sounds like a "stuck" httpd process on the box...

natcire · 05-09-2005, 09:11 PM

Zeos:

Thanks for your reply.

Tried your suggestion of netstat -tlnp | egrep ":80 |:443"
and nothing...no replies...meaning no apps running on :80 and :443, correct?

natcire

zeos · 05-09-2005, 11:45 PM

Thats pretty odd ....are you in a posistion to check the logfiles for any indication of error (/var/log/apache/error_log, /var/log/messages ...etc...)?

What if anything happened between the time when it worked and now? ...has any software been installed or updated, if so what?

Also, have a look @ this thread: http://www.linuxquestions.org/questi...hreadid=311242

Guy was having similar, after we went back and forth a bit it turns out his hosts file wasn't configured correctly....

natcire · 05-10-2005, 01:24 AM

Zeos:

Thanks for the reply and the link...yes, I did see that thread before I posted. I remember doing one of your recommendations to the thread you attached this afternoon before posting my help request.

mail server root: "Isof -i :80"
bash: lsof -i :80: command not found.

Also there was another suggestion you gave that I tested:
apachectl configtest
That came back normal...can't remember the actual server response but there was no error...something like "Passed" or close to that.

My apologies, I am not at the office where the server is. Will follow your instructions on checking for error logs tomorrow and post my findings. I did go into the folder to view the error logs but nothing unusual.

On what has happened since day 1. Installed ES and configured the Dell server to host website and email in late February. All was going great. Each time there was a new update from Redhat I updated the server. The last update I did was last Friday. We (my friend xena and I) ssh into the server to upload test files regularly. Yesterday, on upload I wanted to see the results but noticed could not get to the site. I am not sure when the last time it was running. We don't go to the website to check for it being up...guess that is something we need to do more frequently (on my employer's Window's server we have it ping for my server with "Isitup" but it pings it for the IP address and it shows it being up...the server is running and serving the email very well.)

IP addresses were into Dell server last Monday, May 2, for:
1- Asterisk server (another Linux learning project) and
2- website to host a test site, last Monday but that was only for virtual hosting.

Zeos, thanks for being helpful to us newbies...you are awesome.

Natcire

natcire · 05-10-2005, 12:34 PM

Zeos:

Good morning. In the office now. Tried to get to Linuxquestions.org via the Dell server to copy and paste the error log but got this message:
"The connection was refused when attempting to contact www.linuxquestions.org."

Viewed the 4 error logs back to April 10 and the first error log was Sunday May 08 04:02:12 2005 [notice] SIGHUP received. Attempting to restart (99) Cannot assign requested address: make_sock: could not bind to address 64.122.10.216:80 no listening sockets available, shutting down.
Unable to open logs.
I can post the error logs (really long) or send them to you for review. Most of the logs were hack attempts to gain entry to the server and few reboots after updating Redhat.

I noticed an error yesterday, IP address for Virtual Host server is 64.122.109.216, not 64.122.10.216. Somehow the 9 disappeared from the 109. That part was corrected yesterday. In System Settings/Server Settings/HTTP/Virtual Hosts, the server is listed as Virtual Host ) with IP address of 64.122.109.216. It is IP based Virtual Host.

Server Settings 64.122.109.216 has the correct IP in there with port 80. Under Server Settings: Lock File: /var/lock/httpd.lock; PID File: /var/run/httpd.pid; and Core Dump Directory: /etc/httpd. Odd, there are no files in the folders with those names.

On reboot, get error message: snmpd Error; shutting down snmpd failed; Killall error; lm_senors: no sensors found; No Mountpoints defined; unable to open logs: Failed.
Sites are still down after reboot.

One thing that may or may not have a consequence, but probably not though some Company servers were affected. One of the company's DNS server, 64.122.109.10 died last Tuesday. It is the secondary, primary is 64....9. And on this server is it entered as .9 as first and .10 as second. The net admin has moved .10 to another server so it is working as of last Tuesday.

Zeos, thanks in advance, for your expertise and time.

natcire

zeos · 05-10-2005, 12:46 PM

Hrmm ....several things I'm seeing that seem odd, they may all be related to the dns failure you describe...

First, are you able to connect to other websites via the box (you noted that you were unable to connect to lq)?

Something to try. As with the other thread, apache won't start if 1) there is a dns lookup failure AND 2) you are attempting to run mod_unique_id. Just for testing, check your /etc/httpd/conf/httpd.conf and look for the line which loads mod_unique_id

Code:

LoadModule unique_id_module modules/mod_unique_id.so

and comment if out. Then try starting apache...

You need apache to be able to resolve the hostname you've assigned it (ServerName directive in httpd.conf) either via DNS or your hosts file...

natcire · 05-10-2005, 01:16 PM

Zeos:

Thanks for reply.

Thought it might be due to DNS failure but the DNS server failed on Tuesday and Webserver failed on Sunday...five days later.

Server is able to access other sites: apple.com, costco.com, pemex.com etc.

/etc/httpd/conf/httpd/conf -gedit has nothing in page. Blank, nothing in there. Can that be the problem?

zeos · 05-10-2005, 01:54 PM

look around the directory /etc/httpd/conf/ for your apache configuration file ...on my ES3 boxen, it's httpd.conf ....

natcire · 05-10-2005, 02:36 PM

Zeos:

You won't believe it! It works now...solution?

I had recruited a friend to set it up the server with me originally. He is not a Linux wizard...a Windows kinda guy, we are both learning Linux but he is smarter as you will see. He was away from communication for three days (he had to go fix a broken toilet and pipeline in his vacation home...no phone there). He saved the apache and email config to his user folder. So when he finally got back online to chat I told him about the problem and he told me about his secret stash! Told me to just copy his apache config to /etc/httpd/conf/httpd.conf. Restarted Apache without a hitch...

Maybe that should be a common practice others should use...save a working config version prior to updates to user folder.

Zeos, thanks for the awesome assistance.

Anything I can do for you in return?

Natcire.

zeos · 05-10-2005, 03:08 PM

Glad ya got it working ...and yes, keeping backups of /etc/* is a very good idea (I have a cron job on my servers to rotate a set of backups that include these files on a weekly basis)

As for what to do .....just help someone else who needs it when/if ya can ...thats what this stuff's all about...

Good luck with your system, and have fun with it...

natcire · 05-10-2005, 04:19 PM

Zoes:

Thanks for all your help...so two last question....

You mentioned:
"(I have a cron job on my servers to rotate a set of backups that include these files on a weekly basis)"

What is the process/instructions etc...automate it.

Is there a script or app to install that will check for website being up? Sorry for such a novice question.

Eric

zeos · 05-10-2005, 04:53 PM

It's just a simple shell script that I created to perform menial/repetitive tasks. I'd post it, but it wouldn't do you much good at all since it's tuned to my server(s) file locations and data....What you basically want to do is:

Code:

#!/bin/bash
#set to the desired backup directory -make sure it exists
BACKLOC=/backup
export BACKLOC

DTE=`date +%F`
export DTE

mv $BACKLOC/etc-*.tar $BACKLOC/lastweek-etc.tar
tar -cpP /etc/ > $BACKLOC/etc-$DTE.tar

This is quickly thrown together off the top of my head and will function, but it can be alot better/fancier but would more or less server the purpose for backing up /etc/ to a tar file, saving the prior weeks backup to lastweek-etc and this weeks to etc-xxxx-xx-xx.tar.

Create the script (edited to suit your needs), make it executable (chmod +x) and schedule it to run via a cron job.

It's no substitute for regular, normal backups since it doesn't protect against disk failure or other catastrophic events, but is more or less meant to recover from the occasional "oops" in the /etc directory

As to the second question, search freshmeat.net for "server monitor" ...there's gads of applications out there that will do just that...

natcire · 05-10-2005, 07:12 PM

Zeos:

With all my heart...Thank You, you have been amazingly helpful.

Soon I will try to repay to the community, someway. Definitely not scripting Linux commands.!

Eric