LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 07-07-2008, 09:46 AM   #1
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Rep: Reputation: 39
Question Apache serving blank pages??? Any ideas


Hi All,
I've got some sort of apache problem, what has happened recently is that apache has just stopped serving any files or blank pages. No error messages are outputted into any of the logs and apache appears to start-up fine...again no errors on debugging start-up.

The problem started after adding some more virtualhost files, now i know for sure its not the virtual host (VH) files themselves as if i delete some random files and then re-start apache...it runs fine again. EG: delete a few VH files then apache works, it has no baring on which ones i delete wheather they are recent ones i added or old ones that have been working for ages.

This lead me to beleive it was a file descriptor/open files problem, i've increased the limit in size by 8 times. This make no difference restore the VHs directory and apache restart the same happens again. Which leads me to beleive its not an open file problem as increasing it by 8 times would easily be enough for adding only 5 extra VH files.

Increased the openfile limit with:-
ulimit -H -n [number of files]
ulimit -S -n [number of files]
and edited the /etc/sys[something] making the hard and soft limits be set how i want after re-boot. And i've since rebooted and the limits changed to what i had set.

Could it be to many opened pipes?? there are alot of VHs i donno about 270ish, but other servers are all working fine.

lsof -p [httpd PID] | wc -l
reports about 1042 open files for each apache processes and apache on start-up in only spawning 6 times. Same as before.

Anyideas why would adding a few VHs stop apache working properly in any way.

Cheers Anyone
 
Old 07-08-2008, 02:32 AM   #2
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
Have you run with -e to increase the apache log level ?

What is the value of :

cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr
 
Old 07-08-2008, 05:08 AM   #3
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Hello, Mr.C

Thanks for replying, I had looked at file-max but not file-nr.

Outputs of those files are:-
cat /proc/sys/fs/file-max
209710

cat /proc/sys/fs/file-nr
23040 0 209710

Not liking the fact this middle number is '0', ive looked up what this means and that means there's nothing free left??? is that correct....is this the problem right here?

lsof | wc -l
464054

This is something i found interesting....
lsof | grep rotatelog | wc -l
457044

The server has got worse now...all i've changed is as i've mentioned above.....the ulimit for openfiles. Yes i've tried
/usr/local/apache2/bin/apachectl -e debug -k start
[Tue Jul 08 11:01:35 2008] [debug] mod_so.c(290): loaded module php4_module

and thats all the output i get, i also specified in the httpd.conf file itself that i want the loglevel to be debug.

Now the server still isn't producing any pages.
The way i know that no log files have been written to....
touch /time
find / -newer /time | grep -v '/proc'

In-between the touch and the find...i'm trying to get a web page...i get results in the find but nothing for apache logs.

Thanks for your help on this matter

Last edited by helptonewbie; 07-08-2008 at 05:14 AM.
 
Old 07-08-2008, 12:32 PM   #4
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
Yup, you are out of file slots.

It appears rotatelog is out of control, and can't keep up, or ... ?

I think this is the root of your problem. You've increased file limits, but they just get consumed completely by rotatelog.
 
Old 07-08-2008, 01:30 PM   #5
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Yes, i did notice this....the other thing that was strange that stopping the apache service on the box didn't stop/kill all of the rotatelog processes. Then starting apache again started more rotatelog processes than before. Keep going and you'd get a steady increase. Any ideas why that might be. Restarting the server as expected allows apache to serve pages again but i'd cut down massively on the amount of VHs apache was to use. Any ideas on what may cause such a rediculous use of rotatelog?

Cheers,
Mark
 
Old 07-08-2008, 05:49 PM   #6
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
What do your rotatelog entries look like in your CustomLog, or ErrorLog directives?

I don't use rotatelog myself.
 
Old 07-08-2008, 05:51 PM   #7
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
What do your rotatelog entries look like in your CustomLog, or ErrorLog directives?

I don't use rotatelog myself.
 
Old 07-09-2008, 05:54 AM   #8
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Hi Ya,

What we do since for ages due to another issue which is no longer a problem is have a different log file for every VH.

ErrorLog /usr/local/apache2/logs/error_log_webservname
CustomLog "|/usr/local/apache2/bin/rotatelogs /usr/local/apache2/logs/error_log_webservname 86400" common

....
what method do you use?
 
Old 07-09-2008, 07:26 AM   #9
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Getting closer now...
i did a :-
sed -i 's/[Ee]rror[Ll]og/#ErrorLog/g' * ; sed -i 's/[Cc]ustom[Ll]og/#CustomLog/g' *


That now leaves everything logging to the default apache settings, and this has enabled me to run all of the virtual hosts we have....this is fine but i'd like to know if it is possible to fix the problem and still log using the original method.
 
Old 07-09-2008, 12:15 PM   #10
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
I use logrotate every night or couple of nights as necessary. I see no reason to leave so many extra files open to tee log entries into other processes.

I find logrotate much faster, and it runs only at the time it is supposed to rotate a log. In your case, that might be very beneficial.
 
Old 07-09-2008, 12:55 PM   #11
kenoshi
Member
 
Registered: Sep 2007
Location: SF Bay Area, CA
Distribution: CentOS, SLES 10+, RHEL 3+, Debian Sarge
Posts: 159

Rep: Reputation: 32
I'd suggest you use logrotate like Mr C suggested...

In my shop I define separate logs for each VH, then logrotate every 24hrs...which makes things easier when pushing to a log analyzer like awstats.
 
Old 07-10-2008, 05:32 AM   #12
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Thanks for that guys, i will look into it...

Oddly enough though...everything is still running but :-

cat /proc/sys/fs/file-nr
2175 0 209710

Obviously better than the output before...but that still shows i've got 0 file des/open left???

lsof | wc -l
3867


I guess the only thing i can conclude from this is that the output of a lsof | wc -l needs to be less than the setting in /proc/sys/fs/file-max? Does that sound correct


Cheers
 
Old 07-10-2008, 12:10 PM   #13
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
Yes, but if you are constantly pushing that envelope, you have bigger concerns.

What you aren't potentially seeing is that your setup, for whatever reason, is repeatedly spawning rotatelog processes on itself, causing depletion of open file slots.
 
Old 07-10-2008, 12:12 PM   #14
kenoshi
Member
 
Registered: Sep 2007
Location: SF Bay Area, CA
Distribution: CentOS, SLES 10+, RHEL 3+, Debian Sarge
Posts: 159

Rep: Reputation: 32
helptonewbie,

The 2nd field if I remember is always 0 post 2.6.

Also, lsof lists OPEN files, which is NOT the same as allocated file descriptors...because for example, it will list things like memory mapped library files used by EACH process, when it should be accounted for just once. You can see this in action by monitoring say, vi:
Code:
[root@fenris]# lsof -p `ps -C vi -o pid=`
COMMAND  PID USER   FD   TYPE DEVICE     SIZE    NODE NAME
vi      2577 root  cwd    DIR  253,0     4096 2523137 /root
vi      2577 root  rtd    DIR  253,0     4096       2 /
vi      2577 root  txt    REG  253,0   594740  196654 /bin/vi
vi      2577 root  mem    REG  253,0    46740 1998889 /lib/libnss_files-2.5.so
vi      2577 root  mem    REG  253,0   125728 2002318 /lib/ld-2.5.so
vi      2577 root  mem    REG  253,0  1585788 2002319 /lib/libc-2.5.so
vi      2577 root  mem    REG  253,0    16528 2002320 /lib/libdl-2.5.so
vi      2577 root  mem    REG  253,0    25940 1999043 /lib/libacl.so.1.1.0
vi      2577 root  mem    REG  253,0   242880 2002334 /lib/libsepol.so.1
vi      2577 root  mem    REG  253,0    93508 2002335 /lib/libselinux.so.1
vi      2577 root  mem    REG  253,0    15780 1998936 /lib/libattr.so.1.1.0
vi      2577 root  mem    REG  253,0    13084 2002343 /lib/libtermcap.so.2.0.8
vi      2577 root  mem    REG  253,0 56413888 1675456 /usr/lib/locale/locale-archive
vi      2577 root    0u   CHR  136,2                4 /dev/pts/2
vi      2577 root    1u   CHR  136,2                4 /dev/pts/2
vi      2577 root    2u   CHR  136,2                4 /dev/pts/2
Now, check out the contents of /proc/2577/fd:
Code:
[root@fenris]# ls -la /proc/2577/fd
total 0
dr-x------ 2 root root  0 Jul  6 20:33 .
dr-xr-xr-x 5 root root  0 Jul  6 20:33 ..
lrwx------ 1 root root 64 Jul  6 20:33 0 -> /dev/pts/2
lrwx------ 1 root root 64 Jul  6 20:33 1 -> /dev/pts/2
lrwx------ 1 root root 64 Jul  6 20:33 2 -> /dev/pts/2
See the difference? Hope this helps.

Last edited by kenoshi; 07-10-2008 at 12:16 PM.
 
Old 07-10-2008, 01:07 PM   #15
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
Hmmmm... post 2.6 ? (you mean 2.7...?). I think you mean 2.6 and above.
Code:
$ uname -r
2.6.16.60

$ cat /proc/sys/fs/file-nr
189     0       25288

$ uname -r
2.4.20-30.9smp

$ cat /proc/sys/fs/file-nr
413     211     26174
Yes, there are can more open files by the kernel than open FDs. The kernel doesn't charge for all files it uses.

Last edited by Mr. C.; 07-10-2008 at 01:08 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Apache not serving PHP5 pages rfw62 Linux - Newbie 3 03-06-2008 07:55 PM
blank pages served from apache mjenkins Linux - Server 0 11-15-2006 04:24 PM
Apache not serving pages bcf2 Linux - Software 1 11-22-2005 03:46 AM
Apache displaying only blank pages? punker22 Linux - Software 4 05-03-2004 04:36 PM
Redhat 9, apache + frontpage not serving pages except admin page JRinOKC Linux - Newbie 5 01-12-2004 02:58 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 02:44 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration