Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Hi All,
I've got some sort of apache problem, what has happened recently is that apache has just stopped serving any files or blank pages. No error messages are outputted into any of the logs and apache appears to start-up fine...again no errors on debugging start-up.
The problem started after adding some more virtualhost files, now i know for sure its not the virtual host (VH) files themselves as if i delete some random files and then re-start apache...it runs fine again. EG: delete a few VH files then apache works, it has no baring on which ones i delete wheather they are recent ones i added or old ones that have been working for ages.
This lead me to beleive it was a file descriptor/open files problem, i've increased the limit in size by 8 times. This make no difference restore the VHs directory and apache restart the same happens again. Which leads me to beleive its not an open file problem as increasing it by 8 times would easily be enough for adding only 5 extra VH files.
Increased the openfile limit with:-
ulimit -H -n [number of files]
ulimit -S -n [number of files]
and edited the /etc/sys[something] making the hard and soft limits be set how i want after re-boot. And i've since rebooted and the limits changed to what i had set.
Could it be to many opened pipes?? there are alot of VHs i donno about 270ish, but other servers are all working fine.
lsof -p [httpd PID] | wc -l
reports about 1042 open files for each apache processes and apache on start-up in only spawning 6 times. Same as before.
Anyideas why would adding a few VHs stop apache working properly in any way.
Thanks for replying, I had looked at file-max but not file-nr.
Outputs of those files are:-
cat /proc/sys/fs/file-max
209710
cat /proc/sys/fs/file-nr
23040 0 209710
Not liking the fact this middle number is '0', ive looked up what this means and that means there's nothing free left??? is that correct....is this the problem right here?
lsof | wc -l
464054
This is something i found interesting....
lsof | grep rotatelog | wc -l
457044
The server has got worse now...all i've changed is as i've mentioned above.....the ulimit for openfiles. Yes i've tried
/usr/local/apache2/bin/apachectl -e debug -k start
[Tue Jul 08 11:01:35 2008] [debug] mod_so.c(290): loaded module php4_module
and thats all the output i get, i also specified in the httpd.conf file itself that i want the loglevel to be debug.
Now the server still isn't producing any pages.
The way i know that no log files have been written to....
touch /time
find / -newer /time | grep -v '/proc'
In-between the touch and the find...i'm trying to get a web page...i get results in the find but nothing for apache logs.
Thanks for your help on this matter
Last edited by helptonewbie; 07-08-2008 at 05:14 AM.
Yes, i did notice this....the other thing that was strange that stopping the apache service on the box didn't stop/kill all of the rotatelog processes. Then starting apache again started more rotatelog processes than before. Keep going and you'd get a steady increase. Any ideas why that might be. Restarting the server as expected allows apache to serve pages again but i'd cut down massively on the amount of VHs apache was to use. Any ideas on what may cause such a rediculous use of rotatelog?
Getting closer now...
i did a :-
sed -i 's/[Ee]rror[Ll]og/#ErrorLog/g' * ; sed -i 's/[Cc]ustom[Ll]og/#CustomLog/g' *
That now leaves everything logging to the default apache settings, and this has enabled me to run all of the virtual hosts we have....this is fine but i'd like to know if it is possible to fix the problem and still log using the original method.
I use logrotate every night or couple of nights as necessary. I see no reason to leave so many extra files open to tee log entries into other processes.
I find logrotate much faster, and it runs only at the time it is supposed to rotate a log. In your case, that might be very beneficial.
Oddly enough though...everything is still running but :-
cat /proc/sys/fs/file-nr
2175 0 209710
Obviously better than the output before...but that still shows i've got 0 file des/open left???
lsof | wc -l
3867
I guess the only thing i can conclude from this is that the output of a lsof | wc -l needs to be less than the setting in /proc/sys/fs/file-max? Does that sound correct
Yes, but if you are constantly pushing that envelope, you have bigger concerns.
What you aren't potentially seeing is that your setup, for whatever reason, is repeatedly spawning rotatelog processes on itself, causing depletion of open file slots.
Also, lsof lists OPEN files, which is NOT the same as allocated file descriptors...because for example, it will list things like memory mapped library files used by EACH process, when it should be accounted for just once. You can see this in action by monitoring say, vi:
Code:
[root@fenris]# lsof -p `ps -C vi -o pid=`
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
vi 2577 root cwd DIR 253,0 4096 2523137 /root
vi 2577 root rtd DIR 253,0 4096 2 /
vi 2577 root txt REG 253,0 594740 196654 /bin/vi
vi 2577 root mem REG 253,0 46740 1998889 /lib/libnss_files-2.5.so
vi 2577 root mem REG 253,0 125728 2002318 /lib/ld-2.5.so
vi 2577 root mem REG 253,0 1585788 2002319 /lib/libc-2.5.so
vi 2577 root mem REG 253,0 16528 2002320 /lib/libdl-2.5.so
vi 2577 root mem REG 253,0 25940 1999043 /lib/libacl.so.1.1.0
vi 2577 root mem REG 253,0 242880 2002334 /lib/libsepol.so.1
vi 2577 root mem REG 253,0 93508 2002335 /lib/libselinux.so.1
vi 2577 root mem REG 253,0 15780 1998936 /lib/libattr.so.1.1.0
vi 2577 root mem REG 253,0 13084 2002343 /lib/libtermcap.so.2.0.8
vi 2577 root mem REG 253,0 56413888 1675456 /usr/lib/locale/locale-archive
vi 2577 root 0u CHR 136,2 4 /dev/pts/2
vi 2577 root 1u CHR 136,2 4 /dev/pts/2
vi 2577 root 2u CHR 136,2 4 /dev/pts/2
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.