Hi,
my webserver coredumps on high load, my guess is that the epoll.max_user_instances is too low, but I cannot increase it, can you please help..

root@www4:~# echo "fs.epoll.max_user_instances=4092" >> /etc/sysctl.conf
root@www4:~# sysctl -p
error: "fs.epoll.max_user_instances" is an unknown key


root@www4:~# uname -a
Linux www4 2.6.28-16-server #55-Ubuntu SMP Tue Oct 20 20:37:10 UTC 2009 x86_64 GNU/Linux


Thanks!!

No need to guess. Let's see some relevant sections from your error log.

# gdb /usr/local/apache2/bin/httpd core.25582
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...


warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /var/www/html/usr/local/apache2/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/local/apache2/lib/libaprutil-1.so.0
Reading symbols from /var/www/html/usr/local/apache2/lib/libexpat.so.0...done.
Loaded symbols for /usr/local/apache2/lib/libexpat.so.0
Reading symbols from /var/www/html/usr/local/apache2/lib/libapr-1.so.0...done.
Loaded symbols for /usr/local/apache2/lib/libapr-1.so.0
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /var/www/html/usr/local/apache2/modules/mod_proxy.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_proxy.so
Reading symbols from /var/www/html/usr/local/apache2/modules/mod_proxy_http.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_proxy_http.so
Reading symbols from /var/www/html/usr/local/apache2/modules/mod_ssl.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_ssl.so
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /var/www/html/usr/local/apache2/modules/mod_info.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_info.so
Reading symbols from /var/www/html/usr/local/apache2/modules/mod_chroot.so...done.
Loaded symbols for /usr/local/apache2/modules/mod_chroot.so
Reading symbols from /var/www/html/usr/local/apache2/modules/libphp5.so...done.
Loaded symbols for /usr/local/apache2/modules/libphp5.so
Reading symbols from /usr/lib64/mysql/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.15
Reading symbols from /usr/lib64/libmcrypt.so.4...done.
Loaded symbols for /usr/lib64/libmcrypt.so.4
Reading symbols from /usr/lib64/libltdl.so.3...done.
Loaded symbols for /usr/lib64/libltdl.so.3
Reading symbols from /usr/lib64/libfreetype.so.6...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libX11.so.6...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /usr/lib64/libXpm.so.4...done.
Loaded symbols for /usr/lib64/libXpm.so.4
Reading symbols from /usr/lib64/libpng12.so.0...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libjpeg.so.62...done.
Loaded symbols for /usr/lib64/libjpeg.so.62
Reading symbols from /usr/lib64/libcurl.so.3...done.
Loaded symbols for /usr/lib64/libcurl.so.3
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /usr/lib64/libxml2.so.2...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libidn.so.11...done.
Loaded symbols for /usr/lib64/libidn.so.11
Reading symbols from /usr/lib64/libXau.so.6...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
[New process 25582]
#0 apr_pollset_add (pollset=0x0, descriptor=0x7fff51ab15a0) at poll/unix/epoll.c:150
150 if (pollset->flags & APR_POLLSET_NOCOPY) {
(gdb)

That's a core dump analysis.

Instead, check your main Apache config (not sure what it's called in Ubuntu-land, but on most systems this is httpd.conf). Look for a directive like:
And another directive like:
Then combine the two, forming (in this case) /etc/httpd/logs/error_log.

We need to see snippets from that log -- hopefully that will provide clues. Please use code tags so it's more readable.

here are the error_logs

[Sun Oct 08 05:46:54 2009] [notice] child pid 31117 exit signal Segmentation fault (11)
[Sun Oct 08 05:47:07 2009] [notice] child pid 31255 exit signal Segmentation fault (11)
[Sun Oct 08 05:47:57 2009] [notice] child pid 30396 exit signal Segmentation fault (11)
[Sun Oct 08 05:47:59 2009] [notice] child pid 31512 exit signal Segmentation fault (11)
[Sun Oct 08 05:52:09 2009] [notice] child pid 31569 exit signal Segmentation fault (11)
[Sun Oct 08 05:52:20 2009] [notice] child pid 31550 exit signal Segmentation fault (11)
[Sun Oct 08 05:56:35 2009] [notice] child pid 31680 exit signal Segmentation fault (11)
[Sun Oct 08 05:57:38 2009] [notice] child pid 31748 exit signal Segmentation fault (11)
[Sun Oct 08 06:45:16 2009] [notice] child pid 31767 exit signal Segmentation fault (11)
[Sun Oct 08 07:36:24 2009] [notice] child pid 31699 exit signal Segmentation fault (11)
[Sun Oct 08 07:59:28 2009] [notice] child pid 31442 exit signal Segmentation fault (11)
[Sun Oct 08 08:00:35 2009] [notice] child pid 31493 exit signal Segmentation fault (11)
[Sun Oct 08 08:01:05 2009] [notice] child pid 31531 exit signal Segmentation fault (11)
[Sun Oct 08 08:08:57 2009] [notice] child pid 32644 exit signal Segmentation fault (11)
[Sun Oct 08 09:13:29 2009] [notice] child pid 1125 exit signal Segmentation fault (11)
[Sun Oct 08 09:36:59 2009] [notice] child pid 1851 exit signal Segmentation fault (11)

my apache version is 2.2.11


Thanks

I just upgraded to 2.2.14 but did not help....

[Mon Nov 09 00:09:16 2009] [notice] child pid 26092 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:16 2009] [notice] child pid 26170 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:25 2009] [notice] child pid 26089 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:30 2009] [notice] child pid 26102 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:34 2009] [notice] child pid 26088 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:34 2009] [notice] child pid 26128 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:34 2009] [notice] child pid 26278 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:35 2009] [notice] child pid 26169 exit signal Segmentation fault (11)
[Mon Nov 09 00:09:46 2009] [notice] child pid 26349 exit signal Segmentation fault (11)
[Mon Nov 09 00:10:01 2009] [notice] child pid 26369 exit signal Segmentation fault (11)
[Mon Nov 09 00:10:32 2009] [notice] child pid 26463 exit signal Segmentation fault (11)
[Mon Nov 09 00:10:36 2009] [notice] child pid 26283 exit signal Segmentation fault (11)
[Mon Nov 09 00:10:47 2009] [notice] child pid 26539 exit signal Segmentation fault (11)
[Mon Nov 09 00:10:48 2009] [notice] child pid 26167 exit signal Segmentation fault (11)
[Mon Nov 09 00:10:48 2009] [notice] child pid 26482 exit signal Segmentation fault (11)

What sort of web applications are you running?

Also, let's see the output of: # httpd -M

Maybe one of your modules is configured badly and/or misbehaving.

root@www4:~/httpd-2.2.14# httpd -M
Loaded Modules:
core_module (static)
authn_file_module (static)
authn_default_module (static)
authz_host_module (static)
authz_groupfile_module (static)
authz_user_module (static)
authz_default_module (static)
auth_basic_module (static)
include_module (static)
filter_module (static)
log_config_module (static)
env_module (static)
setenvif_module (static)
version_module (static)
mpm_worker_module (static)
http_module (static)
mime_module (static)
status_module (static)
autoindex_module (static)
asis_module (static)
cgid_module (static)
negotiation_module (static)
dir_module (static)
actions_module (static)
userdir_module (static)
alias_module (static)
so_module (static)
php5_module (shared)
Syntax OK

That's a lot of static modules...

So you're running a php webapp, or what? Any details on that?

yes, it's a PHP5 website.

Try a google query for: "mod_php exit signal Segmentation fault"

Not sure what else to suggest until you give a lot more details about your app and what it's doing.

Any system changes that coincide with the problem?

Did you upgrade PHP recently, do you run a PHP accelerator (like eaccelerator, xcache)...?

I'm runnning php 5.2.10 with eaccelerator