pablo1999

[Log in to get rid of this advertisement]

I recently had an issue with a server that was responding very slowly to https(443) requests. The site is only supposed to accept https requests.
I tried to access the server via ssh but ssh was also responding slowly. I decided to access directly via the console and noticed that it was also responding slowly when I logged in. I ran top and found that most of the resources were consumed by the HTTP service.

Here I have pasted the ssl_error_log, messages logs and
httpd configuration. I'm not sure what exactly caused the problem. Don't know if its the way http is configured or if it was something else.

Any inputs?





ssl_error_log:
=============

[Fri Nov 06 16:52:02 2009] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Nov 06 16:52:02 2009] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.1.211:8029 (192.168.1.211)
[Fri Nov 06 16:52:48 2009] [error] [client 98.191.75.232] proxy: error processing end
[Fri Nov 06 16:54:52 2009] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Nov 06 16:54:52 2009] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.1.211:8029 (192.168.1.211)
[Fri Nov 06 16:55:13 2009] [error] ajp_read_header: ajp_ilink_receive failed



messages logs:
==============

Nov 6 16:53:12 server kernel: type=1400 audit(1257544392.290:35): avc: denied { name_connect } for pid=19047 comm="httpd" dest=8029 scontext=system_u:system_r:httpd_t:s0 tcontext=system_ubject_rort_t:s0 tclass=tcp_socket

Nov 6 16:59:04 server kernel: type=1400 audit(1257544744.195:36): avc: denied { name_connect } for pid=19297 comm="httpd" dest=8029 scontext=user_u:system_r:httpd_t:s0 tcontext=system_ubject_rort_t:s0 tclass=tcp_socket



httpd configuration:
===================

ServerTokens Major
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
Listen 80
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule version_module modules/mod_version.so
Include conf.d/*.conf
User apache
Group apache
ServerAdmin root@localhost
ServerName server.domain.com
UseCanonicalName Off
DocumentRoot "/var/www/html"
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
<Directory "/var/www/html">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
<IfModule mod_userdir.c>
UserDir disable
#UserDir public_html
</IfModule>
DirectoryIndex index.html index.html.var
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>
TypesConfig /etc/mime.types
DefaultType text/plain
<IfModule mod_mime_magic.c>
MIMEMagicFile conf/magic
</IfModule>
HostnameLookups Off
ErrorLog logs/error_log
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog logs/access_log combined
ServerSignature On
Alias /icons/ "/var/www/icons/"
<Directory "/var/www/icons">
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>
<IfModule mod_dav_fs.c>
DAVLockDB /var/lib/dav/lockdb
</IfModule>
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
IndexOptions FancyIndexing VersionSort NameWidth=*
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^
DefaultIcon /icons/unknown.gif
ReadmeName README.html
HeaderName HEADER.html
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW
ForceLanguagePriority Prefer Fallback
AddDefaultCharset UTF-8
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddHandler imap-file map
AddHandler type-map var
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
Alias /error/ "/var/www/error/"
<IfModule mod_negotiation.c>
<IfModule mod_include.c>
<Directory "/var/www/error">
AllowOverride None
Options IncludesNoExec
AddOutputFilter Includes html
AddHandler type-map var
Order allow,deny
Allow from all
LanguagePriority en es de fr
ForceLanguagePriority Prefer Fallback
</Directory>
</IfModule>
</IfModule>
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully

jstephens84

post your output from top and free -m. Just for a baseline of information.

chiragrk

From the logs it appears that you have SELinux enabled. Do you have it in permissive or enforced mode (command `sestatus`)?
You mentioned that https site is listening on port 443 but are the pages/application in your site referring to some other application on port 8029? I ask this cause your logs show failure messages when trying to connect to 192.168.1.211:8029 , which apparently is due to SELinux.

For starts maybe you should put SELinux in permissive mode if it isn't already (echo 0 > /selinux/enforce) and see if that helps.

jstephens84

SELinux is just an ACL program. It should'nt be the problem here as it is really only an acl based system. In the OPs case he can still get to SSL. He also said that even logging in the console screen was slow.

Now I could be wrong but it has been my experience that I have never seen it to cause performance issues. Only blockage of application.

I would check for any running scripts that should be there, check everyone's cronjobs, and look for any flashing lights on the server as an indicator of a bad drive.

pablo1999

The application that is being accessed thru https uses tomcat. At the moment the server is still responding properly to https requests. I can see how the java percentage process has increased during the day from 12% to 47.2% Also the number of httpd process have increased.

Is there any speciall tweak that needs to be performed on tomcat to make sure it will not eventully consume all the system resources?

Here is the top information:



top - 15:41:05 up 2 days, 22:34, 1 user, load average: 0.05, 0.06, 0.01
Tasks: 144 total, 1 running, 143 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 0.5%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 251968k total, 247972k used, 3996k free, 1320k buffers
Swap: 16777208k total, 297576k used, 16479632k free, 19540k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18763 root 15 0 12740 1144 832 R 0.0 0.5 0:00.05 top
18733 root 15 0 66212 1624 1204 S 0.0 0.6 0:00.02 bash
18732 root 18 0 100m 1392 1092 S 0.0 0.6 0:00.01 su
18710 rcrane 16 0 66100 1532 1184 S 0.0 0.6 0:00.02 bash
18709 rcrane 15 0 90900 1816 1072 S 0.0 0.7 0:00.01 sshd
18707 root 16 0 90900 3228 2500 S 0.0 1.3 0:00.09 sshd
18481 apache 15 0 289m 4004 2088 S 0.0 1.6 0:00.12 httpd
18480 apache 15 0 289m 2696 1584 S 0.0 1.1 0:00.12 httpd
18475 apache 15 0 289m 3656 1912 S 0.0 1.5 0:00.11 httpd
18474 apache 15 0 289m 2668 1612 S 0.0 1.1 0:00.10 httpd
18473 apache 15 0 289m 3992 2064 S 0.0 1.6 0:00.12 httpd
18472 apache 15 0 289m 2496 1492 S 0.0 1.0 0:00.09 httpd
18467 apache 15 0 289m 3648 1904 S 0.0 1.4 0:00.11 httpd
18389 apache 15 0 289m 3484 1860 S 0.0 1.4 0:00.21 httpd
18387 apache 15 0 289m 3472 1876 S 0.0 1.4 0:00.26 httpd
17935 apache 15 0 289m 2504 1520 S 0.0 1.0 0:00.37 httpd
17920 root 18 0 3792 156 156 S 0.0 0.1 0:00.00 mingetty
17869 apache 15 0 289m 3476 1848 S 0.0 1.4 0:00.41 httpd
17868 apache 15 0 289m 2924 1612 S 0.0 1.2 0:00.47 httpd
17867 apache 16 0 289m 2728 1524 S 0.0 1.1 0:00.44 httpd
17866 apache 15 0 289m 2508 1508 S 0.0 1.0 0:00.52 httpd
17865 apache 15 0 289m 3848 1968 S 0.0 1.5 0:00.35 httpd
17864 apache 15 0 289m 2512 1508 S 0.0 1.0 0:00.43 httpd
17863 apache 15 0 289m 2980 1624 S 0.0 1.2 0:00.45 httpd
17862 apache 15 0 289m 2724 1524 S 0.0 1.1 0:00.49 httpd
5201 root 18 0 1448m 116m 4284 S 0.0 47.2 1:05.15 java
5071 root 18 0 3792 124 124 S 0.0 0.0 0:00.00 mingetty
5070 root 21 0 3792 124 124 S 0.0 0.0 0:00.00 mingetty
5069 root 18 0 3792 124 124 S 0.0 0.0 0:00.00 mingetty
5068 root 18 0 3792 124 124 S 0.0 0.0 0:00.00 mingetty
5067 root 18 0 3792 124 124 S 0.0 0.0 0:00.00 mingetty
5062 root 18 0 18416 168 152 S 0.0 0.1 0:00.00 smartd
4995 haldaemo 17 0 12324 188 188 S 0.0 0.1 0:00.00 hald-addon-keyb
4970 haldaemo 15 0 12324 184 184 S 0.0 0.1 0:00.00 hald-addon-acpi
4963 root 15 0 21692 520 416 S 0.0 0.2 0:00.01 hald-runner
4962 haldaemo 15 0 31228 1404 856 S 0.0 0.6 0:01.69 hald
4944 avahi 22 0 23148 80 80 S 0.0 0.0 0:00.00 avahi-daemon
4943 avahi 15 0 23272 556 416 S 0.0 0.2 0:00.04 avahi-daemon
4912 root 15 0 58924 136 136 S 0.0 0.1 0:00.00 rhnsd
4888 root 18 0 215m 31m 3832 S 0.0 12.9 0:51.48 dsm_om_connsvc3
4887 root 22 0 5776 72 68 S 0.0 0.0 0:00.00 dsm_om_connsvc3
4848 root 16 0 10096 1564 1160 S 0.0 0.6 0:23.50 dsm_sa_snmp32d
4836 root 18 0 7968 652 528 S 0.0 0.3 0:01.69 dsm_sa_eventmgr
3851 root 16 0 43864 1688 1096 S 0.0 0.7 0:52.19 dsm_sa_datamgr3
3779 root 15 0 155m 3216 1076 S 0.0 1.3 0:19.45 snmpd
3740 root 16 0 36812 132 112 S 0.0 0.1 0:00.00 sfcbd
3719 root 18 0 26548 88 88 S 0.0 0.0 0:00.00 sfcbd
3679 root 17 0 16264 84 84 S 0.0 0.0 0:00.00 sfcbd
3488 root 34 19 0 0 0 S 0.0 0.0 13:54.94 kipmi0
3408 root 18 0 16288 84 84 S 0.0 0.0 0:00.00 sfcbd
3257 root 15 0 107m 528 340 S 0.0 0.2 0:00.07 dsm_om_shrsvc32
3229 root 18 0 18740 216 172 S 0.0 0.1 0:00.00 atd
3212 root 25 0 16392 88 88 S 0.0 0.0 0:00.00 sfcbd
3211 root 19 0 16252 64 64 S 0.0 0.0 0:00.00 sfcbd
3209 root 19 0 16252 76 76 S 0.0 0.0 0:00.00 sfcbd

wfh

Could you have a memory leak in java code?

More or less httpd processes is not an issue, unless they keep growing steadily and never go away. Apache keeps about 25 instances of httpd going on one of my apache servers.

What's going on with dsm? That looks high. Anyone know?

pablo1999

The only tweak that was done to the Java catalina.sh was the following:

JAVA_OPTS="-Xmx1250M -Xms750M -verbosegc"
export JAVA_OPTS

does this java setting looks ok?

Not sure what is going on with dsm. I checked the log dsm_om_connsvc32dIO.log and could not find any clues.

Any other ides on where can I look whats going on with dsm?

chiragrk

Two observations:

- You've set your JVM to start with memory 750M and max as 1250M
- From top I can see your system has physical memory of ~256MB and Swap of ~16GB

Somehow it doesn't look like things have been sized properly - you're bound to see performance issues.
Here's a few things you can do:
- Find the foot print of your tomcat application, find out the max/avg no.of users that will be using the tomcat application and arrive at a total memory requirement. If it isn't more than 256 MB, then you need not increase RAM on your machine.
- Irrespective of whether or not you increase your RAM, swap needs to be sized at par with the RAM. Typically for less than 1024MB of RAM you would size swap to 1.5-2 times the RAM.
- Set the -Xmx -Xms to something realistic w.r.t to RAM on the system. I don't see an advantage of using swap here.