Hi unSpawn,
Thanks for your reply.
I should have mentioned that the web server is a clients that is hosted on our vcenter 5.1, we were asked to deploy a server running redhat 6.7 and install php, apache on one and mysql on another. The client built / deployed their web site and then complained it was not performing, we suggested their web dev should tune the server but unfortunately they only have windows skills and were forced to use the app by another web dev company, all got a bit silly but that's the gist of it.
I am not an apache expert however have been tasked with fixing the issue for the client, which after a week has brought me to the forums for some advice.
The current MPM prefork config was changed as below which certainly helped but has not resolved the issue and today during a peak time the server started to use swap and ended up almost crashing, I'm guessing the 5000 is too much even though the servers memory has been upped to 64GB now.
I have turned KeepAlive Off
And adjusted the settings below in httpd.conf I think I may have over cooked it?
Code:
StartServers 32
MinSpareServers 32
MaxSpareServers 128
ServerLimit 5000
MaxClients 5000
MaxRequestsPerChild 20000
I will try and answer the questions as best I can:
- what virtualization platform is in use? Vcenter 5.1 according the the about screen.
- what virtualization guest storage is used (local, network, type of -)? Network SAN storage
- what other services are running? Server is running httpd and the standard default services.
- what other supporting services are running or needed (NoSQL, caching etc, etc)? All configs were default, mysql on 2nd server, query's are being looked at.
- what software (+plugins themes, addons, custom code) is actually running in the web stack? custom web site, php based does not look like a complex site.
- where are the system utilization stats?
Code:
CPU user nice system %iowait %steal %idle
12:50:01 PM all 6.05 0.00 0.83 0.00 0.00 93.11
01:00:01 PM all 23.79 0.00 4.11 0.00 0.00 72.10
01:10:02 PM all 66.16 0.00 14.58 0.00 0.00 19.25
01:20:01 PM all 55.64 0.00 15.61 0.00 0.00 28.74
01:30:01 PM all 0.17 0.00 0.28 0.00 0.00 99.54
- where are the network utilization stats?
At about 13:00 today the network utilisation went from about 15Mbps to 140Mbps
At this time the web servers memory hit 60% and disk i/o hit 275Mbps
- what was tuned system-wise already (sysctls, Netfilter etc, etc)? All configs default except the change I have made to httpd.conf
Code:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.netfilter.nf_conntrack_max = 1048576
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
- what was tuned service-wise (httpd.conf, my.cnf, %{interpreter}.ini)? As above
- was the setup tested to allow n connections? No but I wish it had been.
- what was changed network, virtualization platform, system, software or configuration-wise or did this setup always have problems? Web site is a new version of an existing site, client migrated from a windows server with an almost static site to a php, media looking site, has a little text and several images these are rotated from the database it does not look overly complex or particularly flashy, way simpler than something like wordpress or drupal.