Slow Apache Web Server needs tuning
 

Hi,

I am having a major headache trying to get a webserver to serve dynamic pages quickly, I want to know if any one has any suggestions that I can look at. I'm ok on Linux but I am not a apache guru.

Apache root servers an index.php page that display a page and pulls some images and data from the mysql server to display, I have been informed there could be upto 10,000 connections to he site at peak times, this is when we experience issues.

The NIC is set to 10GB and Networks have told me they are allowing 1GB of traffic to the servers. the landing page is about 1.5mb in size.

I have noticed that during peak times the load on the server is minimum and idle is often around 80%, this leads me to believe my apache is a bottleneck and not allowing enough connections in.

I would like some advice on tuning the apache config below.
I know each httpd process consumes 20mb
Apache Conf:
I am also seeing errors in /var/log/message at peak times.

I am not quite sure where to start any help would be great.

If you want a quick win then use Firefox + firebug, run it on the web site and check the network loading times and the recommendations / comments tabs.

More in-depth: my first thought is we're missing quite a bit of information:
- what virtualization platform is in use?
- what virtualization guest storage is used (local, network, type of -)?
- what other services are running?
- what other supporting services are running or needed (NoSQL, caching etc, etc)?
- what software (+plugins themes, addons, custom code) is actually running in the web stack?
- where are the system utilization stats?
- where are the network utilization stats?
- what was tuned system-wise already (sysctls, Netfilter etc, etc)?
- what was tuned service-wise (httpd.conf, my.cnf, %{interpreter}.ini)?
- was the setup tested to allow n connections?
- what was changed network, virtualization platform, system, software or configuration-wise or did this setup always have problems?

You see with web applications "more speed" doesn't come from "just" tweaking httpd.conf but starts with a good understanding of the requirements and resources. Ideally you would start by building a baseline (stock configuration throughout), stress the setup, gather information to analyse, make changes, test again and make that your rinse-repeat cycle until done.

Hi unSpawn,

Thanks for your reply.

I should have mentioned that the web server is a clients that is hosted on our vcenter 5.1, we were asked to deploy a server running redhat 6.7 and install php, apache on one and mysql on another. The client built / deployed their web site and then complained it was not performing, we suggested their web dev should tune the server but unfortunately they only have windows skills and were forced to use the app by another web dev company, all got a bit silly but that's the gist of it.

I am not an apache expert however have been tasked with fixing the issue for the client, which after a week has brought me to the forums for some advice.

The current MPM prefork config was changed as below which certainly helped but has not resolved the issue and today during a peak time the server started to use swap and ended up almost crashing, I'm guessing the 5000 is too much even though the servers memory has been upped to 64GB now.

I have turned KeepAlive Off
And adjusted the settings below in httpd.conf I think I may have over cooked it?
I will try and answer the questions as best I can:

- what virtualization platform is in use? Vcenter 5.1 according the the about screen.

- what virtualization guest storage is used (local, network, type of -)? Network SAN storage

- what other services are running? Server is running httpd and the standard default services.

- what other supporting services are running or needed (NoSQL, caching etc, etc)? All configs were default, mysql on 2nd server, query's are being looked at.

- what software (+plugins themes, addons, custom code) is actually running in the web stack? custom web site, php based does not look like a complex site.

- where are the system utilization stats?
- where are the network utilization stats?
At about 13:00 today the network utilisation went from about 15Mbps to 140Mbps
At this time the web servers memory hit 60% and disk i/o hit 275Mbps

- what was tuned system-wise already (sysctls, Netfilter etc, etc)? All configs default except the change I have made to httpd.conf
- what was tuned service-wise (httpd.conf, my.cnf, %{interpreter}.ini)? As above

- was the setup tested to allow n connections? No but I wish it had been.

- what was changed network, virtualization platform, system, software or configuration-wise or did this setup always have problems? Web site is a new version of an existing site, client migrated from a windows server with an almost static site to a php, media looking site, has a little text and several images these are rotated from the database it does not look overly complex or particularly flashy, way simpler than something like wordpress or drupal.

First of all where you go from here depends on what services your company offers. Even if you are not an application level Guru you at least should be able to pinpoint potential OS / application bottlenecks to your client and tell them it's their responsibility to have the "web dev company" (that forced them to use the app, as you say) tune the application or at least provide useful suggestions. (Come to think of it, it could just be you're a new to the whole game and this is the companies "challenge" to judge you by. If that's the case then they have an odd sense of humour having you play with a customers production platform...)


Secondly here's your problem: you keep finicking over Prefork MPM config values instead of looking at the platform as a whole. How can you ever hope to "magically" tune the whole platform while not tuning the other configuration items?!! And what calculation did you base your Prefork MPM config changes on in the first place?..


First of all I suggest you meditate on some aspects:
OS performance tuning:
https://access.redhat.com/documentat...ide/index.html or CentOS documentation equivalent,

Apache:
http://httpd.apache.org/docs/2.4/en/...rf-tuning.html (or the 2.2 version if that's what you're using),
https://www.howtoforge.com/configuri...um_performance,
*research the performance / thread-safe differences between using Nginx or Apache + FastCGI or Apache + mod-php,

MySQL:
http://dev.mysql.com/doc/refman/5.5/en/mysqlcheck.html
https://www.howtoforge.com/tuning-my...ith-mysqltuner
https://www.percona.com/blog/2014/01...-installation/
http://www.askapache.com/mysql/perfo...ing-mysql.html
http://www.mysql.com/why-mysql/performance/
https://dev.mysql.com/doc/refman/5.1...imization.html


After you've read the above get all the unique (non-personalized) requests from the Apache access log and save them to a file. Install apachetop, mysqltop and atop (interval 10 to 30 secs) and run them in terminal windows together with a tail on the Apache access and error logs. Now use 'siege' or 'ab' or any other tool you fancy to fire off concurrent requests when the web site is not busy. Gather all data and logs for the test period and store them safely as this will be your baseline. I trust you will record all your configuration changes in version control to facilitate easy comparison / roll back.


- Tune the OS. Ununstall unnecessary packages. Make /tmp a tmpfs. Ensure docroot, database directory and /var/log do not reside in one partition. Take particularly care of virtual memory management, network port range, keepalives and timeout sysctls. Don't tune network interface memory settings.
- Perform Apache access log reporting. Get the maximum requests per second (RPS) you need to handle (else calculate it yourself) and the resources requested the most (and if they are static items or not), also check the error log.
- Divide the total amount of RAM by how many memory Apache uses per child (see 'ps' the "RES" field in kb). Subtract the amount of RAM the system itself needs for caching (1 GB would be nice) and this then will be the MaxClients and ServerLimit value.
- Set max_connections in my.cnf to the MaxClients value (or change according to mysqltuner output).
- Set StartServers and MinSpareServers to the default (keep at most 20 to 25 MinSpareServers).
- new children are spawned at a rate of max. 32 per sec and MaxSpareServers cost RAM while idling. Initially set to 4 and tune up only when the error log says so.
- MaxRequestsPerChild is a balance as spawning children is costly and blocking to boot but on the other hand it also allows you can control the effects of memory corruption. So this depends on if you're serving complex scripts or much static (cachable) content. Set to 500 initially.
- Run mysqltuner and follow recommendations.


Fire off another test and compare results with baseline. Do more reporting, check logs, run analysis tools again.

*If you want you can ask more detailed questions. Be sure to include configuration files (as in 'egrep -v "^(;|#|//)" configfile|grep .;'), (error) log excerpts, other details, etc, etc.
**And please use the "Quote" button when replying or add the [QUOTE]text here[/QUOTE] vBB tags yourself.