LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 12-18-2015, 05:05 AM   #1
pobrika
Member
 
Registered: Jan 2008
Location: Bournemouth, UK
Distribution: Mint, #!, Fedora, Redhat, Centos
Posts: 69

Rep: Reputation: 18
Slow Apache Web Server needs tuning


Hi,

I am having a major headache trying to get a webserver to serve dynamic pages quickly, I want to know if any one has any suggestions that I can look at. I'm ok on Linux but I am not a apache guru.

Quote:
Web Server Specs:
Redhat 6.7
Virtual Server - vcenter
12 CPUs
32GB Memory
httpd-2.2.15-47.el6
Quote:
Database Server Specs:
Redhat:6.7
Virtual Server -Vcenter
8 CPus
16GB Memory
Apache root servers an index.php page that display a page and pulls some images and data from the mysql server to display, I have been informed there could be upto 10,000 connections to he site at peak times, this is when we experience issues.

The NIC is set to 10GB and Networks have told me they are allowing 1GB of traffic to the servers. the landing page is about 1.5mb in size.

I have noticed that during peak times the load on the server is minimum and idle is often around 80%, this leads me to believe my apache is a bottleneck and not allowing enough connections in.

I would like some advice on tuning the apache config below.
I know each httpd process consumes 20mb
Apache Conf:
Code:
StartServers       32
MinSpareServers    32
MaxSpareServers   128
ServerLimit      1000    
MaxClients       1000
MaxRequestsPerChild 20000
I am also seeing errors in /var/log/message at peak times.

Code:
Dec 17 20:07:55 49-web-001 kernel: possible SYN flooding on port 80. Sending cookies.
Dec 17 20:13:02 49-web-001 kernel: possible SYN flooding on port 80. Sending cookies.
Dec 17 20:15:54 49-web-001 kernel: possible SYN flooding on port 80. Sending cookies.
Dec 17 20:18:20 49-web-001 kernel: possible SYN flooding on port 80. Sending cookies.
I am not quite sure where to start any help would be great.
 
Old 12-19-2015, 05:43 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,394
Blog Entries: 55

Rep: Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565
Quote:
Originally Posted by pobrika View Post
I am not quite sure where to start any help would be great.
If you want a quick win then use Firefox + firebug, run it on the web site and check the network loading times and the recommendations / comments tabs.

More in-depth: my first thought is we're missing quite a bit of information:
- what virtualization platform is in use?
- what virtualization guest storage is used (local, network, type of -)?
- what other services are running?
- what other supporting services are running or needed (NoSQL, caching etc, etc)?
- what software (+plugins themes, addons, custom code) is actually running in the web stack?
- where are the system utilization stats?
- where are the network utilization stats?
- what was tuned system-wise already (sysctls, Netfilter etc, etc)?
- what was tuned service-wise (httpd.conf, my.cnf, %{interpreter}.ini)?
- was the setup tested to allow n connections?
- what was changed network, virtualization platform, system, software or configuration-wise or did this setup always have problems?

You see with web applications "more speed" doesn't come from "just" tweaking httpd.conf but starts with a good understanding of the requirements and resources. Ideally you would start by building a baseline (stock configuration throughout), stress the setup, gather information to analyse, make changes, test again and make that your rinse-repeat cycle until done.
 
Old 12-19-2015, 03:31 PM   #3
pobrika
Member
 
Registered: Jan 2008
Location: Bournemouth, UK
Distribution: Mint, #!, Fedora, Redhat, Centos
Posts: 69

Original Poster
Rep: Reputation: 18
Hi unSpawn,

Thanks for your reply.

I should have mentioned that the web server is a clients that is hosted on our vcenter 5.1, we were asked to deploy a server running redhat 6.7 and install php, apache on one and mysql on another. The client built / deployed their web site and then complained it was not performing, we suggested their web dev should tune the server but unfortunately they only have windows skills and were forced to use the app by another web dev company, all got a bit silly but that's the gist of it.

I am not an apache expert however have been tasked with fixing the issue for the client, which after a week has brought me to the forums for some advice.

The current MPM prefork config was changed as below which certainly helped but has not resolved the issue and today during a peak time the server started to use swap and ended up almost crashing, I'm guessing the 5000 is too much even though the servers memory has been upped to 64GB now.

I have turned KeepAlive Off
And adjusted the settings below in httpd.conf I think I may have over cooked it?
Code:
StartServers       32
MinSpareServers    32
MaxSpareServers   128
ServerLimit      5000
MaxClients       5000
MaxRequestsPerChild 20000
I will try and answer the questions as best I can:

- what virtualization platform is in use? Vcenter 5.1 according the the about screen.

- what virtualization guest storage is used (local, network, type of -)? Network SAN storage

- what other services are running? Server is running httpd and the standard default services.

- what other supporting services are running or needed (NoSQL, caching etc, etc)? All configs were default, mysql on 2nd server, query's are being looked at.

- what software (+plugins themes, addons, custom code) is actually running in the web stack? custom web site, php based does not look like a complex site.

- where are the system utilization stats?
Code:
                CPU     user     nice   system   %iowait    %steal     %idle
12:50:01 PM     all      6.05      0.00      0.83      0.00      0.00     93.11
01:00:01 PM     all     23.79      0.00      4.11      0.00      0.00     72.10
01:10:02 PM     all     66.16      0.00     14.58      0.00      0.00     19.25
01:20:01 PM     all     55.64      0.00     15.61      0.00      0.00     28.74
01:30:01 PM     all      0.17      0.00      0.28      0.00      0.00     99.54
- where are the network utilization stats?
At about 13:00 today the network utilisation went from about 15Mbps to 140Mbps
At this time the web servers memory hit 60% and disk i/o hit 275Mbps

- what was tuned system-wise already (sysctls, Netfilter etc, etc)? All configs default except the change I have made to httpd.conf
Code:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.netfilter.nf_conntrack_max = 1048576
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
- what was tuned service-wise (httpd.conf, my.cnf, %{interpreter}.ini)? As above

- was the setup tested to allow n connections? No but I wish it had been.

- what was changed network, virtualization platform, system, software or configuration-wise or did this setup always have problems? Web site is a new version of an existing site, client migrated from a windows server with an almost static site to a php, media looking site, has a little text and several images these are rotated from the database it does not look overly complex or particularly flashy, way simpler than something like wordpress or drupal.
 
Old 12-20-2015, 11:19 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,394
Blog Entries: 55

Rep: Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565
Quote:
Originally Posted by pobrika View Post
(..) the web server is a clients (..). The client built / deployed their web site and then complained it was not performing, we suggested their web dev should tune the server but unfortunately they (..) were forced to use the app by another web dev company (..) I (..) have been tasked with fixing the issue for the client, (..) Web site is a new version of an existing site, client migrated from a windows server with an almost static site to a php, media looking site, has a little text and several images these are rotated from the database it does not look overly complex or particularly flashy, way simpler than something like wordpress or drupal.
First of all where you go from here depends on what services your company offers. Even if you are not an application level Guru you at least should be able to pinpoint potential OS / application bottlenecks to your client and tell them it's their responsibility to have the "web dev company" (that forced them to use the app, as you say) tune the application or at least provide useful suggestions. (Come to think of it, it could just be you're a new to the whole game and this is the companies "challenge" to judge you by. If that's the case then they have an odd sense of humour having you play with a customers production platform...)


Quote:
Originally Posted by pobrika View Post
Quote:
Originally Posted by unSpawn
what was tuned system-wise already (sysctls, Netfilter etc, etc)?
All configs default except the change I have made to httpd.conf (..) The current MPM prefork config was changed as below which certainly helped but has not resolved the issue
Secondly here's your problem: you keep finicking over Prefork MPM config values instead of looking at the platform as a whole. How can you ever hope to "magically" tune the whole platform while not tuning the other configuration items?!! And what calculation did you base your Prefork MPM config changes on in the first place?..


First of all I suggest you meditate on some aspects:
OS performance tuning:
https://access.redhat.com/documentat...ide/index.html or CentOS documentation equivalent,

Apache:
http://httpd.apache.org/docs/2.4/en/...rf-tuning.html (or the 2.2 version if that's what you're using),
https://www.howtoforge.com/configuri...um_performance,
*research the performance / thread-safe differences between using Nginx or Apache + FastCGI or Apache + mod-php,

MySQL:
http://dev.mysql.com/doc/refman/5.5/en/mysqlcheck.html
https://www.howtoforge.com/tuning-my...ith-mysqltuner
https://www.percona.com/blog/2014/01...-installation/
http://www.askapache.com/mysql/perfo...ing-mysql.html
http://www.mysql.com/why-mysql/performance/
https://dev.mysql.com/doc/refman/5.1...imization.html


After you've read the above get all the unique (non-personalized) requests from the Apache access log and save them to a file. Install apachetop, mysqltop and atop (interval 10 to 30 secs) and run them in terminal windows together with a tail on the Apache access and error logs. Now use 'siege' or 'ab' or any other tool you fancy to fire off concurrent requests when the web site is not busy. Gather all data and logs for the test period and store them safely as this will be your baseline. I trust you will record all your configuration changes in version control to facilitate easy comparison / roll back.


- Tune the OS. Ununstall unnecessary packages. Make /tmp a tmpfs. Ensure docroot, database directory and /var/log do not reside in one partition. Take particularly care of virtual memory management, network port range, keepalives and timeout sysctls. Don't tune network interface memory settings.
- Perform Apache access log reporting. Get the maximum requests per second (RPS) you need to handle (else calculate it yourself) and the resources requested the most (and if they are static items or not), also check the error log.
- Divide the total amount of RAM by how many memory Apache uses per child (see 'ps' the "RES" field in kb). Subtract the amount of RAM the system itself needs for caching (1 GB would be nice) and this then will be the MaxClients and ServerLimit value.
- Set max_connections in my.cnf to the MaxClients value (or change according to mysqltuner output).
- Set StartServers and MinSpareServers to the default (keep at most 20 to 25 MinSpareServers).
- new children are spawned at a rate of max. 32 per sec and MaxSpareServers cost RAM while idling. Initially set to 4 and tune up only when the error log says so.
- MaxRequestsPerChild is a balance as spawning children is costly and blocking to boot but on the other hand it also allows you can control the effects of memory corruption. So this depends on if you're serving complex scripts or much static (cachable) content. Set to 500 initially.
- Run mysqltuner and follow recommendations.


Fire off another test and compare results with baseline. Do more reporting, check logs, run analysis tools again.

*If you want you can ask more detailed questions. Be sure to include configuration files (as in 'egrep -v "^(;|#|//)" configfile|grep .;'), (error) log excerpts, other details, etc, etc.
**And please use the "Quote" button when replying or add the [QUOTE]text here[/QUOTE] vBB tags yourself.
 
  


Reply

Tags
apache, centos6


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Web sites loading slow on Apache-MySql gezginrocker Linux - Server 7 09-02-2012 03:44 PM
Apache Server Updates my web site very slow. Raheel Hassan Linux - Server 5 01-21-2012 12:30 PM
Server Slow : Apache Mysql fine tuning zahirw Linux - Server 2 04-22-2011 02:26 PM
Apache Server maxclient/serverlimit perf Tuning helptonewbie Linux - Server 3 04-01-2010 09:54 AM
LXer: Performance Tuning for a Linux Web Server: Parts 1-5 LXer Syndicated Linux News 0 02-19-2009 07:30 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 12:53 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration