LinuxQuestions.org
Review your favorite Linux distribution.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 02-03-2010, 04:10 PM   #1
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Rep: Reputation: 30
apache is going really slow (and restarting it is the only thing that helps)


hey all i've been having some problems the last few months with my server/apache getting really slow (pages loading slow) and the only way i can fix it is to restart apache which is quite annoying and is not fixing the problem but it gets to the point where pages are almost not loading so i have to keep restarting apache but after 2mins it'll be slow again and once again i'll have to restart apache

and here's my apache2.conf

Code:
#
# Based upon the NCSA server configuration files originally by Rob McCool.
#
# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.2/ for detailed information about
# the directives.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# The configuration directives are grouped into three basic sections:
#  1. Directives that control the operation of the Apache server process as a
#     whole (the 'global environment').
#  2. Directives that define the parameters of the 'main' or 'default' server,
#     which responds to requests that aren't handled by a virtual host.
#     These directives also provide default values for the settings
#     of all virtual hosts.
#  3. Settings for virtual hosts, which allow Web requests to be sent to
#     different IP addresses or hostnames and have them handled by the
#     same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "/var/log/apache2/foo.log"
# with ServerRoot set to "" will be interpreted by the
# server as "//var/log/apache2/foo.log".
#

### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at <URL:http://httpd.apache.org/docs-2.1/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#<IfModule !mpm_winnt.c>
#<IfModule !mpm_netware.c>
LockFile /var/lock/apache2/accept.lock
#</IfModule>
#</IfModule>

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    MaxClients          150
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers         2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#

AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain


#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog /var/log/apache2/error.log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

# Include all the user configurations:
Include /etc/apache2/httpd.conf

# Include ports listing
Include /etc/apache2/ports.conf

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

#
# Define an access log for VirtualHosts that don't define their own logfile
CustomLog /var/log/apache2/other_vhosts_access.log vhost_combined

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections.  We use 
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
#   Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /usr/share/apache2/error/include/ files and copying them to /your/include/path/, 
# even on a per-VirtualHost basis.  The default include files will display
# your Apache version number and your ServerAdmin email address regardless
# of the setting of ServerSignature.
#
# The internationalized error documents require mod_alias, mod_include
# and mod_negotiation.  To activate them, uncomment the following 30 lines.

#    Alias /error/ "/usr/share/apache2/error/"
#
#    <Directory "/usr/share/apache2/error">
#        AllowOverride None
#        Options IncludesNoExec
#        AddOutputFilter Includes html
#        AddHandler type-map var
#        Order allow,deny
#        Allow from all
#        LanguagePriority en cs de es fr it nl sv pt-br ro
#        ForceLanguagePriority Prefer Fallback
#    </Directory>
#
#    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
#    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
#    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
#    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
#    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
#    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
#    ErrorDocument 410 /error/HTTP_GONE.html.var
#    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
#    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
#    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
#    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
#    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
#    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
#    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
#    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
#    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
#    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var



# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
Include /etc/apache2/conf.d/

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/

<Directory /var/www/sharedip>
    Options +Includes -Indexes
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

###############ispconfig_log###############
LogFormat "%v||||%b||||%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_ispconfig
CustomLog "|/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d" combined_ispconfig

<Directory /var/www/*/web>
    Options +Includes -Indexes
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

<Directory /var/www/*/user/*/web>
    Options +Includes -Indexes
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

<Directory /var/www/*/cgi-bin>
    Options ExecCGI -Indexes
    AllowOverride None
    AllowOverride Indexes AuthConfig Limit FileInfo
    Order allow,deny
    Allow from all
    <Files ~ "^\.ht">
    Deny from all
    </Files>
</Directory>

Include /etc/apache2/vhosts/Vhosts_ispconfig.conf

Include /etc/phpmyadmin/apache.conf
right back to the problem i first thought it was some kind of DOS/DDOS attack so i ran the following commands:

Quote:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

netstat -n | grep :80 |wc -l

netstat -n | grep :80 | grep SYN |wc -l
the first shows a bunch of ip's and there connections but there's nothing to weird as there is only a few ip's that have over 20-30 with the rest having very few

the second command shows the total connections and again nothing to odd about the number (150-250 ish)

the third is for SYN but that's always been low on testing it like 1-10

i think all this is quite normal (although i might be wrong) so i'm now quite stumped as to what the problem is and how to go about fixing it

server specs/info:

amd dual core 2.5ghz, 4gb ram, 2x 300gb hard drives etc

ubuntu 8.10 server (dedicated) with apache2-prefork/php5/mysql, webmin/ispconfig

Quote:
Apache Version: Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.6 with Suhosin-Patch
please advise thanks

note: i forgot to say that memory usage is a little higher then normal (500mb-700mb compared to 250mb-300mb - but goes back to normal after restarting apache) and the load is also a little high but always under 1 (also note my server is on a 100mbits port and i can easily push 100mbits 24/7 with no problems at all so it's not bandwidth)
 
Old 02-03-2010, 04:45 PM   #2
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 167Reputation: 167
Turn on status and when it gets slow do an 'apache2ctl status' and see if you have an excessive number of workers in G state.

Last edited by rweaver; 02-03-2010 at 04:46 PM.
 
Old 02-03-2010, 04:47 PM   #3
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by rweaver View Post
Turn on status and when it gets slow do an 'apache2ctl status' and see if you have an excessive number of workers in G state.
status as in the apache module right? if so it was already on... will post back in a few mins with some results

Last edited by steve51184; 02-03-2010 at 04:48 PM.
 
Old 02-03-2010, 04:49 PM   #4
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
Quote:
$ apache2ctl status
Forbidden

You don't have permission to access /server-status on this server.
did that with root and i get a permission error
 
Old 02-03-2010, 05:08 PM   #5
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
ok i got 'apache2ctl status' working

here's an output when it's fast (ie after i restart apache):

Quote:
Apache Server Status for localhost

Server Version: Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.6 with
Suhosin-Patch

Server Built: Nov 13 2009 21:56:01
__________________________________________________________________

Current Time: Thursday, 04-Feb-2010 00:03:58 CET
Restart Time: Thursday, 04-Feb-2010 00:02:36 CET
Parent Server Generation: 0
Server uptime: 1 minute 22 seconds
119 requests currently being processed, 7 idle workers

KWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWKWWWWKWWWWWWWWWWWWWWWKWWWWW
WWWKWWWWKWWWWWWWWWWKW_WW_WWWWWWWWWWWKWWWWWW__WWWKWKWW__WWWWWW_..
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
and here it is when apache is slow (ie 2mins after the last restart):

Quote:
Apache Server Status for localhost

Server Version: Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.6 with
Suhosin-Patch

Server Built: Nov 13 2009 21:56:01
__________________________________________________________________

Current Time: Thursday, 04-Feb-2010 00:08:20 CET
Restart Time: Thursday, 04-Feb-2010 00:02:36 CET
Parent Server Generation: 0
Server uptime: 5 minutes 44 seconds
128 requests currently being processed, 1 idle workers

WWWWWWWWKWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWKWWWWWWWWKWWWWWWWW
WWWWWWKKWWWWKWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWWWW
...................._...........................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
 
Old 02-03-2010, 06:34 PM   #6
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 167Reputation: 167
Typically a lot of W or G on a consistent basis mean connections aren't being closed correctly, sometimes its an intentional action of a remote user sometimes a poorly configured application accessing the site.

Try this (these changes will up cpu usage a tiny bit but help with performance quite a bit long term):

Turn timeout down to 150, turn keepalivetimeout down to 5, drop maxkeepaliverequests down to 75. If that doesn't correct your issue I can make a few more suggestions.
 
Old 02-03-2010, 06:39 PM   #7
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by rweaver View Post
Typically a lot of W or G on a consistent basis Turn timeout down to 150, turn keepalivetimeout down to 5, drop maxkeepaliverequests down to 75. If that doesn't correct your issue I can make a few more suggestions.
done these and i'll will report back if it has helped or not thank you
 
Old 02-03-2010, 07:01 PM   #8
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
nope that's not helping at all... any other suggestions?

edit: second thoughts that IS helping as it was about 2mins before apache 'froze/got slow' now it's MUCH longer like 15-20mins but it's still doing it so if i can get some more help please

Last edited by steve51184; 02-03-2010 at 07:09 PM.
 
Old 02-03-2010, 08:53 PM   #9
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by rweaver View Post
If that doesn't correct your issue I can make a few more suggestions.
what other suggestions do you have?
 
Old 02-04-2010, 12:00 PM   #10
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
second thought after some testing it's just the same...
 
Old 02-04-2010, 01:42 PM   #11
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 167Reputation: 167
Lets define exactly what is occurring, how does your memory usage look at restart vs 20m down the road? Are you dipping into swap for instance? If you perform an iostat/top how does the iowait for system appear in both cases? Does a graceful seem to help the performance or do you need ot do a full restart?
 
Old 02-04-2010, 05:06 PM   #12
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
well i remember before all this htop reported i had about 300mb usage but now after a apache restart i have about 450 used and 20mins later i have 1300mb used (and it's claiming fast and will continue to do so until i restart apache)

also i don't have any swap as i have 4gb of ram and didn't think it was needed

and iostat before and after looks the same

after a restart (fast):

Quote:
$ iostat
Linux 2.6.27.9 (H045) 02/04/10 _i686_

avg-cpu: %user %nice %system %iowait %steal %idle
2.87 0.02 2.93 1.77 0.00 92.41

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 17.00 368.87 318.36 3994335186 3447339704
sda1 17.00 368.87 318.36 3994334886 3447339704
sda2 0.00 0.00 0.00 292 0
sdb 10.06 318.72 88.33 3451300154 956505104
sdb1 10.06 318.72 88.33 3451300146 956505104
10mins after a restart (slow):

Quote:
$ iostat
Linux 2.6.27.9 (H045) 02/05/10 _i686_

avg-cpu: %user %nice %system %iowait %steal %idle
2.87 0.02 2.93 1.77 0.00 92.41

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 17.00 369.11 318.35 3997361570 3447584920
sda1 17.00 369.11 318.35 3997361270 3447584920
sda2 0.00 0.00 0.00 292 0
sdb 10.06 319.05 88.32 3455231178 956521400
sdb1 10.06 319.05 88.32 3455231170 956521400

also i just want to clarify a little by what i mean by 'apache is slow' as after a restart pages load with a 1.2 seconds to about 1.5 seconds load time and after 2-5mins there up to about 30 seconds+

edit: had to restart again as i was upto 1600mb ram usage and 30 seconds load times :\

Last edited by steve51184; 02-04-2010 at 05:13 PM.
 
Old 02-04-2010, 10:03 PM   #13
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
rweaver can you help with the above post please?
 
Old 02-05-2010, 04:11 PM   #14
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 167Reputation: 167
Turn on the coredump directory in apache and post the output of a gdb /path/to/apache -c core.pid and we'll attempt to diagnose what is causing your issues since you mentioned that you were seeing massive numbers of segfaults in your error log.
 
Old 02-05-2010, 04:36 PM   #15
steve51184
Member
 
Registered: Dec 2006
Posts: 381

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by rweaver View Post
Turn on the coredump directory in apache and post the output of a gdb /path/to/apache -c core.pid and we'll attempt to diagnose what is causing your issues since you mentioned that you were seeing massive numbers of segfaults in your error log.
it's now been 4 hours since i tuned this on but there's still not been any files in the folder... did i do it wrong?

Code:
mkdir /home2/apache
chown www-data.www-data /home2/apache
chmod 777 /home2/apache
echo "CoreDumpDirectory /home2/apache" >> /etc/apache2/apache2.conf
/etc/init.d/apache2 restart
but still no new files in /home2/apache?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: One thing that preload helps run quickly: OpenOffice LXer Syndicated Linux News 0 08-15-2008 01:30 PM
Restarting apache fof3 Linux - Newbie 2 07-22-2006 10:01 PM
Apache helps needed !!! chuck77 Linux - General 1 05-15-2002 10:08 AM
Apache helps needed !!! chuck77 Linux - General 11 01-28-2002 09:46 PM
restarting Apache Xiled Linux - Networking 7 10-09-2001 10:24 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 07:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration