LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Trying to understand this readout from top -b (https://www.linuxquestions.org/questions/linux-newbie-8/trying-to-understand-this-readout-from-top-b-4175449304/)

unSpawn 02-10-2013 08:04 PM

The fact your server hasn't crashed since you made changes is hopeful but it does make me wonder what other changes you made that could have adversely affected performance. Maybe it would be best if you start back at the beginning and show us the machine hardware specs (you seem to have 8GB RAM?) and an inventory of installed software. (For example your CentOS 6.2 is one update behind and how about your web stack, Drupal and other software?) Collecting system statistics is a good move but you should monitor other aspects as well like network traffic, your web server and your database. For example Atsar works like "regular" SAR but can also log Apache HTTP traffic (or see "Apache Top"). Like Atsar Dstat aims to replace vmstat, iostat and ifstat but it can also log MySQL statistics (else see "MysQL Top" or "mtop" for short). As for your system logs I suggest you run them through Logwatch at detail level 10 as that's the easiest, quickest way to generate leads. I know this all wasn't very detailed but you have to start somewhere, right?

david59 02-13-2013 04:15 AM

Hello, am traveling right now... will be back tomorrow and respond. Apologies for the delay and thank you so much, david

david59 02-14-2013 05:35 AM

This is a reply to chrism01, from another thread I started on this topic in Linux - Newbie (after realizing that perhaps I had started in the wrong forum). That thread is here:

Code:

http://www.linuxquestions.org/questions/linux-newbie-8/trying-to-understand-this-readout-from-top-b-4175449304/#post4887575
My sincere apologies for the confusion.



Hello Chris, thank you for writing and apologies for the delay in getting back to you. The crashes began quite suddenly, seemingly every 24 hours at more or less the same time. As for the /var/log files, I have looked, and there is nothing remarkable there.

Code:

[root@u1584224530 /]# cat /etc/*release*
CentOS release 6.2 (Final)
CentOS release 6.2 (Final)
CentOS release 6.2 (Final)
cpe:/o:centos:linux:6:GA

Code:

[root@u16271690 /]# uname -a
Linux u16271690.onlinehome-server.com 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64 GNU/Linux


Following is the .htaccess file.

.htaccess

Code:

#
# Apache/PHP/Drupal settings:
#

# Protect files and directories from prying eyes.
<FilesMatch "\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(\..*|Entries.*|Repository|Root|Tag|Template)$">
  Order allow,deny
</FilesMatch>


# Don't show directory listings for URLs which map to a directory.
Options -Indexes

# Follow symbolic links in this directory.
Options +FollowSymLinks

# Make Drupal handle any 404 errors.
ErrorDocument 404 /index.php

# Set the default handler.
DirectoryIndex index.php index.html index.htm

# Override PHP settings that cannot be changed at runtime. See
# sites/default/default.settings.php and drupal_environment_initialize() in
# includes/bootstrap.inc for settings that can be changed at runtime.

# PHP 5, Apache 1 and 2.
<IfModule mod_php5.c>
  php_flag magic_quotes_gpc                off
  php_flag magic_quotes_sybase              off
  php_flag register_globals                off
  php_flag session.auto_start              off
  php_value mbstring.http_input            pass
  php_value mbstring.http_output            pass
  php_flag mbstring.encoding_translation    off
</IfModule>

# Requires mod_expires to be enabled.
<IfModule mod_expires.c>
  # Enable expirations.
  ExpiresActive On

  # Cache all files for 2 weeks after access (A).
  ExpiresDefault A1209600

  <FilesMatch \.php$>
    # Do not allow PHP scripts to be cached unless they explicitly send cache
    # headers themselves. Otherwise all scripts would have to overwrite the
    # headers set by mod_expires if they want another caching behavior. This may
    # fail if an error occurs early in the bootstrap process, and it may cause
    # problems if a non-Drupal PHP file is installed in a subdirectory.
    ExpiresActive Off
  </FilesMatch>
</IfModule>

# Various rewrite rules.
<IfModule mod_rewrite.c>
  RewriteEngine on

  # Block access to "hidden" directories whose names begin with a period. This
  # includes directories used by version control systems such as Subversion or
  # Git to store control files. Files whose names begin with a period, as well
  # as the control files used by CVS, are protected by the FilesMatch directive
  # above.
  #
  # NOTE: This only works when mod_rewrite is loaded. Without mod_rewrite, it is
  # not possible to block access to entire directories from .htaccess, because
  # <DirectoryMatch> is not allowed here.
  #
  # If you do not have mod_rewrite installed, you should remove these
  # directories from your webroot or otherwise protect them from being
  # downloaded.
  RewriteRule "(^|/)\." - [F]

  # If your site can be accessed both with and without the 'www.' prefix, you
  # can use one of the following settings to redirect users to your preferred
  # URL, either WITH or WITHOUT the 'www.' prefix. Choose ONLY one option:
  #
  # To redirect all users to access the site WITH the 'www.' prefix,
  # (http://example.com/... will be redirected to http://www.example.com/...)
  # uncomment the following:
  # RewriteCond %{HTTP_HOST} !^www\. [NC]
  # RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
  #
  # To redirect all users to access the site WITHOUT the 'www.' prefix,
  # (http://www.example.com/... will be redirected to http://example.com/...)
  # uncomment the following:
  # RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
  # RewriteRule ^ http://%1%{REQUEST_URI} [L,R=301]

  # Modify the RewriteBase if you are using Drupal in a subdirectory or in a
  # VirtualDocumentRoot and the rewrite rules are not working properly.
  # For example if your site is at http://example.com/drupal uncomment and
  # modify the following line:
  # RewriteBase /drupal
  #
  # If your site is running in a VirtualDocumentRoot at http://example.com/,
  # uncomment the following line:
  # RewriteBase /

  # Pass all requests not referring directly to files in the filesystem to
  # index.php. Clean URLs are handled in drupal_environment_initialize().
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !=/favicon.ico
  RewriteRule ^ index.php [L]

  # Rules to correctly serve gzip compressed CSS and JS files.
  # Requires both mod_rewrite and mod_headers to be enabled.
  <IfModule mod_headers.c>
    # Serve gzip compressed CSS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    # Serve gzip compressed JS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

    # Serve correct content types, and prevent mod_deflate double gzip.
    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

    <FilesMatch "(\.js\.gz|\.css\.gz)$">
      # Serve correct encoding type.
      Header set Content-Encoding gzip
      # Force proxies to cache gzipped & non-gzipped css/js files separately.
      Header append Vary Accept-Encoding
    </FilesMatch>
  </IfModule>
</IfModule>


david59 02-14-2013 05:40 AM

httpd.conf file, part 1 (I moved the directives for bots there, as you advised above)

Code:

#
# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.2/> for detailed information.
# In particular, see
# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
# for a discussion of each configuration directive.
#
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned. 
#
# The configuration directives are grouped into three basic sections:
#  1. Directives that control the operation of the Apache server process as a
#    whole (the 'global environment').
#  2. Directives that define the parameters of the 'main' or 'default' server,
#    which responds to requests that aren't handled by a virtual host.
#    These directives also provide default values for the settings
#    of all virtual hosts.
#  3. Settings for virtual hosts, which allow Web requests to be sent to
#    different IP addresses or hostnames and have them handled by the
#    same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "logs/foo.log"
# with ServerRoot set to "/etc/httpd" will be interpreted by the
# server as "/etc/httpd/logs/foo.log".
#

### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# Don't give away too much information about all the subcomponents
# we are running.  Comment out this line if you don't mind remote sites
# finding out what major optional modules you are running
ServerTokens OS

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation
# (available at <URL:http://httpd.apache.org/docs/2.2/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/etc/httpd"

#
# PidFile: The file in which the server should record its process
# identification number when it starts.  Note the PIDFILE variable in
# /etc/sysconfig/httpd must be set appropriately if this location is
# changed.
#
PidFile run/httpd.pid

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 60

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers      8
MinSpareServers    5
MaxSpareServers  20
ServerLimit      256
MaxClients      256
MaxRequestsPerChild  4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers        4
MaxClients        200
MinSpareThreads    25
MaxSpareThreads    75
ThreadsPerChild    25
MaxRequestsPerChild  0
</IfModule>

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 80

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
#LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule version_module modules/mod_version.so

#
# The following modules are not loaded by default:
#
#LoadModule asis_module modules/mod_asis.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule cern_meta_module modules/mod_cern_meta.so
#LoadModule cgid_module modules/mod_cgid.so
#LoadModule dbd_module modules/mod_dbd.so
#LoadModule dumpio_module modules/mod_dumpio.so
LoadModule filter_module modules/mod_filter.so
#LoadModule ident_module modules/mod_ident.so
#LoadModule log_forensic_module modules/mod_log_forensic.so
#LoadModule unique_id_module modules/mod_unique_id.so
#

#
# Load config files from the config directory "/etc/httpd/conf.d".
#
Include conf.d/*.conf

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
#ExtendedStatus On

#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch. 
#
# User/Group: The name (or #number) of the user/group to run httpd as.
#  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
#  . On HPUX you may not be able to use shared memory as nobody, and the
#    suggested workaround is to create a user www and use that user.
#  NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
#  when the value of (unsigned)Group is above 60000;
#  don't use Group #-1 on these systems!
#
User apache
Group apache

### Section 2: 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin@your-domain.com
#
ServerAdmin root@localhost

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If this is not set to valid DNS name for your host, server-generated
# redirections will not work.  See also the UseCanonicalName directive.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
# You will have to access it by its address anyway, and this will make
# redirections work in a sensible way.
#
#ServerName www.example.com:80

#
# UseCanonicalName: Determines how Apache constructs self-referencing
# URLs and the SERVER_NAME and SERVER_PORT variables.
# When set "Off", Apache will use the Hostname and Port supplied
# by the client.  When set "On", Apache will use the value of the
# ServerName directive.
#
UseCanonicalName Off

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"

#
# Each directory to which Apache has access can be configured with respect
# to which services and features are allowed and/or disabled in that
# directory (and its subdirectories).
#
# First, we configure the "default" to be a very restrictive set of
# features. 
#
<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# This should be changed to whatever you set DocumentRoot to.
#
<Directory "/var/www/html">

#
# Possible values for the Options directive are "None", "All",
# or any combination of:
#  Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#options
# for more information.
#
    Options Indexes FollowSymLinks

#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
#  Options FileInfo AuthConfig Limit
#
    AllowOverride None

#
# Controls who can get stuff from this server.
#
    Order allow,deny
    Allow from all

</Directory>

#
# UserDir: The name of the directory that is appended onto a user's home
# directory if a ~user request is received.
#
# The path to the end user account 'public_html' directory must be
# accessible to the webserver userid.  This usually means that ~userid
# must have permissions of 711, ~userid/public_html must have permissions
# of 755, and documents contained therein must be world-readable.
# Otherwise, the client will only receive a "403 Forbidden" message.
#
# See also: http://httpd.apache.org/docs/misc/FAQ.html#forbidden
#
<IfModule mod_userdir.c>
    #
    # UserDir is disabled by default since it can confirm the presence
    # of a username on the system (depending on home directory
    # permissions).
    #
    UserDir disabled

    #
    # To enable requests to /~user/ to serve the user's public_html
    # directory, remove the "UserDir disabled" line above, and uncomment
    # the following line instead:
    #
    #UserDir public_html

</IfModule>

#
# Control access to UserDir directories.  The following is an example
# for a site where these directories are restricted to read-only.
#
#<Directory /home/*/public_html>
#    AllowOverride FileInfo AuthConfig Limit
#    Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
#    <Limit GET POST OPTIONS>
#        Order allow,deny
#        Allow from all
#    </Limit>
#    <LimitExcept GET POST OPTIONS>
#        Order deny,allow
#        Deny from all
#    </LimitExcept>
#</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
# The index.html.var file (a type-map) is used to deliver content-
# negotiated documents.  The MultiViews Option can be used for the
# same purpose, but it is much slower.
#
DirectoryIndex index.html index.html.var index.shtml index.cfm index.php index.htm

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#
AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</Files>

#
# TypesConfig describes where the mime.types file (or equivalent) is
# to be found.
#
TypesConfig /etc/mime.types

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
<IfModule mod_mime_magic.c>
#  MIMEMagicFile /usr/share/magic.mime
    MIMEMagicFile conf/magic
</IfModule>

#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

#
# EnableMMAP: Control whether memory-mapping is used to deliver
# files (assuming that the underlying OS supports it).
# The default is on; turn this off if you serve from NFS-mounted
# filesystems.  On some systems, turning it off (regardless of
# filesystem) can improve performance; for details, please see
# http://httpd.apache.org/docs/2.2/mod/core.html#enablemmap
#
#EnableMMAP off

#
# EnableSendfile: Control whether the sendfile kernel support is
# used to deliver files (assuming that the OS supports it).
# The default is on; turn this off if you serve from NFS-mounted
# filesystems.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile
#
#EnableSendfile off

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog logs/error_log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

# "combinedio" includes actual counts of actual bytes received (%I) and sent (%O); this
# requires the mod_logio module to be loaded.
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here.  Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
#CustomLog logs/access_log common

#
# If you would like to have separate agent and referer logfiles, uncomment
# the following directives.
#
#CustomLog logs/referer_log referer
#CustomLog logs/agent_log agent

#
# For a single logfile with access, agent, and referer information
# (Combined Logfile Format), use the following directive:
#
CustomLog logs/access_log combined

#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature On

#
# Aliases: Add here as many aliases as you need (with no limit). The format is
# Alias fakename realname
#
# Note that if you include a trailing / on fakename then the server will
# require it to be present in the URL.  So "/icons" isn't aliased in this
# example, only "/icons/".  If the fakename is slash-terminated, then the
# realname must also be slash terminated, and if the fakename omits the
# trailing slash, the realname must also omit it.
#
# We include the /icons/ alias for FancyIndexed directory listings.  If you
# do not use FancyIndexing, you may comment this out.
#
Alias /icons/ "/var/www/icons/"

<Directory "/var/www/icons">
    Options Indexes MultiViews FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

#
# WebDAV module configuration section.
#
<IfModule mod_dav_fs.c>
    # Location of the WebDAV lock database.
    DAVLockDB /var/lib/dav/lockdb
</IfModule>

#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

#
# Redirect allows you to tell clients about documents which used to exist in
# your server's namespace, but do not anymore. This allows you to tell the
# clients where to look for the relocated document.
# Example:
# Redirect permanent /foo http://www.example.com/bar

#
# Directives controlling the display of server-generated directory listings.
#

#
# IndexOptions: Controls the appearance of server-generated directory
# listings.
#
IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable Charset=UTF-8

#
# AddIcon* directives tell the server which icon to show for different
# files or filename extensions.  These are only displayed for
# FancyIndexed directories.
#
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip

AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*

AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core

AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^

#
# DefaultIcon is which icon to show for files which do not have an icon
# explicitly set.
#
DefaultIcon /icons/unknown.gif

#
# AddDescription allows you to place a short description after a file in
# server-generated indexes.  These are only displayed for FancyIndexed
# directories.
# Format: AddDescription "description" filename
#
#AddDescription "GZIP compressed document" .gz
#AddDescription "tar archive" .tar
#AddDescription "GZIP compressed tar archive" .tgz

#
# ReadmeName is the name of the README file the server will look for by
# default, and append to directory listings.
#
# HeaderName is the name of a file which should be prepended to
# directory indexes.
ReadmeName README.html
HeaderName HEADER.html

#
# IndexIgnore is a set of filenames which directory indexing should ignore
# and not include in the listing.  Shell-style wildcarding is permitted.
#
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t

#
# DefaultLanguage and AddLanguage allows you to specify the language of
# a document. You can then use content negotiation to give a browser a
# file in a language the user can understand.
#
# Specify a default language. This means that all data
# going out without a specific language tag (see below) will
# be marked with this one. You probably do NOT want to set
# this unless you are sure it is correct for all cases.
#
# * It is generally better to not mark a page as
# * being a certain language than marking it with the wrong
# * language!
#
# DefaultLanguage nl
#
# Note 1: The suffix does not have to be the same as the language
# keyword --- those with documents in Polish (whose net-standard
# language code is pl) may wish to use "AddLanguage pl .po" to
# avoid the ambiguity with the common suffix for perl scripts.
#
# Note 2: The example entries below illustrate that in some cases
# the two character 'Language' abbreviation is not identical to
# the two character 'Country' code for its country,
# E.g. 'Danmark/dk' versus 'Danish/da'.
#
# Note 3: In the case of 'ltz' we violate the RFC by using a three char
# specifier. There is 'work in progress' to fix this and get
# the reference data for rfc1766 cleaned up.

part 2 below

david59 02-14-2013 05:40 AM

part 2 of httpd.conf file

Code:

#
# Catalan (ca) - Croatian (hr) - Czech (cs) - Danish (da) - Dutch (nl)
# English (en) - Esperanto (eo) - Estonian (et) - French (fr) - German (de)
# Greek-Modern (el) - Hebrew (he) - Italian (it) - Japanese (ja)
# Korean (ko) - Luxembourgeois* (ltz) - Norwegian Nynorsk (nn)
# Norwegian (no) - Polish (pl) - Portugese (pt)
# Brazilian Portuguese (pt-BR) - Russian (ru) - Swedish (sv)
# Simplified Chinese (zh-CN) - Spanish (es) - Traditional Chinese (zh-TW)
#
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw

#
# LanguagePriority allows you to give precedence to some languages
# in case of a tie during content negotiation.
#
# Just list the languages in decreasing order of preference. We have
# more or less alphabetized them here. You probably want to change this.
#
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW

#
# ForceLanguagePriority allows you to serve a result page rather than
# MULTIPLE CHOICES (Prefer) [in case of a tie] or NOT ACCEPTABLE (Fallback)
# [in case no accepted languages matched the available variants]
#
ForceLanguagePriority Prefer Fallback

#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
#AddDefaultCharset UTF-8

#
# AddType allows you to add to or override the MIME configuration
# file mime.types for specific file types.
#
#AddType application/x-tar .tgz

#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
# Despite the name similarity, the following Add* directives have nothing
# to do with the FancyIndexing customization directives above.
#
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz

# If the AddEncoding directives above are commented-out, then you
# probably should define those extensions to indicate media types:
#
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

#
#  MIME-types for downloading Certificates and CRLs
#
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
#AddHandler cgi-script .cgi

#
# For files that include their own HTTP headers:
#
#AddHandler send-as-is asis

#
# For type maps (negotiated resources):
# (This is enabled by default to allow the Apache "It Worked" page
#  to be distributed in multiple languages.)
#
AddHandler type-map var

#
# Filters allow you to process content before it is sent to the client.
#
# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

#
# Action lets you define media types that will execute a script whenever
# a matching file is called. This eliminates the need for repeated URL
# pathnames for oft-used CGI file processors.
# Format: Action media/type /cgi-script/location
# Format: Action handler-name /cgi-script/location
#

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections.  We use
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
#  Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /var/www/error/include/ files and
# copying them to /your/include/path/, even on a per-VirtualHost basis.
#

#block spider
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC]
RewriteRule .* - [F]
#
#Block bots.
SetEnvIfNoCase user-Agent ^$ bad_bot
SetEnvIfNoCase user-Agent "^AESOP_com_SpiderMan" bad_bot
SetEnvIfNoCase user-Agent "^Alexibot" bad_bot
SetEnvIfNoCase user-Agent "Anonymouse.org" bad_bot
SetEnvIfNoCase user-Agent "^asterias" bad_bot
SetEnvIfNoCase user-Agent "^attach" bad_bot
SetEnvIfNoCase user-Agent "^BackDoorBot" bad_bot
SetEnvIfNoCase user-Agent "^BackWeb" bad_bot
SetEnvIfNoCase user-Agent "bandit" bad_bot
SetEnvIfNoCase user-Agent "^Baiduspider" bad_bot
SetEnvIfNoCase user-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase user-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase user-Agent "^Black.Hole" bad_bot
SetEnvIfNoCase user-Agent "^BlackWidow" bad_bot
SetEnvIfNoCase user-Agent "^BlowFish" bad_bot
SetEnvIfNoCase user-Agent "^Bot\ mailto:craftbot@yahoo.com" bad_bot
SetEnvIfNoCase user-Agent "^BotALot" bad_bot
SetEnvIfNoCase user-Agent "Buddy" bad_bot
SetEnvIfNoCase user-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase user-Agent "^Bullseye" bad_bot
SetEnvIfNoCase user-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase user-Agent "^Cegbfeieh" bad_bot
SetEnvIfNoCase user-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase user-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase user-Agent "^ChinaClaw" bad_bot
SetEnvIfNoCase user-Agent "Collector" bad_bot
SetEnvIfNoCase user-Agent "Copier" bad_bot
SetEnvIfNoCase user-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase user-Agent "^cosmos" bad_bot
SetEnvIfNoCase user-Agent "^Crescent" bad_bot
SetEnvIfNoCase user-Agent "^Curl" bad_bot
SetEnvIfNoCase user-Agent "^Custo" bad_bot
SetEnvIfNoCase user-Agent "^DA" bad_bot
SetEnvIfNoCase user-Agent "^DISCo" bad_bot
SetEnvIfNoCase user-Agent "^DIIbot" bad_bot
SetEnvIfNoCase user-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase user-Agent "^Download" bad_bot
SetEnvIfNoCase user-Agent "^Download\ Demon" bad_bot
SetEnvIfNoCase user-Agent "^Download\ Devil" bad_bot
SetEnvIfNoCase user-Agent "^Download\ Wonder" bad_bot
SetEnvIfNoCase user-Agent "Downloader" bad_bot
SetEnvIfNoCase user-Agent "^dragonfly" bad_bot
SetEnvIfNoCase user-Agent "^Drip" bad_bot
SetEnvIfNoCase user-Agent "^eCatch" bad_bot
SetEnvIfNoCase user-Agent "^EasyDL" bad_bot
SetEnvIfNoCase user-Agent "^ebingbong" bad_bot
SetEnvIfNoCase user-Agent "^EirGrabber" bad_bot
SetEnvIfNoCase user-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase user-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase user-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase user-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase user-Agent "^Exabot" bad_bot
SetEnvIfNoCase user-Agent "^Express\ WebPictures" bad_bot
SetEnvIfNoCase user-Agent "Extractor" bad_bot
SetEnvIfNoCase user-Agent "^EyeNetIE" bad_bot
SetEnvIfNoCase user-Agent "^FileHound" bad_bot
SetEnvIfNoCase user-Agent "^FlashGet" bad_bot
SetEnvIfNoCase user-Agent "^Foobot" bad_bot
SetEnvIfNoCase user-Agent "^flunky" bad_bot
SetEnvIfNoCase user-Agent "^FrontPage" bad_bot
SetEnvIfNoCase user-Agent "^GetRight" bad_bot
SetEnvIfNoCase user-Agent "^GetSmart" bad_bot
SetEnvIfNoCase user-Agent "^GetWeb!" bad_bot
SetEnvIfNoCase user-Agent "^Go!Zilla" bad_bot
SetEnvIfNoCase user-Agent "Google\ Wireless\ Transcoder" bad_bot
SetEnvIfNoCase user-Agent "^Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase user-Agent "^gotit" bad_bot
SetEnvIfNoCase user-Agent "Grabber" bad_bot
SetEnvIfNoCase user-Agent "^GrabNet" bad_bot
SetEnvIfNoCase user-Agent "^Grafula" bad_bot
SetEnvIfNoCase user-Agent "^Harvest" bad_bot
SetEnvIfNoCase user-Agent "^hloader" bad_bot
SetEnvIfNoCase user-Agent "^HMView" bad_bot
SetEnvIfNoCase user-Agent "^httplib" bad_bot
SetEnvIfNoCase user-Agent "^HTTrack" bad_bot
SetEnvIfNoCase user-Agent "^humanlinks" bad_bot
SetEnvIfNoCase user-Agent "^ia_archiver" bad_bot
SetEnvIfNoCase user-Agent "^IlseBot" bad_bot
SetEnvIfNoCase user-Agent "^Image\ Stripper" bad_bot
SetEnvIfNoCase user-Agent "^Image\ Sucker" bad_bot
SetEnvIfNoCase user-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase user-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase user-Agent "^InfoTekies" bad_bot
SetEnvIfNoCase user-Agent "^Intelliseek" bad_bot
SetEnvIfNoCase user-Agent "^InterGET" bad_bot
SetEnvIfNoCase user-Agent "^Internet\ Ninja" bad_bot
SetEnvIfNoCase user-Agent "^Iria" bad_bot
SetEnvIfNoCase user-Agent "^Jakarta" bad_bot
SetEnvIfNoCase user-Agent "^JennyBot" bad_bot
SetEnvIfNoCase user-Agent "^JetCar" bad_bot
SetEnvIfNoCase user-Agent "^JOC" bad_bot
SetEnvIfNoCase user-Agent "^JustView" bad_bot
SetEnvIfNoCase user-Agent "^Jyxobot" bad_bot
SetEnvIfNoCase user-Agent "^Kenjin.Spider" bad_bot
SetEnvIfNoCase user-Agent "^Keyword.Density" bad_bot
SetEnvIfNoCase user-Agent "^larbin" bad_bot
SetEnvIfNoCase user-Agent "^LeechFTP" bad_bot
SetEnvIfNoCase user-Agent "^LexiBot" bad_bot
SetEnvIfNoCase user-Agent "^lftp" bad_bot
SetEnvIfNoCase user-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase user-Agent "^likse" bad_bot
SetEnvIfNoCase user-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase user-Agent "^LinkScan/8.1a.Unix" bad_bo
SetEnvIfNoCase user-Agent "^LNSpiderguy" bad_bott
SetEnvIfNoCase user-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase user-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase user-Agent "^LWP::Simple" bad_bot
SetEnvIfNoCase user-Agent "^Magnet" bad_bot
SetEnvIfNoCase user-Agent "^Mag-Net" bad_bot
SetEnvIfNoCase user-Agent "^MarkWatch" bad_bot
SetEnvIfNoCase user-Agent "^Mass\ Downloader" bad_bot
SetEnvIfNoCase user-Agent "^Mata.Hari" bad_bot
SetEnvIfNoCase user-Agent "^Memo" bad_bot
SetEnvIfNoCase user-Agent "^Microsoft.URL" bad_bot
SetEnvIfNoCase user-Agent "^Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase user-Agent "^MIDown\ tool" bad_bot
SetEnvIfNoCase user-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase user-Agent "^Mirror" bad_bot
SetEnvIfNoCase user-Agent "^Missigua\ Locator" bad_bot
SetEnvIfNoCase user-Agent "^Mister\ PiX" bad_bot
SetEnvIfNoCase user-Agent "^moget" bad_bot
SetEnvIfNoCase user-Agent "^Mozilla/3.Mozilla/2.01" bad_bot
SetEnvIfNoCase user-Agent "^Mozilla.*NEWT" bad_bot
SetEnvIfNoCase user-Agent "^NAMEPROTECT" bad_bot
SetEnvIfNoCase user-Agent "^Navroad" bad_bot
SetEnvIfNoCase user-Agent "^NearSite" bad_bot
SetEnvIfNoCase user-Agent "^NetAnts" bad_bot
SetEnvIfNoCase user-Agent "^Netcraft" bad_bot
SetEnvIfNoCase user-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase user-Agent "^NetSpider" bad_bot
SetEnvIfNoCase user-Agent "^Net\ Vampire" bad_bot
SetEnvIfNoCase user-Agent "^NetZIP" bad_bot
SetEnvIfNoCase user-Agent "^NextGenSearchBot" bad_bot
SetEnvIfNoCase user-Agent "^NG" bad_bot
SetEnvIfNoCase user-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase user-Agent "^NimbleCrawler" bad_bot
SetEnvIfNoCase user-Agent "^Ninja" bad_bot
SetEnvIfNoCase user-Agent "^NPbot" bad_bot
SetEnvIfNoCase user-Agent "^Octopus" bad_bot
SetEnvIfNoCase user-Agent "^Offline\ Explorer" bad_bot
SetEnvIfNoCase user-Agent "^Offline\ Navigator" bad_bot
SetEnvIfNoCase user-Agent "^Openfind" bad_bot
SetEnvIfNoCase user-Agent "^OutfoxBot" bad_bot
SetEnvIfNoCase user-Agent "^PageGrabber" bad_bot
SetEnvIfNoCase user-Agent "^Papa\ Foto" bad_bot
SetEnvIfNoCase user-Agent "^pavuk" bad_bot
SetEnvIfNoCase user-Agent "^pcBrowser" bad_bot
SetEnvIfNoCase user-Agent "^PHP\ version\ tracker" bad_bot
SetEnvIfNoCase user-Agent "^Pockey" bad_bot
SetEnvIfNoCase user-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase user-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase user-Agent "^psbot" bad_bot
SetEnvIfNoCase user-Agent "^Pump" bad_bot
SetEnvIfNoCase user-Agent "^QueryN.Metasearch" bad_bot
SetEnvIfNoCase user-Agent "^RealDownload" bad_bot
SetEnvIfNoCase user-Agent "Reaper" bad_bot
SetEnvIfNoCase user-Agent "Recorder" bad_bot
SetEnvIfNoCase user-Agent "^ReGet" bad_bot
SetEnvIfNoCase user-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase user-Agent "^RMA" bad_bot
SetEnvIfNoCase user-Agent "Siphon" bad_bot
SetEnvIfNoCase user-Agent "sitecheck.internetseer.com" bad_bot
SetEnvIfNoCase user-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase user-Agent "^SlySearch" bad_bot
SetEnvIfNoCase user-Agent "^SmartDownload" bad_bot
SetEnvIfNoCase user-Agent "^Snake" bad_bot
SetEnvIfNoCase user-Agent "^Snapbot" bad_bot
SetEnvIfNoCase user-Agent "^Snoopy" bad_bot
SetEnvIfNoCase user-Agent "^sogou" bad_bot
SetEnvIfNoCase user-Agent "^SpaceBison" bad_bot
SetEnvIfNoCase user-Agent "^SpankBot" bad_bot
SetEnvIfNoCase user-Agent "^spanner" bad_bot
SetEnvIfNoCase user-Agent "^Sqworm" bad_bot
SetEnvIfNoCase user-Agent "Stripper" bad_bot
SetEnvIfNoCase user-Agent "Sucker" bad_bot
SetEnvIfNoCase user-Agent "^SuperBot" bad_bot
SetEnvIfNoCase user-Agent "^SuperHTTP" bad_bot
SetEnvIfNoCase user-Agent "^Surfbot" bad_bot
SetEnvIfNoCase user-Agent "^suzuran" bad_bot
SetEnvIfNoCase user-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase user-Agent "^tAkeOut" bad_bot
SetEnvIfNoCase user-Agent "^Teleport" bad_bot
SetEnvIfNoCase user-Agent "^Telesoft" bad_bot
SetEnvIfNoCase user-Agent "^TurnitinBot/1.5" bad_bot
SetEnvIfNoCase user-Agent "^The.Intraformant" bad_bot
SetEnvIfNoCase user-Agent "^TheNomad" bad_bot
SetEnvIfNoCase user-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase user-Agent "^Titan" bad_bot
SetEnvIfNoCase user-Agent "^toCrawl/UrlDispatcher" bad_bot
SetEnvIfNoCase user-Agent "^True_Robot" bad_bot
SetEnvIfNoCase user-Agent "^turingos" bad_bot
SetEnvIfNoCase user-Agent "^TurnitinBot" bad_bot
SetEnvIfNoCase user-Agent "^URLy.Warning" bad_bot
SetEnvIfNoCase user-Agent "^Vacuum" bad_bot
SetEnvIfNoCase user-Agent "^VCI" bad_bot
SetEnvIfNoCase user-Agent "^VoidEYE" bad_bot
SetEnvIfNoCase user-Agent "^Web\ Image\ Collector" bad_bot
SetEnvIfNoCase user-Agent "^Web\ Sucker" bad_bot
SetEnvIfNoCase user-Agent "^WebAuto" bad_bot
SetEnvIfNoCase user-Agent "^Webbandit" bad_bot
SetEnvIfNoCase user-Agent "^Webclipping.com" bad_bot
SetEnvIfNoCase user-Agent "^WebCopier" bad_bot
SetEnvIfNoCase user-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase user-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase user-Agent "^WebFetch" bad_bot
SetEnvIfNoCase user-Agent "^WebGo\ IS" bad_bot
SetEnvIfNoCase user-Agent "^Web.Image.Collector" bad_bot
SetEnvIfNoCase user-Agent "^WebLeacher" bad_bot
SetEnvIfNoCase user-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase user-Agent "^WebReaper" bad_bot
SetEnvIfNoCase user-Agent "^WebSauger" bad_bot
SetEnvIfNoCase user-Agent "^WebSite" bad_bot
SetEnvIfNoCase user-Agent "^Website\ eXtractor" bad_bot
SetEnvIfNoCase user-Agent "^Website\ Quester" bad_bot
SetEnvIfNoCase user-Agent "^Webster" bad_bot
SetEnvIfNoCase user-Agent "^WebStripper" bad_bot
SetEnvIfNoCase user-Agent "^WebWhacker" bad_bot
SetEnvIfNoCase user-Agent "^WebZIP" bad_bot
SetEnvIfNoCase user-Agent "^Wget" bad_bot
SetEnvIfNoCase user-Agent "Whacker" bad_bot
SetEnvIfNoCase user-Agent "^Widow" bad_bot
SetEnvIfNoCase user-Agent "^WISENutbot" bad_bot
SetEnvIfNoCase user-Agent "^WWWOFFLE" bad_bot
SetEnvIfNoCase user-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase user-Agent "^Xaldon" bad_bot
SetEnvIfNoCase user-Agent "^Xenu" bad_bot
SetEnvIfNoCase user-Agent "^Zeus" bad_bot
SetEnvIfNoCase user-Agent "^Zyborg" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

# Modification for dealing with botnet DDoS via high CPU utilization
# by overwhelming PHP with POST data
#
# Referer is empty
RewriteCond %{HTTP_REFERER} ^$
# User agent is empty
RewriteCond %{HTTP_USER_AGENT} ^$
# The request is for the home page
RewriteCond %{REQUEST_URI} ^/$
# It is a POST request
RewriteCond %{REQUEST_METHOD} POST
# Forbid the request
RewriteRule ^(.*)$ - [F,L]


Alias /error/ "/var/www/error/"

<IfModule mod_negotiation.c>
<IfModule mod_include.c>
    <Directory "/var/www/error">
        AllowOverride None
        Options IncludesNoExec
        AddOutputFilter Includes html
        AddHandler type-map var
        Order allow,deny
        Allow from all
        LanguagePriority en es de fr
        ForceLanguagePriority Prefer Fallback
    </Directory>

#    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
#    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
#    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
#    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
#    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
#    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
#    ErrorDocument 410 /error/HTTP_GONE.html.var
#    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
#    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
#    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
#    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
#    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
#    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
#    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
#    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
#    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
#    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var

</IfModule>
</IfModule>

#
# The following directives modify normal HTTP response behavior to
# handle known problems with browser implementations.
#
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0

#
# The following directive disables redirects on non-GET requests for
# a directory that does not include the trailing slash.  This fixes a
# problem with Microsoft WebFolders which does not appropriately handle
# redirects for folders with DAV methods.
# Same deal with Apple's DAV filesystem and Gnome VFS support for DAV.
#
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-status>
#    SetHandler server-status
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
#</Location>

#
# Allow remote server configuration reports, with the URL of
#  http://servername/server-info (requires that mod_info.c be loaded).
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-info>
#    SetHandler server-info
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
#</Location>

#
# Proxy Server directives. Uncomment the following lines to
# enable the proxy server:
#
#<IfModule mod_proxy.c>
#ProxyRequests On
#
#<Proxy *>
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
#</Proxy>

#
# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
#
#ProxyVia On

#
# To enable a cache of proxied content, uncomment the following lines.
# See http://httpd.apache.org/docs/2.2/mod/mod_cache.html for more details.
#
#<IfModule mod_disk_cache.c>
#  CacheEnable disk /
#  CacheRoot "/var/cache/mod_proxy"
#</IfModule>
#

#</IfModule>
# End of proxy directives.

### Section 3: Virtual Hosts
#
# VirtualHost: If you want to maintain multiple domains/hostnames on your
# machine you can setup VirtualHost containers for them. Most configurations
# use only name-based virtual hosts so the server doesn't need to worry about
# IP addresses. This is indicated by the asterisks in the directives below.
#
# Please see the documentation at
# <URL:http://httpd.apache.org/docs/2.2/vhosts/>
# for further details before you try to setup virtual hosts.
#
# You may use the command line option '-S' to verify your virtual host
# configuration.

#
# Use name-based virtual hosting.
#
#NameVirtualHost *:80
#
# NOTE: NameVirtualHost cannot be used without a port specifier
# (e.g. :80) if mod_ssl is being used, due to the nature of the
# SSL protocol.
#

#
# VirtualHost example:
# Almost any Apache directive may go into a VirtualHost container.
# The first VirtualHost section is used for requests without a known
# server name.
#
#<VirtualHost *:80>
#    ServerAdmin webmaster@dummy-host.example.com
#    DocumentRoot /www/docs/dummy-host.example.com
#    ServerName dummy-host.example.com
#    ErrorLog logs/dummy-host.example.com-error_log
#    CustomLog logs/dummy-host.example.com-access_log common
#</VirtualHost>
GracefulShutDownTimeout 3
AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps


david59 02-14-2013 05:57 AM

1 Attachment(s)
Quote:

Originally Posted by unSpawn (Post 4888767)
The fact your server hasn't crashed since you made changes is hopeful but it does make me wonder what other changes you made that could have adversely affected performance. Maybe it would be best if you start back at the beginning and show us the machine hardware specs (you seem to have 8GB RAM?) and an inventory of installed software. (For example your CentOS 6.2 is one update behind and how about your web stack, Drupal and other software?) Collecting system statistics is a good move but you should monitor other aspects as well like network traffic, your web server and your database. For example Atsar works like "regular" SAR but can also log Apache HTTP traffic (or see "Apache Top"). Like Atsar Dstat aims to replace vmstat, iostat and ifstat but it can also log MySQL statistics (else see "MysQL Top" or "mtop" for short). As for your system logs I suggest you run them through Logwatch at detail level 10 as that's the easiest, quickest way to generate leads. I know this all wasn't very detailed but you have to start somewhere, right?

Thank you very much for your help, its greatly appreciated. Here is some information from the Plesk panel:
Code:

CPU        AuthenticAMD, Quad-Core AMD Opteron(tm) Processor 1356
Version        Parallels Plesk Panel v11.0.9_build110120608.16 os_CentOS 6
OS        CentOS 6.2 (Final)
Key number        PLSK.------08
System Uptime:        1 day(s) 02:36

Please see screenshot for additional hardware information.

I will look into Atsar today.

david59 02-14-2013 09:10 AM

Here is some data from sar February 10:

Code:

[root@u1---~]# sar -r -f /var/log/sa/sa10
Linux 2.6.32-220.13.1.el6.x86_64 (u----0.onlinehome-server.com)      02/10/2013      _x86_64_        (4 CPU)

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  %commit
12:10:01 AM  1355352  6671460    83.11    275256  4224112  1719772    17.22
12:20:01 AM  1347736  6679076    83.21    275572  4228152  1711808    17.14
12:30:01 AM  1341460  6685352    83.29    275848  4232332  1710256    17.13
12:40:01 AM  1331908  6694904    83.41    276132  4235856  1711264    17.14
12:50:01 AM  1311024  6715788    83.67    276444  4239544  1726340    17.29
01:00:01 AM  1319112  6707700    83.57    276752  4242968  1712136    17.14
01:10:02 AM    404016  7622796    94.97    267872  5071832  1859188    18.62
01:20:02 AM    157812  7869000    98.03    70812  6142068  1797364    18.00

01:30:01 AM  3843188  4183624    52.12    78208  2541604  1736676    17.39
01:40:01 AM  3799716  4227096    52.66    89472  2582732  1727676    17.30
01:50:01 AM  3777992  4248820    52.93    93720  2602416  1723836    17.26
02:00:01 AM  3770644  4256168    53.02    96628  2614208  1712204    17.14
02:10:01 AM  3739440  4287372    53.41    100468  2629368  1721892    17.24
02:20:01 AM  3728912  4297900    53.54    103408  2645892  1712244    17.15
02:30:01 AM  3711172  4315640    53.77    106672  2657860  1717508    17.20
02:40:01 AM  3692440  4334372    54.00    110528  2670808  1711708    17.14
02:50:01 AM  3653576  4373236    54.48    115876  2696428  1714036    17.16
03:00:01 AM  3630904  4395908    54.77    118956  2709512  1714192    17.16
03:10:01 AM  3562868  4463944    55.61    126248  2748860  1736020    17.38
03:20:01 AM  3560440  4466372    55.64    128756  2759064  1719104    17.21
03:30:01 AM  3264592  4762220    59.33    141432  2958644  1806476    18.09
03:40:01 AM  3675164  4351648    54.21    149324  2927480  1364612    13.66
03:50:01 AM  3589136  4437676    55.29    151316  2934020  1439068    14.41
04:00:01 AM  3560944  4465868    55.64    153400  2940336  1462156    14.64
04:10:01 AM  3551776  4475036    55.75    156332  2950064  1454236    14.56
04:20:01 AM  3530152  4496660    56.02    158316  2954984  1457280    14.59
04:30:01 AM  3521112  4505700    56.13    160168  2962020  1460100    14.62
04:40:01 AM  3500772  4526040    56.39    164784  2968140  1459692    14.62
04:50:01 AM  3427584  4599228    57.30    168308  2978500  1540716    15.43
05:00:01 AM  3392552  4634260    57.73    170328  2984684  1565084    15.67
05:10:01 AM  3360012  4666800    58.14    172180  2990216  1582184    15.84
05:20:01 AM  3304364  4722448    58.83    174572  3006868  1620124    16.22
05:30:01 AM  3290316  4736496    59.01    176312  3012164  1629360    16.32
05:40:01 AM  3220564  4806248    59.88    178136  3051616  1661504    16.64
05:50:01 AM  3184756  4842056    60.32    179844  3056732  1682020    16.84
06:00:01 AM  3156744  4870068    60.67    181204  3064084  1696792    16.99
06:10:01 AM  3128436  4898376    61.03    183944  3075556  1702904    17.05
06:20:01 AM  3119308  4907504    61.14    185652  3083372  1701632    17.04
06:30:01 AM  3107744  4919068    61.28    187728  3085156  1694128    16.96
06:40:01 AM  3090916  4935896    61.49    189328  3096928  1697192    16.99
06:50:01 AM  3080460  4946352    61.62    191344  3106652  1692056    16.94
07:00:01 AM  3042240  4984572    62.10    194320  3129888  1694044    16.96
07:10:01 AM  3027148  4999664    62.29    195992  3137500  1706144    17.08
07:20:01 AM  3016716  5010096    62.42    198344  3143596  1697572    17.00
07:30:01 AM  2999120  5027692    62.64    199680  3152252  1712080    17.14
07:40:01 AM  3001892  5024920    62.60    202036  3146604  1705244    17.08
07:50:01 AM  2990264  5036548    62.75    203392  3152332  1704264    17.07
08:00:01 AM  2963760  5063052    63.08    205508  3170972  1702976    17.05
08:10:01 AM  2944092  5082720    63.32    206924  3177500  1706508    17.09
08:20:01 AM  2871024  5155788    64.23    208444  3202492  1798712    18.01
08:30:01 AM  2923464  5103348    63.58    209640  3189632  1702928    17.05
08:40:01 AM  2922968  5103844    63.58    211056  3189380  1706804    17.09
08:50:01 AM  2895176  5131636    63.93    212908  3205000  1707892    17.10
09:00:01 AM  2718220  5308592    66.14    215476  3331544  1769788    17.72

09:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  %commit
09:10:01 AM  2703536  5323276    66.32    217188  3337680  1762408    17.65
09:20:01 AM  2687320  5339492    66.52    218372  3345516  1768020    17.70
09:30:01 AM  2656192  5370620    66.91    220000  3361068  1774852    17.77
09:40:01 AM  2645080  5381732    67.05    221308  3365868  1778960    17.81
09:50:01 AM  2632692  5394120    67.20    222236  3371552  1775268    17.78
10:00:01 AM  2621316  5405496    67.34    224096  3380220  1768512    17.71
10:10:01 AM  2612988  5413824    67.45    224612  3383848  1770452    17.73
10:20:01 AM  2609348  5417464    67.49    225248  3389648  1764540    17.67
10:30:01 AM  2590028  5436784    67.73    226444  3395036  1770260    17.73
10:40:01 AM  2587984  5438828    67.76    227268  3399672  1764616    17.67
10:50:01 AM  2570140  5456672    67.98    228540  3406116  1765792    17.68
11:00:01 AM  2564992  5461820    68.04    229468  3411000  1762976    17.65
11:10:01 AM  2557824  5468988    68.13    230272  3413452  1768084    17.70
11:20:01 AM  2548216  5478596    68.25    231020  3419136  1769060    17.71
11:30:01 AM  2536932  5489880    68.39    231868  3424408  1767416    17.70
11:40:01 AM  2525760  5501052    68.53    232544  3429520  1763992    17.66
11:50:01 AM  2521576  5505236    68.59    233176  3436768  1761152    17.63
12:00:01 PM  2501488  5525324    68.84    233884  3442604  1768340    17.71
12:10:01 PM  2494156  5532656    68.93    234904  3447436  1764524    17.67
12:20:01 PM  2480624  5546188    69.10    235452  3451656  1769124    17.71
12:30:01 PM  2472196  5554616    69.20    236004  3456060  1769872    17.72
12:40:01 PM  2471804  5555008    69.21    236852  3456048  1764764    17.67
12:50:01 PM  2466120  5560692    69.28    237428  3460652  1761680    17.64
01:00:01 PM  2450580  5576232    69.47    238048  3466396  1768612    17.71
01:10:01 PM  2446028  5580784    69.53    238620  3470496  1762696    17.65
01:20:01 PM  2434016  5592796    69.68    239112  3474836  1764768    17.67
01:30:01 PM  2427672  5599140    69.76    239852  3479176  1761856    17.64
01:40:01 PM  2417816  5608996    69.88    240608  3483760  1768596    17.71
01:50:02 PM  2412276  5614536    69.95    241420  3487372  1763716    17.66
02:00:01 PM  2397192  5629620    70.14    242356  3492316  1772396    17.75
02:10:01 PM  2387056  5639756    70.26    243260  3497020  1773776    17.76
02:20:01 PM  2381680  5645132    70.33    243960  3501776  1770424    17.73
02:30:01 PM  2366860  5659952    70.51    244724  3506576  1775684    17.78
02:40:01 PM  2362772  5664040    70.56    245216  3511376  1764500    17.67
02:50:01 PM  2341988  5684824    70.82    245728  3516120  1781236    17.84
03:00:01 PM  2347132  5679680    70.76    246508  3520716  1759672    17.62
03:10:01 PM  2292188  5734624    71.44    247332  3550832  1775412    17.78
03:20:01 PM  2281888  5744924    71.57    247948  3558844  1768952    17.71
03:30:01 PM  2265076  5761736    71.78    248624  3566580  1775800    17.78
03:40:01 PM  2249956  5776856    71.97    249148  3573008  1783488    17.86
03:50:01 PM  2248472  5778340    71.99    249432  3580124  1777600    17.80
04:00:01 PM  2237216  5789596    72.13    249936  3585092  1775428    17.78
04:10:01 PM  2233164  5793648    72.18    250412  3581576  1776808    17.79
04:20:01 PM  2221880  5804932    72.32    250776  3583920  1786356    17.89
04:30:01 PM  2165256  5861556    73.02    251212  3588744  1842136    18.45
04:40:01 PM  2163656  5863156    73.04    251256  3592916  1835720    18.38
04:50:01 PM  2154944  5871868    73.15    251420  3595868  1841024    18.43
05:00:01 PM  2132256  5894556    73.44    251684  3600836  1850128    18.53
05:10:01 PM  2136352  5890460    73.38    251652  3604784  1840676    18.43
05:20:01 PM  2122292  5904520    73.56    251172  3610748  1841752    18.44
05:30:01 PM  2110556  5916256    73.71    251776  3620240  1849768    18.52
05:40:01 PM  2101516  5925296    73.82    251928  3629396  1842596    18.45
05:50:01 PM  2090316  5936496    73.96    251644  3636564  1848136    18.51
06:00:01 PM  2082312  5944500    74.06    251020  3643380  1836352    18.39

06:00:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  %commit
06:10:01 PM  2073308  5953504    74.17    251272  3650736  1840564    18.43
06:20:01 PM  2062928  5963884    74.30    250968  3654212  1850704    18.53
06:30:01 PM  2081324  5945488    74.07    250600  3638536  1848952    18.51
06:40:01 PM  2074808  5952004    74.15    250864  3642408  1844880    18.47
06:50:01 PM  2076008  5950804    74.14    250860  3646716  1834824    18.37
07:00:01 PM  2072724  5954088    74.18    251140  3649280  1833124    18.36
07:10:01 PM  2061124  5965688    74.32    251432  3652968  1838912    18.41
07:20:01 PM  2054356  5972456    74.41    252032  3657716  1837228    18.40
07:30:01 PM  2332972  5693840    70.94    253212  3735924  1386448    13.88
07:40:01 PM  2412664  5614148    69.94    253668  3768904  1320880    13.23
07:50:01 PM  2254920  5771892    71.91    254252  3750792  1443728    14.46
08:00:01 PM  2249132  5777680    71.98    254688  3754384  1442460    14.44
08:10:01 PM  2238192  5788620    72.12    255060  3758456  1444344    14.46
08:20:01 PM  2227924  5798888    72.24    255324  3761072  1446840    14.49
08:30:01 PM  2214836  5811976    72.41    255756  3765396  1450280    14.52
08:40:01 PM  2172272  5854540    72.94    256064  3767708  1497164    14.99
08:50:01 PM  2152788  5874024    73.18    256456  3771520  1503912    15.06
09:00:01 PM  2143084  5883728    73.30    256800  3776716  1509960    15.12
09:10:01 PM  2126156  5900656    73.51    257260  3781652  1511972    15.14
09:20:01 PM  2123084  5903728    73.55    257640  3787212  1504120    15.06
09:30:01 PM  2113648  5913164    73.67    257712  3792372  1506464    15.08
09:40:01 PM  2096256  5930556    73.88    258100  3796992  1514600    15.17
09:50:01 PM  2035768  5991044    74.64    258324  3799560  1587936    15.90
10:00:01 PM  1987748  6039064    75.24    258600  3802084  1630436    16.33
10:10:02 PM  1985500  6041312    75.26    258928  3804376  1628204    16.30
10:20:01 PM  1979544  6047268    75.34    258564  3806912  1625192    16.27
10:30:01 PM  1968372  6058440    75.48    258656  3810292  1624444    16.27
10:40:01 PM  1965624  6061188    75.51    258564  3813140  1621616    16.24
10:50:01 PM  1957800  6069012    75.61    258536  3816576  1625916    16.28
11:00:01 PM  1943560  6083252    75.79    258748  3820976  1631844    16.34
11:10:01 PM  1655944  6370868    79.37    258684  4116996  1629572    16.32
11:20:01 PM  1637616  6389196    79.60    258372  4119348  1638280    16.40
11:30:01 PM  1651516  6375296    79.43    258472  4122088  1613292    16.15
11:40:01 PM  1646488  6380324    79.49    258496  4124164  1612164    16.14
11:50:01 PM  1630488  6396324    79.69    258620  4126660  1622980    16.25
Average:      2534226  5492586    68.43    219371  3458278  1698423    17.01


And from today, the 14th:

Code:

[root@u1------- ~]# sar -r -f /var/log/sa/sa14
Linux 2.6.32-220.13.1.el6.x86_64 (u-----0.onlinehome-server.com)      02/14/2013      _x86_64_        (4 CPU)

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  %commit
12:10:01 AM  1594672  6432140    80.13    277968  4145112  1556908    15.59
12:20:01 AM  1584708  6442104    80.26    278356  4150880  1555892    15.58
12:30:01 AM  1570940  6455872    80.43    278748  4155996  1557404    15.59
12:40:01 AM  1554844  6471968    80.63    279112  4160532  1565224    15.67
12:50:01 AM  1562848  6463964    80.53    279372  4146252  1568824    15.71
01:00:01 AM  1557728  6469084    80.59    279612  4151184  1563592    15.66
01:10:01 AM    403232  7623580    94.98    280784  5208152  1687732    16.90
01:20:02 AM    164332  7862480    97.95    90432  6211500  1670400    16.73
01:30:01 AM  3660020  4366792    54.40    95652  2780508  1618220    16.20
01:40:01 AM  3581848  4444964    55.38    110616  2850616  1609084    16.11
01:50:01 AM  3543580  4483232    55.85    116252  2888660  1604168    16.06
02:00:01 AM  3524248  4502564    56.09    119452  2904868  1600156    16.02
02:10:01 AM  3472640  4554172    56.74    123620  2936368  1613460    16.16
02:20:01 AM  3450224  4576588    57.02    126520  2950824  1610060    16.12
02:30:01 AM  3410036  4616776    57.52    131580  2973408  1617100    16.19
02:40:01 AM  3388092  4638720    57.79    134972  2987932  1618908    16.21
02:50:01 AM  3359628  4667184    58.14    137756  2999156  1625912    16.28
03:00:01 AM  3364560  4662252    58.08    139560  3007840  1606996    16.09
03:10:01 AM  3340080  4686732    58.39    141260  3016648  1615180    16.17
03:20:01 AM  3345592  4681220    58.32    142896  3013348  1603952    16.06
03:30:01 AM  3065380  4961432    61.81    159012  3138520  1718940    17.21
03:40:01 AM  3216224  4810588    59.93    164572  3079468  1607972    16.10
03:50:01 AM  3188628  4838184    60.28    166080  3101652  1609524    16.12
04:00:01 AM  3157756  4869056    60.66    167476  3107312  1626484    16.29
04:10:01 AM  3169492  4857320    60.51    169172  3113968  1597416    16.00
04:20:01 AM  3130592  4896220    61.00    170880  3120256  1624576    16.27
04:30:01 AM  3144708  4882104    60.82    172832  3113652  1607912    16.10
04:40:01 AM  3134328  4892484    60.95    174520  3119744  1611920    16.14
04:50:01 AM  3117476  4909336    61.16    180956  3127768  1603884    16.06
05:00:01 AM  3106036  4920776    61.30    182108  3134232  1603052    16.05
05:10:01 AM  3093600  4933212    61.46    184132  3140092  1601756    16.04
05:20:01 AM  3080180  4946632    61.63    185840  3146848  1596844    15.99
05:30:01 AM  3023392  5003420    62.33    187896  3152980  1657348    16.60
05:40:01 AM  3018708  5008104    62.39    189284  3158528  1649656    16.52
05:50:01 AM  3008228  5018584    62.52    190796  3164352  1648952    16.51
06:00:01 AM  2992060  5034752    62.72    192516  3171620  1651828    16.54
06:10:01 AM  2907268  5119544    63.78    197328  3230216  1730732    17.33
06:20:01 AM  2720644  5306168    66.11    200852  3261320  1886436    18.89
06:30:01 AM  2780516  5246296    65.36    202184  3267704  1797136    18.00
06:40:01 AM  2866036  5160776    64.29    203892  3242436  1680224    16.82
06:50:01 AM  2839328  5187484    64.63    205432  3249724  1696008    16.98
07:00:01 AM  2841548  5185264    64.60    206672  3255416  1677136    16.79
07:10:01 AM  2827284  5199528    64.78    207788  3260888  1683712    16.86
07:20:01 AM  2820428  5206384    64.86    208968  3266712  1674184    16.76
07:30:01 AM  2786656  5240156    65.28    210212  3275360  1698120    17.00
07:40:01 AM  2782924  5243888    65.33    211520  3284872  1688504    16.91
07:50:01 AM  2777596  5249216    65.40    212816  3291456  1676960    16.79
08:00:01 AM  2758964  5267848    65.63    213872  3297668  1677248    16.79
08:10:01 AM  2741436  5285376    65.85    215060  3304056  1683088    16.85
08:20:01 AM  2739440  5287372    65.87    215952  3310120  1680016    16.82
08:30:01 AM  2726984  5299828    66.03    216948  3318444  1672452    16.75
08:40:01 AM  2690168  5336644    66.49    218332  3325784  1692940    16.95
08:50:01 AM  2699796  5327016    66.37    219484  3333796  1667504    16.70
09:00:01 AM  2682808  5344004    66.58    220640  3340428  1667380    16.70

09:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  %commit
09:10:01 AM  2671976  5354836    66.71    221308  3345988  1667512    16.70
09:20:01 AM  2660992  5365820    66.85    221912  3351500  1666020    16.68
09:30:01 AM  2649912  5376900    66.99    222564  3357816  1665384    16.68
09:40:01 AM  2634464  5392348    67.18    223044  3363040  1670984    16.73
09:50:01 AM  2627580  5399232    67.26    223676  3370496  1662568    16.65
10:00:01 AM  2614684  5412128    67.43    224356  3376472  1663108    16.65
10:10:01 AM  2594936  5431876    67.67    224928  3382304  1678372    16.81
Average:      2779098  5247714    65.38    192661  3350734  1645754    16.48


unSpawn 02-14-2013 10:50 AM

Quote:

Originally Posted by david59 (Post 4891448)
OS CentOS 6.2 (Final)

It's one release behind as CentOS has been at 6.3 for some time now.


Quote:

Originally Posted by david59 (Post 4891448)
I will look into Atsar today.

Maybe I wasn't clear enough when I posted my reply but the general idea is to monitor as much aspects as possible and monitor them continuously.
The reason I explained those applications is so you have an idea which app can monitor what. So wrt choice think "and and" and not "or or".
The reason for continuously monitoring system, network, process and database stats is that over time it becomes easier to spot for example recurring events (say cron jobs), spikes you can trace back (say manually importing a humongous table), steadily increasing resource allocation you can correlate in other ways (advertising, new site launch, complete and utter misconfiguration, etc, etc.). So if you have baseline data, the machine being in rest without any activity, then over time it becomes easier to determine what the bottlenecks are and what can be attributed to usage or abuse.
So while posting SAR data (missing 3 days worth of data in between BTW) is neat, a single spike lasting approximately 40 minutes itself doesn't reveal much unless you correlate it with process and network resource usage at that time, amount and type of requests, etc, etc.

*BTW reading your httpd.conf my first questions would be 0) what changes exactly you made and 1) why would you be running Apache with about all modules including proxy_.* enabled?..

*BTW[1] If it is correct you host about 22 web sites and mail servers then I wonder at what stage you began experiencing problems. Right from the start or after you had x amount of sites running?

david59 02-14-2013 11:44 AM

OK, sorry, I misunderstood. I understand fully what you are talking about. I appreciate the advice.

Re: specific changes to the httpd.conf, I highlighted them in blue bold italics in post #20 above. I have since removed those items and the server is running fine. If, as I expect, bots return to my site, I will have to restore some sort of script to block them.


You ask above: why would you be running Apache with about all modules including proxy_.* enabled?..

The short answer is because I am an absolute beginner and if someone with expertise told me which modules I did not need to be activated, I would gladly comment them out. :)

Let me think about the answer to your second question about 22 sites. I will get back to you. many thanks.

unSpawn 02-14-2013 12:12 PM

Do you have a machine at home or work you could use? Doesn't have to be a dedicated one: you could install virtualization software like VMware, VirtualBox, QEmu or whatever you like. Point is once you have that you can install Linux. Then you have a machine you can test use for testing purposes w/o the fear of fscking up your production machine.

david59 02-15-2013 09:14 AM

Thanks, I have set up VirtualBox and am experimenting with it.

I am not running 22 sites per se, I have three drupal installations (will be removing one of these) and each installation runs a few sites off the same codebase.

Regarding the proxy-related Apache modules, I looked at what I have enabled:

Code:

proxy
proxy_ajp
proxy_balancer
proxy_connect
proxy_ftp
proxy_http

And one that is disabled:

Code:

proxy_scgi
My online help tells me that the following modules are always enabled for CentOS:

Code:

    env
    auth_digest
    authn_file
    authz_host
    authz_user
    actions
    alias
    autoindex
    dav
    dir
    log_config
    mime
    negotiation
    setenvif
    ssl

I am not at all sure that I need the proxy modules enabled. If I want to disable them, is there a particular order in which I must do so?

Thanks for your advice.

unSpawn 02-15-2013 10:17 AM

Quote:

Originally Posted by david59 (Post 4892295)
Thanks, I have set up VirtualBox and am experimenting with it.

The idea is that you load your production configuration (rsync?) so you can test modification before you apply them for real. Processes like that aren't meant to convolute your work flow but to ensure your production environment doesn't crap out over misconfiguration or botched upgrades. Don't see it as a hassle but as you protecting your investment.


Quote:

Originally Posted by david59 (Post 4892295)
I am not running 22 sites per se, I have three drupal installations (will be removing one of these) and each installation runs a few sites off the same codebase.

Yah, I should've said you've got x-something domains registered.
Are these Drupal installations all up to date?


Quote:

Originally Posted by david59 (Post 4892295)
I am not at all sure that I need the proxy modules enabled. If I want to disable them, is there a particular order in which I must do so?

No, not really.


Quote:

Originally Posted by david59 (Post 4892295)
Thanks for your advice.

That's OK, it's what we're here for. Now come on and supply the nfo that helps us help you!


All times are GMT -5. The time now is 03:44 PM.