Ubuntu: apache2 + mysql REAAALLLY SLOW even SSH access
 

Hello Gentlemen,

Problem is that:

I have a VM with Ubuntu 12.04, RAM 4GB, processor shares are higher than the other VMs.

I use this VM for only one website, "Based on Joomla"

When I issue the "top" command I see that CPU are 100% all the time, apache2 and mysql are consuming it.

Sometimes, and all of a sudden, the access to the website gets REALLY slow, but pinging to the server is normal.
Browsing to the hosted website is dead, even SSH to the server is very slow, While the CPU Consumption is nothing.

I checked the logs for the mysql and the apache but nothing shows there.

So what I do is that I restart the services and the website is fired up again, and CPU gets 100% again.

RAM is not using more than 2GB.

Can you please give me a clue on what is really going on here?

How can I improve the performance of this server?

Thank you in advance guys.

Ubuntu: apache2 + mysql REAAALLLY SLOW even SSH access
 

Hi Tarikc

Please check your Mysql and httpd configuration.

use mysqltuner to analyse mysql configuration.

Dear surya7,

Thank you very much for your reply.

Here is the output from mysqltuner:

Dear All,

I'm still facing the same problem yet again.

I've managed to get the attached image from the console.

Attachment 11272

Guys please I need your help with this.

Looking forward for any kind of help.

Best regards,

This is my syslog:

Hope to get some reply from you guys.

What to say? The VM ran out of memory and then out of swap at which point it started killing processes. 4 GB RAM "should be enough" to run Joomla so the question is what system and service configuration modifications you made (for example some people think that when Apache complains about memory they should multiply all configuration values w/o diagnosis as if that should help), what "one website, Based on Joomla" means (is it brand new, does it contain a gazillion badly written homebrewn scripts, is it a migration meaning, did it previously run on its own physical hardware, is it well-developed, etc, etc?), if system and daemon logs show errors / clues and what metrics you're going to watch using which tools (Apachetop, mtop aka MySQL top, Atop, Dstat, collectl, etc, etc)...

Hello unSpawn,
That is exactly what happened, the server just went into the same thing again, and I managed to check the free -m:

It is not a brand new website, it was hosted on another location and I backed it up there and migrate it here.

This is what I get from apachetop:
Can you please let me know what config I can do on apache to improve it?

Thank you a million in advance.

Do read me saying "the question is" as an invitation to be methodical about it and be verbose sharing information:
- Which system and service configuration modifications have you made?
- What were the HW specs of the old location?
- Did the web site exhibit problems there too?
- What is the Apache version? Is it up to date?
- Did you increase any values in your Apache configuration?
- What worker type is Apache running?
- What is the PHP version? Is it up to date?
- How do you use PHP? As FastCGI or mod_php?
- Did you increase any resource settings in your PHP configuration?
- What is the Joomla version? Is it up to date?
- Does the web site run any outdated / risky / unnecessary mambots / agents / extensions / components?
- Have you looked at system and daemon logs for errors and clues?
- Do you know what the standard requirements for Joomla are? In which way do the configuration files deviate from those standards?
- Which documentation do you use to troubleshoot the problem?
- What system counters are you going to watch over time to start diagnosing the problem?
- Which tools are you going to use?

Dear unSpawn,

I know you are going to hate me for this,
But I'm kinda newbie with this, I generally do not alter any config in apache, i only set virtual hosts on it.

- i made no system modification, i only changed my.cnf as suggested my the mysqltuner suggestions.
- I honestly don't know what was the older config, i got the backup files from the old host.
- The old website was rumning very slow on the old host, but i haven't heard of a problem on that old host.
- Apache version is 2.2.22, I only updated it yesterday.
- The worker is prefork
- php version 5.3.10
- I haven't made any modification to the php conf.
- I can't really tell if the website running any outdated version/risky components.
- Joomla is 2.5.6
- I tried to search for any clues and logs on the system, but couldn't find anything on neither apache, mysql, or syslog.

Just let me know if it is an apache configuration so i can dig in and try to find a solution and make some changes,

I'm sorry if I'm waisting your time, but I might make use of any kind of any kind of hints and clues.

Performance tuning is more than tweaking just one setting in each configuration file: there is no "magical" key to turn to make everything right and you will have to work hard to understand all the aspects involved. For example if you made no alterations then Apache will be running with all modules enabled and with default settings, PHP may be wasting memory on Innodb while it's not being used and the site may be rife with mambots or SEF rewrites that bug everything down. I'm going to ask you put in some effort and do your own research first: search the 'net and check the Apache, PHP, MySQL and Joomla web sites for performance-related recoomendations. Post what URI's you judge relevant here, along with the contents of your Apache, PHP and MySQL configuration files (strip out comments by running grep -v ^# /path/application.conf|grep .) and the sites settings you find in the Joomla administrator menu. With respect to the latter you should of course obfuscate any identifying information. Meanwhile run Logwatch on the available system and daemon logs, let ApacheTop and MySQLTop run as long as possible, run FPA (easier overview of Joomla administrator settings) and save all information for posting (text preferred over screen shots).

Dear unSpawn,

Again thank you very much for bearing with me, and thank you for your patience.

As instructed, I googled for tuning apache configuration, I have found as you said, that the apache is running all three modules, while it was compiled only with prefork module, so I have disabled the worker and the event modules, thinking of course that they might consume some memory.

This is the tutorial I followed for the apache tuning:

http://cloudinservice.com/tune-apach...refork-module/

As for the MySQL, I disabled InnoDB from the my.cnf, but the server did not start syslog says:
So I have re-enabled the InnoDB Engine.

This is the Forum I followed for the MySQL tuning:

http://forums.cpanel.net/f189/mysql-...lp-174272.html

As for the contents of the configuration files:

Apache2 configuration file:

php.ini <located in /etc/php5/apache2/>

my.cnf configuration file:

FPA Report file:

I hope I did not miss anything.

Again, thank you very much for your time.

Best Regards,

OK. Let's start with the web server first. The link you point to is good because that it explains the method behind deriving values based on your actual situation. However it doesn't explain all values (see for instance this or this) and the effect it has (or should have) on how you configure your my.cnf next. Also I don't know if you logged process and memory data before and after the changes or if you actually derived your posted values from it and if you looked at changes in resource usage. The thing is that adjusting (in small increments) by making changes and comparing data is the kind of process you need to go through: take the "original" httpd.conf and log process and memory usage and consider that your baseline data. Create a backup of httpd.conf, make changes, restart the web server, hit the server with traffic (there's tools for that), log process and memory usage. Then compare both process and memory usage sets. Rinse, repeat.

Also note your httpd.conf lists several includes. While they may not hold surprises it's good to actually inspect them and ensure they don't. In the meanwhile I do hope you still collect data with at least ApacheTop and MySQLTop because the longer the period the better averages you have to base decisions on. Also please check your FPA extensions output (well done for posting) against http://docs.joomla.org/Vulnerable_Extensions_List please and correct the listed access rights (777 is too much), see http://docs.joomla.org/Security_Chec...le_permissions for an explanation (do read the whole page though).


Do persevere! This may seem like small steps now but once you notice the effect it has you know the effort wasn't wasted.

Dear unSpawn,

I don't know how to thank you for your replies, and I can't say how much it makes me happy to know that I might be somehow on the right track to solve my problems,

I have changed more on the apache config, the site is a little bit getting improved, stalling a little but not freezing, memory consumption is much better, I might increase the limits on the prefork module a little bit until I get a satisfied results.

As for the apachetop, I am running it till now, but wanted to share this results with you, as honestly I searched for the tool to have something useful to learn about it, but didn't hit anything:

I will post the mytop results soon.

I really wish I will have to mark this problem as solved :).

Thank you a million times for your extraordinary help.

Best Regards,

This the continued apache top monitoring statistics:

I'm stopping it now coz I've read it consumes some memory.

But As I understand from what I see, it is http requests, 2xx OK, 4xx contains 404 not found.
And All: number of requests?

but what should I make use of it?

And what is R on the third row stands for?

Do you think that I may reduce the timeout to less time and increase MaxClients directive?
Coz I see from the top command, that apache processes are staying alive and dies after some time, at which time the website is a little bit stales.

But this is just me :), or you suggest to try it myself and check?

Best Regards,

Usually you'll want to measure, lower values, put the pressure on, measure again. Also consider the other processes you still need to look at before enthusiastically tuning things for "performance". Because what if new insights mean the values in php.ini and my.cnf actually need to be increased? Doing that and increasing httpd.conf values simultaneously may be actually bad for performance. That's why you need to log data.


What's the rush? And think of it as gaining knowledge instead.


No, you're running with default settings, should be OK.


The README for ApacheTop is pretty clear about things: how long it's been running, two lines accumulated data and two lines of stats in the last 30 seconds. Note nearly 25 per cent of your traffic isn't 2xx with over 20 per cent of redirects (usually 301: Moved Permanently / 302: Moved Temporarily). This may be due to .htaccess rewrites (or in httpd conf's we didn't see) or some SEO plug-in: best look at the 3xx messages in access_log.


Lower timeouts seems a good thing and I let the link I gave you before argue against increasing MaxClients.


That is the effect of the MPM you chose. Having children start costs resources too but OTOH having children die also cleans up memory, keepalives, etc, etc. Have a go at the official Apache MPM docs.


Observed how?


Heh. You catch on pretty quickly.

BTW one way to look at process details is to use 'top' (or atop, htop or mpstat) but if you need a quick way to isolate the current memory usage of say a process called "apache2" then you could also use 'ps': In the long run it's better to run any capable SAR like Atop, Dstat or collectl.