I have problems with a production system using a LAMP distribution.

The clients connect to the front load balancer LVS with 3 to 5 apache servers backside. Each request is served by php scripts (web services) that connect to the mysql database server. My clients upload analog signal samples files of about 100k each at a maximum rate of about 180 per sec and extract them later at about the same rate.

With a hight workload, with more clients the entire system can have bad response times (eg more than a sec) and sometimes the load balancer or the apache servers freezes themselves. Sometimes the database claims it has too many connections.

Have you ideas of where I must direct my searches ?

What type of load balancer do you have? Also, what switches are between your apache backend servers and your db server?

Thanks for being interested in my solution.

The load balancer uses the layer 4 mode, it owns a VIP for client connections.
It's a software load balancer IPVS on a linux box (ipvsadm ).

All the servers are in the same LAN ( LVS, Apache, Mysql), connected to "allied telesys" Gigabit/s stackable switches with 2 or 3 stacks maximum.

looks like it's time to start a sniff. Several ways - if you get lucky and are sniffing your session and get the hang, can you post the tcp dump around the hang and maybe something will stick out? Unless someone else has an idea, but that's where i would go...

The entire system is in a managed hosting service and I am not allowed to make tcpdumps on it.

From the client side we can note that the communication could'nt be established on the requests when the system begin to freeze, yet I can access with a SSH connection to each element of the system, it's like the apache servers refuse to accept new requests.

The hosting service staff made some basic tcpdump not showing infrastructure problems (with our heavey traffic they can't record all packets with tcpdump). They also look at their switches monitors, all parameters seems ok, and don't show any bottlenecks.

Since the last post I have tried to analyse the database activity before the problems and the response times to queries tend to increase.

Also I can see "Too many connections" messages in the syslog of the database coming from memcache.

whats the limit on your connections?

is it reasonable to increase the maximum number of connections perhaps because the service you are requiring demands more connections?

Login to DB as root and SHOW VARIABLES http://dev.mysql.com/doc/refman/5.0/...variables.html looking for max_connections http://dev.mysql.com/doc/refman/5.0/...ax_connections

the max_connections is 5000.

following, I have post a few other ones.

+----------------------------+----------------------+
| Variable_name | Value |
+----------------------------+----------------------+
| ft_max_word_len | 84 |
| group_concat_max_len | 1024 |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| max_allowed_packet | 10485760 |
| max_binlog_cache_size | 18446744073709551615 |
| max_binlog_size | 1073741824 |
| max_connect_errors | 10 |
| max_connections | 5000 |
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 16777216 |
| max_insert_delayed_threads | 20 |
| max_join_size | 18446744073709551615 |
| max_length_for_sort_data | 1024 |
| max_prepared_stmt_count | 16382 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 18446744073709551615 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 18446744073709551615 |
| myisam_max_sort_file_size | 9223372036854775807 |
| sql_max_join_size | 18446744073709551615 |
+----------------------------+----------------------+
+---------------------------------+----------------------+
| Variable_name | Value |
+---------------------------------+----------------------+
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| delayed_queue_size | 1000 |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_buffer_pool_size | 8388608 |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 268435456 |
| join_buffer_size | 131072 |
| key_buffer_size | 402653184 |
| key_cache_block_size | 1024 |
| large_page_size | 0 |
| max_binlog_cache_size | 18446744073709551615 |
| max_binlog_size | 1073741824 |
| max_heap_table_size | 16777216 |
| max_join_size | 18446744073709551615 |
| max_relay_log_size | 0 |
| myisam_data_pointer_size | 6 |
| myisam_max_sort_file_size | 9223372036854775807 |
| myisam_sort_buffer_size | 67108864 |
| preload_buffer_size | 32768 |
| query_alloc_block_size | 8192 |
| query_cache_size | 33554432 |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 4096 |
| read_buffer_size | 2097152 |
| read_rnd_buffer_size | 8388608 |
| sort_buffer_size | 2097152 |
| sql_max_join_size | 18446744073709551615 |
| thread_cache_size | 8 |
| tmp_table_size | 16777216 |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
+---------------------------------+----------------------+
+------------------------------+----------------------+
| Variable_name | Value |
+------------------------------+----------------------+
| binlog_cache_size | 32768 |
| have_query_cache | YES |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| max_binlog_cache_size | 18446744073709551615 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 33554432 |
| query_cache_type | DEMAND |
| query_cache_wlock_invalidate | OFF |
| table_definition_cache | 256 |
| table_open_cache | 512 |
| thread_cache_size | 8 |
+------------------------------+----------------------+

i would bump max connections up to about 7,500 and see if that helps.