I had a server that ran fine for years. But after upgrading to the latest version of CentOS and httpd, my server keeps on locking up when there are too many connections (under a nikto security scan for example).

I found lines like this in the message log
kernel: [<c0459c79>] out_of_memory+0x72/0x1a5 httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
kernel: [<c0459c79>] out_of_memory+0x72/0x1a5
...
kernel: Total swap = 2096440kB
kernel: Free swap: 0kB

"top" would show about a bunch of httpd processes taking up 5, 10% of MEM. The only non-default setups are
post_max_size = 200M
upload_max_filesize = 200M
max_execution_time = 72000
max_input_time = 72000
in php.ini

I tried very small value for ServerLimit and MaxClients (10, 15) and these seems to prevent the server from hanging, but still the MEM% went to dangerously value (> 5%).

My provider blames it on a mem leak in http/mod_ssl, but I'm not so sure. Any suggestions? Thank you.

access.log file size also effect the usage of total httpd memory usage. so just verify the access.log size

On our test server, log file size is only 5+ MBytes. And it runs into the same problems as the production servers (300+ MBytes).

Here is an errata on the problem:
https://rhn.redhat.com/errata/RHSA-2009-1075.html

But the update does not fix the problem!