I'm not sure how to diagnose this problem and was hoping someone could point me in the right direction.

I have a website in code igniter that has hundreds of processes running and is tanking my cpu usage. If I restart mysql all the processes die and then CPU goes back to 2% usage, wait 3 days and it's back to 70%+ usage.

In htop I see hundreds of processes like this:
7377 username 15 0 98064 11604 5920 S 0.0 0.3 1:02.78 /usr/bin/php /home/username/public_html/index.php

All referencing the same file and same username.

pstack on any of them gives me the same thing:

#0 0x00000034d5ec5de0 in __read_nocancel () from /lib64/libc.so.6
#1 0x00000034daa86ee5 in vio_read_buff () from /usr/lib64/libmysqlclient.so.16
#2 0x00000034daa88129 in my_real_read () from /usr/lib64/libmysqlclient.so.16
#3 0x00000034daa88515 in my_net_read () from /usr/lib64/libmysqlclient.so.16
#4 0x00000034daa82152 in cli_safe_read () from /usr/lib64/libmysqlclient.so.16
#5 0x00000034daa83019 in cli_read_query_result () from /usr/lib64/libmysqlclient.so.16
#6 0x00000034daa818be in mysql_real_query () from /usr/lib64/libmysqlclient.so.16
#7 0x00000000005946f1 in php_mysql_do_query_general ()
#8 0x0000000000594b38 in php_mysql_do_query ()
#9 0x0000000000732709 in zend_do_fcall_common_helper_SPEC ()
#10 0x000000000073185e in execute ()
#11 0x00000000007321f8 in zend_do_fcall_common_helper_SPEC ()
#12 0x000000000073185e in execute ()
#13 0x000000000070044d in zend_call_function ()
#14 0x000000000064c3e7 in zif_call_user_func_array ()
#15 0x0000000000732709 in zend_do_fcall_common_helper_SPEC ()
#16 0x000000000073185e in execute ()
#17 0x000000000074fd42 in ZEND_INCLUDE_OR_EVAL_SPEC_TMP_HANDLER ()
#18 0x000000000073185e in execute ()
#19 0x000000000070af69 in zend_execute_scripts ()
#20 0x00000000006b7b78 in php_execute_script ()
#21 0x000000000079476f in main ()

I'm running WHM / CPanel and I tried looking at the apache logs, but I don't see anything significant.

grep 'username' error_log | less

The site is running code igniter and there's no special setup or anything, and it's not showing any PHP errors.

Outputs nothing new in over months. What would be the next steps in troubleshooting something like this?

I may be wrong, but it looks like PHP is being invoked through CGI, which means that each HTTP request spawns a new instance of PHP. Having Apache use mod_php would be far less resource-intensive.

Make sure httpd.conf contains a LoadModule directive for php5_module (commonly implemented by including a separate file, "mod_php.conf", containing the LoadModule command and several other PHP-related settings).

I see what you're saying, but I believe the problem is not so much that it spawns a different instance of PHP, but that the instance of PHP never dies on its own.

To be more specific, what I'd like to know is why it's getting stuck and hanging? Since I can't find where PHP is throwing any errors, what am I missing to further debug?

Well, as the bt tells it gets stuck in a read operation of a mysql query.
Now this may happen but there also should be a timeout for this kind of operations.
Maybe you should have a look at the code or just low down your user resources so he will fix his code.