We have a box that contains user-installed scripts (php/perl/mysql - same as normal share hosting environment), we suspect it's sending out spam (email admin already put a cap on this server). However maillog only shows user apache so we suspect it's one of the web applications/scripts has been compromised. We only inherited this box recently (I really want to get rid of it...). Normally in our environment it's one web application per box and mostly for internal use only therefore figure out which script is sending out email is simple.

How do I figure out which script/webapp (running as apache) is sending out email?

You can use the lsof command to list all files opened by the apache user.

lsof -i -u apache

will show you all files opened by the apache user that are currently accessing the network. You can do a lot more with the lsof tool. For more granular information on its options, I would suggest taking a look at the lsof man page.

I ban "apache" and/or "nobody" sent emails at the door with an appropriate bounce message telling why it's being refused. 98% percent of the time it's this exact case here. If you run a mailserver, you might want to consider this case-in-point.