Rogue stuff started by Apache
 

I'm getting rather frequent exploits on my server resulting in crap being put into the /tmp dir and executed.

Can I flag /tmp not executable but it seems that MySQL is using the dir with x files.

Or can I secure the apache user better maybe? I'm also having some difficulty in finding which site is vulnerable.

Any advice would be appreciated.

Sounds like you have been penetrated (rooted -use rootkit) as no one should be able to put anything on your server. The normal post intrusion procedure applies - disconnect the server, take an image of the disk(s) and do a fresh install using the latest software and the configure new complex passwords and look at your firewall.

You may want to take a close look at how Apache was being abused, so that you don't simply put the same content on your site after rebuilding the box. Make sure to check that Apache/PHP/mySQL/BulletinBoard software were updated versions. You can also check for vulnerable content (like poorly writtten PHP scripts) with nikto. Awstats is also commonly abused and may be the problem if it's installed.

A good place to start looking is your Apache logs, especially for any errors or URLs that contain any of the file names you found in /tmp