This is the situation...
I have a mail server that accepts to relay from system in the trusted network. One of the systems in the trusted network is a webserver. On the webserver there are several scripts that send email.

Let's focus on the PHP scripts. These use the mail() function for that.

I am looking for means to reduce the potential abuse of the mail server when one of the PHP scripts is hacked.

For the situation that the code is modified by a hacker or a new script is installed I would like to take this approach:
1) scan the system for scripts using the mail() function
2) generate a checksum list from these scripts
3) intercept email (being sent to sendmail) from these scripts
4) check if they match the checksum list

In theory (I will still have to implement it) this would take care of the situation in which new/modified scripts try to send email.

However, there is also the option of an exploit of some script. Are there any ideas on means to prevent email abuse for this situation (other that: make sure scripts cannot be exploited )?

Best to take steps in locking the script down first.
mod_security or mod_selinux would help here.

If after this, you were compromised, then this would help here..

http://www.snertsoft.com/sendmail/milter-limit/

Whilst this would not stop messages, it can certainly limit them.

Somehow from this I get the idea those PHP mail scripts aren't really "trusted". If they are not then
0) shouldn't the server reside in a DMZ instead of the "trusted" part of the network? Or
1) shouldn't you only use scripts that have been examined for these?:
http://cwe.mitre.org/top25/archive/2...sans_top25.pdf
http://www.owasp.org/index.php/Secure_Coding_Principles
http://owasptop10.googlecode.com/fil...20-%202010.pdf
http://www.devshed.com/index2.php?op...ge=0&hide_js=1
http://www.devshed.com/index2.php?op...ge=0&hide_js=1


With #1, #2 and $4 you're basically describing a HIDS like Samhain, Aide, Afick, md5deep / sha1deep or even tripwire. However the point is that prevention is better than curing. For this centosboy is right that "defending the web stack" with mod_security and (not or) mod_selinux makes sense, but it's not all:
0) placement is everything. If a web server can not be trusted it should not reside in a trusted part of a network,
1) harden the machine before exposing it to a (any) network. Review if running Hardened PHP or the Banshee secure PHP website framework, mod_security, a reverse proxy, an IDS, GreenSQL could provide additional security.
2) review what is run. If the machine runs a web-based email package, CMS, board software, shopping cart (or whatever else SW on top of a database with Perl, PHP, Python, Ruby or whatever else interpreted language) that is home-brewn then think hard about substituting it for a current version of a maintained, supported equivalent package. If you already run the current version of maintained, supported software then ensure you update when security fixes are rolled out by the vendor. If running home-brewn SW is a given, then ensure you have sufficient (beg, steal, borrow) access to resources for code audits, security fixes and maintenance updates. In all cases familiarize yourself with the documentation before installing and remove or restrict access to those parts of the installation that should not be publicly accessible (say /phpmyadmin or wp/admin/).
3) review who has (public, maintenance) access to what. If you have only a few "trusted" users updating components then that makes for an easier case then say in the case of web hosting where all sorts of more or less knowledgeable "users" update stuff "just because they can". Enforce good passwords on user accounts and chage them. Restrict user access to only those parts they require access to. Use ACLs.
4) log about everything and enable (and do read) reporting of security problems, traffic statistics and errors. Most attacks are noisy and knowing about attacks enables you to take action quickly.

That's about it I think.