Here are the software I have used:
Postfix (
http://www.postfix.org/)
--------
Postfix is much better than Sendmail as far as the security is concerned and easy to configure. Get postfix version 2 or grater.
dovecot (
http://www.dovecot.org/)
----------
Dovecot is a light weight IMAP & POP3 server (also imapS & pop3S) and it has great security features . It is easy to set up and doesn't require special maintenance. Only thing you need is to get the authentication working properly - if your users are in /etc/passwd there's hardly anything you have to do. You can also set up sasldb2 database and authenticate your users against it. Dovecot should be pretty fast. There are still some optimizations that could be done, but I believe it already beats most other IMAP servers in overall performance.
Mailman (
http://www.gnu.org/software/mailman/)
----------
Use it only if you needed.
SquirrelMail (
http://www.squirrelmail.org/)
--------------
Webmail software.
Spamassassin (
http://spamassassin.apache.org/)
------------------
Spam filter software.
Openssl (
http://www.openssl.org/)
----------
Get all these installed on your machine. Here are the ports you need to open using the firewall (iptable):
SMTP - 25
SMTPS - 465
HTTPS - 443
IMAPS - 993
SPAMD - 783
Modify /usr/lib/sasl2/smtpd.conf (choose your favorite authentication: auxprop or saslauthd)
Create SSL certificates for IMAP & SMTP (both private and certs keys)
Create SSL certificates for apache web (both ssl.key & ssl.crt)
Turn on all the services. Check your services using "lsof -i' & and check their ports using "netstat -an".
If you are using RH version 4, you can pretty much use the software that came with the system (unless you want the latest & gratest).
Hope it helps.