The default mail server for RedHat is going to be Sendmail, which has always had a rash of security holes and is rather cantankerous to configure, but it is the UNIX industry standard SMTP server. If you want to run with it, just poke through the how-tos over at linuxdoc.org and find one that covers the current sendmail series.
A better alternative is Qmail, you could get the RPM from www.rpmfind.net
and read up on it at www.qmail.org.
Its easy to configure relaying (open is off by default so no one will be using you to relay spam), and its easy to black hole incoming spam. Look for the link on the Qmail page called "Life with Qmail", which will walk you through just about any aspect of the configuration. Qmail also comes with its own pop3 server. For imap, there is a imapd which can be made to work with Qmail.
Or... There's the Courier MTA, which will do all 3, plus webmail! I got this to compile without much hastle, although configuring the webmail front-end is a bit of a juggle and relies heavily on CSS which some older browsers (like all of the 4.x Netscapes), choke on. Its still usable under them, just ugly as sin.
To answer your questions in order, assuming you just want to stick with Sendmail:
1) sendmail is hard NOT to install. Basically to find out real fast if its listening:
telnet yourmachinesurl.com 25
If you get anything other than "service denied" or whatever, its up.
2) /etc/mail and read up on "man aliases" "man sendmail" etc.
3) Sendmail, by O'Reilly and Associates. Be sure to pick it up with both hands, the 1000 pages or so has been known to pull a wrist tendon free.