The major difference between all of these things is how they store mail, and how they are configured.
The 3 MTA's I am most familure with are postfix, qmail, and sendmail. Here is a brief description:
sendmail: very basic, comes with most systems. Fairly easy to configure as it doesn't support many options.
postfix: very powerful. Easy to configure and can be made to support many cool options such as Maildir (better way to store mail then mbox) and stuff like that. Many mission critical mail servers use this software. This makes it very easy to not run an open-relay (please don't run an open relay).
qmail: The source code hasn't changed since 1998. The reason is it just plain works, and it has never been broken into. Problem is the only way to get some newer features is patching the source, as DJB (The author) didn't GPL the software. That said, once it is patched and running it is easily on par wiht postfix in features, speed, and ease of use. Problem is the initial setup. As it was written by a security expert (
http://cr.yp.to ) you can be sure it is secure. Qmail also uses Maildir's and regardless of what you use as an MTA I recommend you look into Maildir's. Read the README.maildir file in the qmail source to see DJB's explination as to why.
As was said before, the MTA handles mail delivery. Its job is to accept mail through the SMTP port and deliver it to the local machine. It is also used to deliver mail from the local machine to other SMTP servers. If setup correctly an MTA will only deliver mail to remote accounts if a) the sender has gone through an authentication process, or b) the sender is sending from the local machine. Anything else is considered an open relay, which helps spammers deliver spam without easily being catchable.
As far as IMAP goes, that is a protocol you use to connect to read your mail. The two servers I hear of the most are uw-imap and courier-imap.
The only one I have ever used was courier-imap. The reason for this is most large system admins I have talked to swear by it and it fully supports Maildirs with no patches or trouble. Supposivly it is the ONLY imap server with out of the box Maildir support.
If you decide to go the qmail route I can help as far as pointing out good patches and tell you how to install the qmail-scanner, which virus checks all e-mails that pass through the system (both directions). I can also help out with any courier-imap problems you might have.