my vote would actually go to Slackware. I can't say anything for or against Debian, since I havn't used it, but Slackware can be very
minimalistic, and you definately don't need to install more than what's necessary.
For mail server, I would go to http://www.linux.org/apps/all/Daemons/Mail.html
and check out what's listed there, and then compare the products. But like I said, you should look at slack as a possible alternative to debian. The servers and apps in slack are top-notch and are easy to set up.