I'm looking to setup a server just for email. It is going to need to handle about 3000 email accounts, from about 500 different domains.

The domains are on a shared server now, but the server is getting overloaded and filled with spam.

Is CentOS / Postfix / Dovecot / Clam a good setup? What about qmail?

If I can get a Barracuda firewall used, will it work, or do I have to pay for updates (I'm hoping someone knows, because Barracuda's site only wants you to contact sales)

You didn't mention using spamassassain. Most folks that do their own use that to mitigate spam.

We use spamassassain on the server we have now, and it seems to do a pretty bad job. Maybe we don't have it configured right?

I'd love to use what gmail uses - that thing is accurate as shit - though to be fair, a lot of that comes from me marking / unmarking emails as spam.

Configuring SpamAssassin is going to depend on the nature of your email addresses. If you have a bunch of wallflowers who don't share their email address with everyone, then you don't have to be very tight about it. If all your users spam their email all over their websites on myspace then you are screwed. It is highly adjustable.

I set up a mail stack on my web server a few years ago and it's running CentOS 5, Postfix, Dovecot, AmavisD, ClamAV, and SpamAssassin. I recall that it was pretty confusing but I found this tutorial to be at least informative (although sadly not complete).

The basic idea is that mail arrives on your server and postfix (I think its postfix) puts the mail somewhere such that AmavisD (a high-performance daemon to shuttle stuff between Mail Transfer Agent and content checkers) will check each incoming message with both SpamAssassin and ClamAV before they mail gets put into people's mailboxes. Once the mail has been delivered to a mailbox, Dovecot handles POP and IMAP requests from your users.

Another option that just occurred to me is using google apps for mail. I've seen it work really really well. It's availability is excellent, you only see ads if you check using the gmail interface they offer, and you can set up pretty much any device to check your mail via IMAP. They have mail filters, all kinds of flexibility, encrypted mail transfer, and best-in-class spam filtering. The only reason not to use it is privacy really.

CentOS, Postfix and Dovecot sound very sensible. I haven't used Clam, but it should be a good choice too. Qmail is efficient, but Postfix is easier to manage.

I must recommend using the Maildir format for mailboxes, with local filesystems optimized for many small files; ext4 should do for < 16TB storage. Postfix and Dovecot have full support for Maildir. With 3000 accounts, even with spam filtering, CPU power is not an issue, but disk access is usually a bottleneck. Because each message must be written to disk, they cannot be safely cached in RAM. Thus, you don't need that much of RAM either, something like 4 to 8 GB should be enough.

If you can, I'd also recommend getting two identical physical servers at once, connected via a direct gigabit ethernet. This way you can do updates and upgrades with minimum service interruptions, and you have a backup machine when hardware failure occurs. Basically, you set up the slave first, then migrate the service to the other machine. It is also possible to keep the data on the slave machine constantly up to date, but that generates a lot of network traffic, and is a bit of a pain to set up correctly. I'd still use either software RAID0 or RAID10, or hardware RAID 5 or RAID 6 over a number of SATA disks, locally on each server, plus at least a daily backup to remote storage. RAID does not a backup make.

I don't really want to recommend any products, but if you have a rack, or can rent suitable rack space, I'd look at the 2U servers with support for 4 to 8 hotswap 3.5" SATA drives, and buy either Samsung (cheaper) or Western Digital (long warranty) drives separately. You should be able to get a suitable rack server for 1500 € to 3000 €, depending on your needs, location and sales contacts. Make sure you have an IP-based KVM module on the server, it makes troubleshooting remote hardware problems much easier. If you house the servers in an office-like environment without a rack, you can build a reliable, efficient and quiet real server machine for about the same price, if you are careful. If you choose the parts carefully, you can get an average five year lifespan. Just be always prepared for at least one hard drive failure, having a couple of spare drives always at hand is important. Buying two servers should also get you (a small) discount.

Buying complete (or at least partially configured) servers is a good idea, but knowing the hardware components makes it easier to decide which one suits your needs best. If you want to delve into the details, contact me via e-mail; I've built Linux servers and workstations as a hobby for over a decade now.

Nominal Animal