thnx guys!

Well, I am glad others have found this useful. Wow, it has revolutionized spam control for me. Thanks for the dnsbl.sorbs.net "all in one" solution Donboy. I have adopted it in addition to spamhaus.

Slightly off topic ...
Today I was testing something on my main email server. I'll spare you the details. I relayed a test email from one server bouncing off of my main server (I had temporarily allowed a relay from one remote ip address). The final destination was a rogers.com address of mine (Rogers is a Canadian ISP that is somehow associated with Yahoo). Yahoo SpamGuard must have seen my server as an open relay (I think). It is not a relay, but my server is now blacklisted by Yahoo. Arrrghhhh! I have asked Yahoo to remove me. However I suspect they are an unresponsive corporate behemoth. Be careful out there. Innocent actions can get you blacklisted.

Apollo

On running Sorbs or other blacklists locally ...

One thing that bothers me about the whole rblsmtpd setup is that every email arriving at my server causes a hit on Spamhaus and/or Sorbs (or wherever). So, I decided to run a local whitelist also. The list contains a few ip addresses that cover about 75% of the email arriving at my server. Whitelists use the -a option (instead of -r) and you place them ahead of the -r blacklists. If you get a whitelist hit, the blacklists are skipped. Anyways, early on in researching how to set up a whitelist, I realized I was also learning how to setup a local blacklist. Many of the lists allow you to rsync their entire database hourly or daily or whatever. So, I am now running dnsbl.sorbs.net locally instead of hitting sorbs every time an email arrives. Rsync is an extremely efficient way of keeping the local blacklist in sync with the main Sorbs list. It downloads only the delta (differences) since your last sync. I use a cron job that runs every 12 hours. They seem happy to let you rsync hourly if you want. It's great.

I am using a setup that combines BIND 9 with rbldnsd. My MTA is qmail. I'm not going to detail exactly how to do this (unless someone has specific questions). There is adequate documentation out there. Start by getting rbldnsd (not to be confused with rbldns) and looking here:

http://njabl.org/rsync.html and here http://njabl.org and do some googling. It's a fairly involved setup, but not too bad. It went smoothly for me.

This is a beautiful setup once you get it working. It uses a lot of RAM -- currently my local dnsbl.sorbs.net is using ~50MB of RAM. No big deal. Performance has improved because each and every email does not result in a hit on the real Sorbs (I hit my local Sorbs copy instead). Also, in the event that Sorbs becomes temporarily unavailable for rsync-ing ... well, worst case -- I'm temporarily using a slightly old local Sorbs list which is probably still quite effective. (until I can rsync again).

One final interesting observation:
In my logs I have recently noticed flurries of emails from dynamic ip addresses (all blocked with rblsmtpd) around certain times (usually on some hour). These flurries typically last 2-3 minutes. I assume they are from windoze PCs infected with viruses that activate at certain times. My server has no problem handling the volumes, but these events must load services like Spamhaus and Sorbs significantly. By running Sorbs locally, I don't contribute to that.

thnx u for sharing your knowladge with us.

i wonder if it is possbile to check remote reverse dns record before accepting the mails which come to our domains. so most of infected, spam mails that come with fake domain address, will be blocked. and using local nameserver wont eat bandwith much to check reverse record. and also we will save the cpu and ram usage.

thank u.

Yes, I think it is possible. I assume you are using qmail. See tcpcerver man page:

http://www.die.net/doc/linux/man/man1/tcpserver.1.html

Look at the -p option (paranoid). I think this does a reverse DNS lookup, then checks that the host returned resolves to the original IP address. However, I'm not sure what happens then. Does an offending email get blocked or is it just logged? I don't know.

There might be a performance cost to this, but maybe worth it.

I have thought about implementing this, but have not researched it yet. I will report any findings here if/when I get around to it. In the meantime, let me know if you discover anything about this.

Apollo

Here's a brief discussion of this:

http://www.webmasterworld.com/forum40/521.htm

My server is configured correctly for reverse DNS, but at one time it was not. I think correctly configured reverse DNS is a good policy. Unfortunately, many small time servers don't have control of the PTR record so you'd be shutting them out.

I may turn tcpserver paranoid on just to see what happens.

Something tells me tcpserver -p is just for logging purposes. I could be wrong. Here's a patch for tcpserver that apparently does reject email from ip addresses with incorrectly configured reverse DNS:

http://www.gentei.org/~yuuji/software/qmpatch/

(look for "paranoid check" section about half way down the page)

Also look for the "mfcheck" patch. This does reverse lookup on the "mail from" header. If somebody sent you a message and the from address is "sombody@nowhere.com" and "nowhere.com" is not a real domain, mfcheck will reject the connection with a 5.x.x. error. This sounds the most like what you're looking for. Also remember to create the control file "mfcheck" with the contents being "1" which activates the feature. Of course you need to do this after applying the patch.

thank both of u so much.
i am starting to browse the links and i will do some tests such as applying patches that u suggest, on test box. i will post the results here. btw it can take long while. i will do that in free time.

thnx again.

here is an example about tcpserver with -p option. there are some other explanations to block spammers too.
http://www.chrishardie.com/tech/qmai...ysadminoptions

i tried it on test box, and couldnt connect via smtp from a box has private ip, i got "451 bad reverse DNS" error as it must be.
but i must allow some internet clients to send mails. they have dynamic ips. so i cannot add them relay list.

Donboy's suggestion "mfcheck" can be better solution for me. i will try it soon.
the following can be a good solution too:
http://www.fehcom.de/qmail/spamcontrol.html

i will post the results.

thank both of you again.

Here's something I just implemented that is very interesting. Linuxmagic distrubutes an opensource drop-in replacement for qmail-smtpd (called magic-smtpd).

http://www.linuxmagic.com/opensource...l/magic-smtpd/

The install is pretty easy.

By default it is configured to function exactly the same as the original Qmail qmail-smtpd. However, you can configure magic-smtpd to do some very useful things. Here's an excerpt from their install manual that sums it up:
-------------
The main advantages of using magic-smtpd as a replacement for the SMTP daemon that comes with qmail (qmail-smtpd), are that magic-smtpd has:
1. The ability to check the validity of users before accepting mail destined to them, in order to reduce the number of bounced messages.
2. The ability to do SMTP authentication to allow legitimate users with dynamic addresses the ability to relay through your server.
3. The ability to limit the rate/amount of email coming in on a given SMTP session in order to discourage spamming and reduce server resource usage.
4. The ability to do basic spam checking at the SMTP level.
5. Support for encrypted sessions using TLS extensions.
------------
The main thing that I like is I can reject email for non-existent users at the SMTP level. That means no more bounces and subsequent double bounces generated by spammers "fishing" for addresses at my domains. These are rejected right up front -- end of story. A great way to lessen server load if you get lots of spam requiring your server to respond with a bounce (and then some other unlucky innocent server to bounce your bounce).

The same thing can be done with spam checking at the SMTP level, although I have not yet configured this. Magic-smtpd can be configured to check FROM addresses to see if the domain is valid (checks if A and/or MX records even exist) or to check PTR record (reverse DNS) or even to check user-level spam preferences. Then the email can be blocked while your server is still communicating with the sending server -- no bounces required because the bad emails don't even get initially accepted.

If you refer back to my very first post in this thread, I have now almost fully implemented my initial wish to stop all kinds of spam right at the front door. Anything that gets past the blacklists now must get past magic-smtpd too. Almost no bad email gets through now, but I can see it in the logs. False positives? I guess my volumes are relatively low (growing daily), but I have yet to learn of a single false positive.

Apollo

Another update on my progress ...

I have added both virus scanning (clamav) and Spamassassin filters after magic-smtpd accepts the mail. So, now my server consists of these levels of filtering in this order:

1. rblsmtpd (ip address check against blacklists)
2. magic-smtpd (checks that TO user exists)
3. clamav (antivirus)
4. Spamassassin (spam filter based on scoring system)

I have Spamassassin set so that any spam that reaches it (a rare event) gets flagged as "PROBABLE SPAM" in the Subject line and is then delivered. The very first step (rblsmtpd) blocks mabye 95%+ of the spam and virus email (the latter almost always from dynamic ip addresses). Step 2 (magic-smtpd) rejects almost all of the remaining spam, usually addressed to non-existent users, thereby avoiding a a large number of bounces and double-bounces. Only trace amounts of spam even get to clamav and spamassassin.

As remarkable as this sounds, since implementing this latest setup I have not had a single report of spam getting through unnoticed (that is, not flagged as spam by Spamassassin). I also do not have a single report of legitimate email being blocked. I'm sure both of these events will eventually happen, but they are clearly rare events.

One final observation: I have recently noticed a dramatic drop (~80% decrease) in the number of positive hits to the blacklists (sorbs and spamhaus) even as the number of users and domains I host has increased. I attribute this to rejecting non-existent users up-front (ie. at the smtp level with magic-smtpd). Maybe I am wrong, but I think by not initially accepting mail for non-existent users (then bouncing it), my domains have been dropped by some spammers.

Ok, so is there a large cost to all this filtering? Not really. A typcial SMTP connection including all the filetering typically takes between 0.5 and 1 second per message. Sometimes less, sometimes more. I'm now running on an Athlon XP 2600+ with 512 MB of RAM. Sure, my server is not super-busy, but there is room for a lot more volume.

Apollo

Ok, man, you've got me convinced. I'm gonna try installing magic-smtpd. I just want to be able to block incoming spam at the smtp level as you are doing. That sounds like what the doctor ordered. Are you using vpopmail with mysql? I can't remember if you are.

Frankly, I would like to rid myself of Spamassassin. I think it's using too many CPU cycles for my tastes and I don't feel that it's benefitting me that much. I need to verify this, but I don't think it's doing me that much good. All SA is doing for me currently is auto-deleting messages that score over 10 points. At one point, I was having it do subject tagging, but I turned that off, as I was getting too many false positives. I even bought a book on SA in hopes that I could train the Beys filter to be more accurate, but unfortunately, I'm not understanding the material I'm reading well-enough to know what I need to do. I am afraid I will simply screw something up. To be honest, I'm just the kind of person who prefers howto's instead of reading the "raw material" and determining for myself what needs to be done. I'd rather read a howto and then read about the nuts and bolts later on so I can better understand how to tweak my setup. So with all that being said, I am searching for a good howto on Spamassassin that talks about training the filters for better accuracy. I will post back here if I find anything, but its low priority for now.

I use vpopmail. Not with mysql. Actually, I built a new server in January based on the qmailrocks.org formula (with some modifications, like magic-smtpd). It was pretty painless and works well. I'd recommend that route to anyone starting from scratch. I'm planning to use this setup as a template of sorts for client installations.

Spamassassin is definitely a hog and the benefit is questionable if you are using blacklists. If my CPU resources get squeezed it will be the first thing to get chopped. I sped it up a bit by disabling Razor2, but that surely cuts its effectiveness too. The benefit of Clamav in this setup is questionable too since the blacklists eliminate 99% of virus messages (usually from dynamic ranges -- infected machines on cable/dsl connections). Clamav is not as much of a hog as SA. I will probably keep Clamav.

Uggh! I was just reading the documentation for magic-smtpd and apparently it uses syslog for logging! I'm not too crazy about that. My syslog is pretty overloaded already and adding more to it isn't something I'm thrilled about. Plus, there are inherent flaws with syslog that DJB talks about on his website. If I had my way, I would log everything with multilog... but that's another battle to fight.

I'm not saying I won't use it... but switching to syslog for my smtp service sure does suck. Also, I'm still not completely clear about how you define some things like rblsmtpd? Is there a control file that magic-smtpd uses for that? Is there a way to define softlimit?? Maybe it would help to see your qmail-smtpd/run file now that you've modified it for this. Maybe there is another document I should be reading?? I just read the PDF documentation that came with it and didn't see much else available.

Hey Donboy, I wasn't aware of any issues with syslog, but I will take a look at DJBs site now. I am no logger guru. I use whatever is standard in the distro. I'm running this on FreeBSD 5.3, so I think it's newsyslog (?). Not sure what's involved in using multilog for this. No clue, but I bet someone has done it.

It was dead simple to add magic-smtpd. As it turned out I did not even need to modify my /var/qmail/supervise/qmail-smtpd/run file at all. The install creates a backup of your old qmail-smtpd file, then links qmail-smtpd to the magic-smtpd file. That's why you won't see "magic-smtpd" in the run file. It truly is a "drop-in" replacement for qmail-smtpd. As mentioned, I installed qmail initially according to qmailrocks and this run file came from qmailrocks with few or no changes. Here is my run file:

------------------
#!/bin/sh
#PM:Jan-19-2005: modified to add rblsmtpd support.
#PM:Jan-21-2005:Added magic-smtpd package. As it turns out, no modifications were
# needed to this run file to implement magic-smtpd.
QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl" export QMAILQUEUE
QMAILDUID=`id -u vpopmail`
NOFILESGID=`id -g vpopmail`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
LOCAL=`head -1 /var/qmail/control/me`
if [ -z "$QMAILDUID" -o -z "$NOFILESGID" -o -z "$MAXSMTPD" -o -z "$LOCAL" ]; then
echo QMAILDUID, NOFILESGID, MAXSMTPD, or LOCAL is unset in
echo /var/qmail/supervise/qmail-smtpd/run
exit 1
fi
if [ ! -f /var/qmail/control/rcpthosts ]; then
echo "No /var/qmail/control/rcpthosts!"
echo "Refusing to start SMTP listener because it'll create an open relay"
exit 1
fi
exec \
/usr/local/bin/softlimit -m 40000000 \
/usr/local/bin/tcpserver -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
rblsmtpd \
-a whitelist.local \
-r blacklist.local \
-r dnsbl.sorbs.net \
-r sbl-xbl.spamhaus.org \
-r l2.spews.dnsbl.sorbs.net \
/var/qmail/bin/qmail-smtpd mydomain.goeshere.com \
/usr/home/vpopmail/bin/vchkpw /usr/bin/true 2>&1
---------------------

Note: whitelist.local, blacklist.local, and dnsbl.sorbs.net are all local lists. I rsync sorbs every 12 hours and run it locally. Spamhaus and spews are not local.

I think your softlimit and rblsmtpd questions may be clarified with the above (?). That softlimit is suggested in qmailrocks.org set up. Seems to work. rblsmtpd works identically as with the standard qmail-smtpd (as you can see in my run file).

As for magic-smtpd documentation, I found the install manual to be adequate. It defines all the control files, etc:

http://www.linuxmagic.com/opensource...pd/manual.html

Is the documentation sparse? I didn't think so, but maybe they are hoping many people will opt for their commercial version over the opensource. You do have to wonder if companies that do this sort of thing (mix commercial and opensource) are really committed to the opensource concept.

Apollo