Not sure what mail server you use, I run postfix. I can blacklist right in the postfix config, as well as utilize many many other UCE (unsolicited commercial email) blocking features that are simple to implement, reject mail to unknown users, reject mail from servers that do not resolve using rdns, etc.. etc.. on top of that I use Spamassassin to inspect messages and tag those that look like spam. SA also can use Black Lists, custom lists etc..
so if nothing else take a look at spamassassin.
http://spamassassin.apache.org/
I would also be surprised if all the black lists contain addressess you communicate with, I use about 6 different black lists on my servers. unless you receive mail from asia, and people that host mail on Dynamic IP addressess and people that send valid mail through open relays (no such thing as valid mail through open relays imho) there are definately Black lists you can use to block some of that junk. I reject about 60-70% of all mail that touches my server through black lists and other tests.. then it gets filtered through SA and gets tagged.
It gets examined and tagged again when users retrieve it by a commercial device I purchase from Fortigate.. I have had very few complaints about valid mail being blocked. most people ask if I can make it tighter to block more.. if they only knew the percentage I block vs what makes it to their mialboxes..
Well heres some Black lists you can review to find one or several that suit your needs.. read up on their policies for adding and removing people first to make sure their methods agree with you...
http://www.declude.com/Articles.asp?ID=97
I use some of the following lists
sorbs - some not all
spamcop
spamhaus - some not all
ordb
Best of luck in your battle...