I've got someone or multiple people sending spam from my mail server but I cannot figure out which accounts their sending from is there a script I can run that will monitor the amount of mail each person sends out? I know there was an excellent one I used awhile back that notified me if someone CC'd a bunch of people in their email but I don't remember what that was called.

First of all, unplug your machine or turn your MTA off. Secondly, make sure your MTA is not allowing or is configured to be an open relay. It might not be an actual user but you have your system setup to allow anyone to send mail thru it.

SMTP AUTH is another option.

i cannot unplug the machine its in production I work for an ISP. There isnt any open relay i've double checked that, and SMTP auth wouldnt help since the spammer is spamming through squirrel mail so i ts sending from 127.0.0.1

Check your mail logs then to determine who's sending spam and close their account. At least thru logs you can narrow down the culprit.

So if you were sending out viruses you wouldn't unplug the machine? Spamming is just as bad as getting hacked. I would suffer downtime to save spam or viruses from spreading from my network or you'll soon find your server and IP or IP Block on every gray or black list and soon you'll find no one can send email. You decide what's more important.

And what kind of ISP uses SquirrelMail and what type of spammer uses SquirrelMail? They can't be sending too much with such a application.

And from my knowledge, you can still use SMTP auth it being local and all. How do you know people are spamming anyways?

alot of isp's use SquirrelMail actually.. atleast in western Newyork anyway it is also something Redhat suggested. Regardless I have checked the logs and I cannot see it coming from a specific user actually all I see is a ton of bounce back messages about being blacklisted. If I do a qshape it will jump from 0 to 1000 emails in seconds and 500 of them would be to yahoo for example.

So this is when you start digging thru users home directories in hopes they have a copy of the message in a sent or draft folder of the ones you're seeing getting sent out. I'd say yank the network, make a small downtime to stop sending spam, as it seems you've already been blacklisted, it's only going to get worse.

And bounceback messages should have some type of return address. A user should have something configured for outgoing that matches the headers that are getting sent. If not, then I think you better recheck your config as it seems you may even have a cracker using your server to spam email. But seriously, yank the damn network cable now before matters get worse. Spam is just as bad as a virus, treat it as such and a possible break in attempt.

I just found out from one of the other admins that our server was recently hacked into so im assuming they installed something or setup a backdoor. Whats the best way to scan for this.

chrootkit but now's the time to unplug the machine if you haven't done so already.

There are ways to prevent this in the future, install tripwire, cfengine and other tools that do sanity checks on your system. Enforce strong passwords from users. And of course, don't login as root and disable root login remotely.

And now for the fun part, don't use this machine again, rebuild it with the user base and hopefully you have backups from before the crack was made, to ensure no one has any files that are affected, etc. Redeploying a machine that has been tampered with is not a good idea, best to rebuild and before redeploying, wipe and reinstall as new server.

looks like in the mail queue im sending mail from hotmail addresses and others not on my domain. I scanned for an open relay and nothing showed up.

You probably want to try working through the CERT Intruder Checklist and get a grip on how you got cracked. You don't want to redeploy only to have the same thing happen all over again.

I just wanted to go back over what others have said.

You will have to rebuild this server
There is no way to keep using it without a complete rebuild.

I've seen this happen before, with people not wanting to rebuild or whatever after a hack. There is no way to 100% know everything that the hacker(s) did to the machine.

Steps to take once again:

1. Unplug network cable.
2. Make sure the network cable is unplugged.
3. Stop looking at the goddamn mail queue and unplug it already.
4. Ask a peer to double check your unplugging of the network cable.
5. Do not plug back in until the machine is wiped, reinstalled and you've read some documentation and books on how to prevent these types of cracks to your network and server OS of choice.
6. Beat yourself on the head for not unplugging the damn machine when you were first told to do so.
7. Have yourself demoted to tape monkey without any login access to the servers.
8. I think McDonald's is hiring, you'll make more money there after your demotion.

Seriously, have you unplugged this machine yet? Hold on, I think I just got some more spam in my Inbox from your server, I hope there's no viruses attached.

thanks for all the help it ended up being a users account that had been compromised that was the issue. I scheduled a maintenance window to take down the email server and let the secondary cluster node take over. The isp I work for has over 15k users and to just unplug the mail server would cause major issues with the customers, yes I wanted to unplug it but I have a boss and he decides what happens. I ended up installing a 3rd party application to view mail messages in queue in a more human readable format which showed me what account it was coming from so i'm glad I tracked down the problem instead of taking some of your advice and unplugging it and formatting it because its on a cluster with a shared home directory which means it would of had the same issues. Oh and I will be formatting the other server in the cluster as well because access to one server permits access to the other in the cluster.

again thanks for all the help I really appreciate it.

If I were a customer and I was told that my mail was unaccessible cause you were trying to prevent a user's account from sending spam after it was compromised I would have been happy. But you mention a second node or clustered machine that picked up the load, so why couldn't this have happened sooner? If that were my boss at some of the places I've worked at, he would have been fired by now not taking action sooner by taking this server offline. Honestly, protection of customers is more important than their email they can always retrieve afterwards, especially if it's only mail and not their actual connection.