My mail server may be sending spam..

davidstvz · 08-02-2010, 12:53 PM

I'm a little overwhelmed and unprepared for this. I'm not even sure the best way to verify whether the server is or isn't compromised (we're using sendmail). If it is compromised, how is it compromised (it would only take one compromised account to send spam)? Complicated questions obviously.

The IP is on the BarracudaCentral.org list with a "poor" reputation so that's a bad sign and my first indication that there is a problem.

EDIT

Ok, ignore the old post below. It's a bad idea to even look at sendmail logs as if the machine is compromised it's probably sending mail directly from port 25 rather than using my MTA. Anyone know a good way to filter raw packets to determine what emails are outgoing?

EDIT (old post below

I'm looking at the mail.log and I'm not sure how to distinguish outgoing from incoming mail definitively. I don't know how to read the output of 'ps aux | grep sendmail' either. Having trouble finding a reference on that except stuff about people having mail stuck in the queue, a non security problem.

I'm going to continue fishing around... any suggestions would be greatly appreciated.

Code:

root     12642  0.0  0.0  32400  3252 ?        S<   09:59   0:00 sendmail: ./o6SJTCLn022960 embarqhsd.net.: user open
root     13735  0.0  0.0  32080  2488 ?        S<   10:14   0:00 sendmail: ./o6TAZ2gb010359 tcisl.net.in.: user open
root     14466  0.0  0.0  32080  2412 ?        S<   10:29   0:00 sendmail: ./o6TMae0N031513 e-sms.ctbcnetsuper.com.br.: user open
root     15171  0.0  0.0  32080  2400 ?        S<   10:44   0:00 sendmail: ./o6ULeWCv006079 embarqhsd.net.: user open
root     15798  0.0  0.0  30024  2312 ?        S<   10:59   0:00 sendmail: ./o6V9o9XH025016 dvb-brasil.org.: user open
root     16657  0.0  0.0  30024  2308 ?        S<   11:14   0:00 sendmail: ./o6VNPk8W011998 asterix.ednet.lsu.edu.: user open
root     17319  0.0  0.0  30372  3140 ?        S<   11:29   0:00 sendmail: ./o713sXKP016527 m.scs.tec.ut.us.: user open
root     17804  0.0  0.0  30024  2284 ?        S<   11:44   0:00 sendmail: ./o71JWa6o004434 mail.pressdisplay.com.: user open
root     18327  0.0  0.0  30024  2320 ?        S<   11:59   0:00 sendmail: ./o726i7JW025753 gbainc.net.: user open
root     19333  0.0  0.0  31872  1972 ?        S<   12:08   0:00 sendmail: server host213-123-115-199.in-addr.btopenworld.com [213.123.115.199] (may be forged) cmd read
root     19334  0.0  0.0  31872  1972 ?        S<   12:08   0:00 sendmail: server host213-123-115-199.in-addr.btopenworld.com [213.123.115.199] (may be forged) cmd read
root     19467  0.0  0.0  31872  1972 ?        S<   12:10   0:00 sendmail: server host213-123-115-199.in-addr.btopenworld.com [213.123.115.199] (may be forged) cmd read
root     19468  0.0  0.0  31872  1972 ?        S<   12:10   0:00 sendmail: server host213-123-115-199.in-addr.btopenworld.com [213.123.115.199] (may be forged) cmd read
root     19617  0.0  0.0  31872  1972 ?        S<   12:11   0:00 sendmail: server host213-123-115-199.in-addr.btopenworld.com [213.123.115.199] (may be forged) cmd read
root     19620  0.0  0.0  31872  1972 ?        S<   12:11   0:00 sendmail: server host213-123-115-199.in-addr.btopenworld.com [213.123.115.199] (may be forged) cmd read
root     19873  0.0  0.0  30228  3068 ?        S<   12:14   0:00 sendmail: ./o72AveI1002148 asiapacific.org.: user open
root     20566  0.0  0.0  30224  3020 ?        S<   12:29   0:00 sendmail: ./o72GGTLu016724 gamblingplanet.org.: user open
root     20617  0.0  0.0  31872  1948 ?        S<   12:31   0:00 sendmail: server [72.252.69.26] cmd read
root     20620  0.0  0.0  31872  1948 ?        S<   12:31   0:00 sendmail: server [72.252.69.26] cmd read
root     20633  0.0  0.0  30252  2936 ?        S<   12:31   0:00 sendmail: ./o72HVvOK020633 mx6.telecomitalia.it.: client DATA 354
root     20640  0.0  0.0  29816  1648 ?        S<   12:32   0:00 sendmail: startup with 3-184-93-178.pool.ukrtel.net
root     20645  0.0  0.0   5164   796 pts/7    R<+  12:32   0:00 grep sendmail
root     24411  0.0  0.0  29816  1764 ?        S<s  Jul15   4:42 sendmail: accepting connections
smmsp    24414  0.0  0.0  25704  1384 ?        S<s  Jul15   0:00 sendmail: Queue runner@00:15:00 for /var/spool/clientmqueue

arizonagroovejet · 08-02-2010, 01:16 PM

Since your question is related to sendmail, not Linux, you might do better asking for help in sendmail orientated forum. http://www.sendmail.org/ would be a good place to start. Where are you more likely to find in depth knowledge regarding sendmail, here on a forum for discussion of one of the operating system on which sendmail can be used, or... You may well get some very useful help here, but just saying.

davidstvz · 08-02-2010, 01:17 PM

^^^ good point

Ok, proper messages coming from my server show "relay=localhost.localdomain" in the log file and nothing seems amiss among those entries.

Of course, if it is compromised, the messages aren't necessarily being logged accurately.

bathory · 08-02-2010, 01:36 PM

Hi,

This is a ps output, not /var/log/maillog. Anyway I guess that these are hanging connections from hosts that are trying to find if your sendmail can be used for spam.
You can restart the service, so they will go away.

You can test here to see if your server is an open relay

Regards

davidstvz · 08-02-2010, 01:49 PM

Thanks, it's not an open relay and I just used this page:

http://www.dnsqueries.com/en/check_banned_ip.php

...to test block lists. We're only the one list... barracudacentral.org... so perhaps it is a mistake?

davidstvz · 08-02-2010, 03:06 PM

Anyone know how to look for outgoing emails by inspecting packets? I'm using TCPdump right now, but I need to refine the output of the program to be a bit more meaningful. I may figure it out myself... working on it.

yanger · 08-02-2010, 03:49 PM

i get weary sometimes too with my mail server.. let us know if you find any clues

that way it can help others. you could wireshark monitor and filter mail [similar to tcpdump.. eh.. you can filter mail in tcpdump too, right?], but that's only if the server's close by...

davidstvz · 08-02-2010, 03:54 PM

Have you ever ended up on a block list for no good reason??

If I could just count the number of outgoing messages by inspecting packets I could compare that to the mail.log. There have only been maybe 80 legitimate emails outgoing since 7 am yesterday.

bathory · 08-02-2010, 04:16 PM

You can run:

Code:

mailq -v
mailq -Ac -v

to find the queued messages.
Also running

Code:

tail 0f /var/log/maillog

should show you what is your server doing

davidstvz · 08-02-2010, 04:31 PM

Those mailq commands? Those are for exim (exim4?) I'm using sendmail actually.

I'm wondering if my users with forwarding accounts (I have several) aren't causing spam to be forwarded to other servers (since we are getting bombarded by spam lately) and one or some of those have mistaken my server as the origin of the spam and got us on the block list.

Maybe I should have the filter just delete the spam instead of tagging it.

bathory · 08-02-2010, 04:49 PM

mailq commands are for sendmail. In fact mailq is an alias of "sendmail -bp"

Anyway why do you have to wonder and don't look at maillog.
And of course you can setup the spam filter to discard spam instead of forwarding it to a user's forward address.

yanger · 08-02-2010, 05:12 PM

Actually, I was once. I don't remember where I was blacklisted, but I do remember mailing them to remove me off the list. Sometimes some sites blacklist by mistake or if you just got your server, you may of got a reused ip which was previously blacklisted.

davidstvz · 08-02-2010, 05:17 PM

Yep, I've just adjusted the spamass-milter to start deleting messages scored 12.0 or higher (5.0 or higher is simply marked and I've yet to have one false positive in my inbox in 2 years).

A few weeks ago I also changed the spamassassin configuration so that messages would be tagged with ***SPAM*** in the subject and sent along, whereas for the previous two years they were mime-encapsulated in another email and the capsule was tagged with ***SPAM*** in the subject. That and the fact that spam has increased greatly in the last month or so (the reason I tweaked the encapsulation setting) could be the culprit.

I tell you that's a lot better than having a hacked server... so I'm keeping my fingers crossed!

@bathory: about the mailq. I ran it with those options and I got some kind of exim4 error. If I try to run it unprivileged, I get "exim: permission denied". Exim was installed on this server by default... I removed it and installed sendmail. Maybe I didn't remove it as much as I thought.

mailq -Ac gives:
"exim abandoned: unknown, malformed, or incomplete option -Ac"

when I run mailq -v I get an empty queue every time

bathory · 08-03-2010, 03:19 AM

Quote:

mailq -Ac gives:
"exim abandoned: unknown, malformed, or incomplete option -Ac"

when I run mailq -v I get an empty queue every time

Looks like mailq is a symnlink to exim and I guess exim does not support the "-Ac" option. To verify run:

Code:

ls -l `which mailq`

Then delete the symlink and create a new one to sendmail

davidstvz · 08-03-2010, 09:31 AM

Ok, thanks. Now I have a queue. I need to look up the procedure for deleting things from the queue. There's a bunch of crap in there that's stuck because a forwarding address is apparently gone.