Welcome everybody (this is my first post) !

Today i noticed that there was spam sent from my server (centos5 + cpanel)
from headers it looks like the spam was sent as root:

X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12]

all packages on this server are up to date, including kernel.

Is it possible to send e-mail message so these headers will show that the Originator's UID/GID is 0/0 ? but as _regular_ user ?

if it's possible then there is still hope that the server is not hacked with root prvileges, but if it's not possible, that would indicate hacked server with root privs...


i have tried to send e-mail from regular user's account but when doing so, in headers i see:

X-AntiAbuse: Originator/Caller UID/GID - [558 555] / [47 12]

Welcome to LQ, hope you like it here.


As far as I've read adding the X-AntiAbuse may be configured through a web-based panel but are actually set by the MTA. I doubt anyone would want to spoof or explicitly set it to UID == 0.
You could:
- generate detailed listings of processes, open files, users and network connections (host, pastebin or attach as plain text?),
- stop your web server and MTA while investigating,
- check user login records (just in case),
- run all system and daemon logs through Logwatch without service exclusions and using "--Archives" (generate leads),
- look for any network-using processes that run as UID root or
- have odd CWDs like /var/www or
- have odd process names (like your httpd is /usr/sbin/httpd but you see "/usr/local/bin/apache -DSSL"),
- search your docroot(s) for any mailforms,
- check if any web log, forum, web mail, statistics or other applications you run in your web stack have known LFI/RFI/other net-attackable vulnerabilities.

2 members found this post helpful.

Thank you for your reply.

but is there confirmed information that this spoof is doable ? even in theory ?

i'm going to use your hints, hope to find some more info, thanks for it.

Greetings,

this is my fourth server affected in past 15 days (Centos5 + cpanel), as the result, spam is being send by the root user:

headers:
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.XXX.XXX
X-AntiAbuse: Original Domain - tmail.com
X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12]
X-AntiAbuse: Sender Address Domain - server.XXX.XXX

two example exim's log lines:
2010-10-14 16:29:51 [24688] 1P6Pkd-0006QC-8o <= root@server.XXX.XXX U=root P=local S=635 T="Hi Joe Smith. It's Glenda. Wanna date?" from <root@server.XXX.XXX> for at18sharks@aim.com
2010-10-14 16:29:51 [24691] 1P6Pkd-0006QF-9B <= root@server.XXX.XXX U=root P=local S=659 T="Hello Pothead666. It's Charlene. Wanna date?" from <root@server.XXX.XXX> for bielfeedback666@yahoo.com


as far as i know, this header can't be faked by regular user:
X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12]
so it must be really root that sends this spam. all Centos packages are up to date and ksplice (rebotlees kernel updates) are in effect (if that have any matter).

for me it's odd that someone (or some scanner) exploit a server with root privileges just to send spam.


have anyone exprienced this problem ? or have any ideas or anything ? thanks upfront.

Good heavens! 4 different machines? Are they all in the same physical location? Have you checked to see if they are rooted? What mail server are you using?

It's not clear from the information you posted; are these mails being sent via your mail server, or are these system emails from the root user destined for the terminal?

If the machines are rooted (a root kit installed) or even wormed the first thing I would suspect, unfortunately, is ksplice. When they announced it's release I looked into it and had doubts as to it's security.

Before anyone here can suggest possible mitigation we'd need to know the answers to the above.

It's unusual for a root system account to be compromised. Usually it's the server app, rarely is anything in the underlying system hit. It can be done, but normally through the server(s) being exposed. I would be quite disturbed if the server was healthy but the root account has been cracked.

What other servers and/or services are open on these machines? Please tell me not telnet...

I'm going to merge this thread with your previous one to keep discussions in one place.

Yes, all are in the same physical location, but the DC technicans are 100% trusted, so this is not an option. I have double checked each server, and found nothing. All i have are these exim AntiAbuse headers that says the spam is being sent by root.

yes, each server have it's own mailserver on localhost

i've contacted the ksplice team:
"I don't believe we're aware of any known exploits going around." is their reply. Obviously i can't verify that info ;-)

i hope i provided you the informations you wanted, if you need more let me know.

agree, unless some kernel/system service/cpanel 0day gone wild..

named, snmp, mysqld, httpd, exim, named, cupsd, sshd, pureftpd - all are latest centos5's packages versions

no telnet

Those are all exposed to the internet? Any of them could be a vector for gaining root access on the host, particularly cupsd and ftp.

Is the server hosting a SQL database that's accessible from the outside? If it's only a backend to something apache is doing you should close off the port.

Have you checked logs to see if any untoward burst of activity occurred seeking to connect to any of those services? You might want to run snort on the thing for a while and see if whoever comes back, once you shut off the mass mailings.

I'd run wireshark on the internet facing adapter to see which, if any, port or ports are the vector of attack.

If the body of the mails that are going out are not files residing on the hard drive then the content of each has to be coming in through an exploited channel along with the mail command that's sending them. That ought to be very easy to see with wireshark.

Another tool I can't live without is iptstate. I believe there's a .deb for that. It's a console tool that shows all active connections in the heart of the IP stack, the addresses connected and on which port.

My first go-to tool when I log in to a remote server for a hands-on session I run iptstate. I can tell right off if someone unauthorized is present, that snort or the mail subsys might not be working etc.

Lastly, ftp. Is this for admins? Or is there a public use? I always shudder when there is a requirement for ftp to be facing the public. It's hard as heck to secure, harder to move off standard ports and there's no getting around passwords... in-the-clear usually, the script kiddies have a field day with dictionary attacks on ftp.

If it's strictly for admins to update content, I'd use rsync over ssh, or even consider sshfs. (I have set up quite a few systems to utilize sshfs, never seen anyone poking at it, and I almost always move ssh off port 22 to thwart the automated mayhem)

The only time I ever expose cups to the internet is on a honeypot.

BTW do you run X on this machine? That is a major vector for compromise. I have never run X (let alone a desktop environment) on a public-facing server. Someone may have been able to access one of your services via 'normal,' apparently legitimate means then compromised the system via X.

1 members found this post helpful.

catworld, i am sure you want to deliver best answer, and your hints and points are valuable, but i know this all, have dealt with such cases so many times before that i can't even count it all. i have investigated all these services that are open to the public and if even one of them would be affected, i would figure that out, keep on mind i posted here after trying just about _everything_ that was possible to do in the production environment.

the original question was if anyone from the huge LQ community is aware of such problem - the spammer who gets root on latest centos5 machines, just to send spam. everything that is needed for someone to recognize such attack is already posted before, this must be 0day and i even caught the attacker's IP, perl process responsible for spam, debugging the process and attaching gdb to the pid gave me nothing but ptrace: Operation not permitted. Well, even if i have the source code of program that is sending spam i will find, hm, how he is sending it? that's useless, i'm trying to search for way he exploited it at first place, this is most important, and i believe this will hit others soon, not only me.

do you recognize this IP ? : 65.110.42.90
the 6th machine that got owned, recorded this IP and that's all we have for now, except the AntiAbuse headers, btw the perl process was running as root, so it's confirmed now to be root exploit.


sure there is a chance for password leak from third party or even sabotage possibility, but at this point i simply don't believe on that. i believe it's 0day, and someone must know something.

whatever it is, thank you, catworld, for trying to help. appreciate.

awaiting some news, hopefully.

so not knowing who/where you are, the real answer must be you have responsibility for protecting something someone who can't be denied wants, or someone who is paying you for the same pissed off the wrong people, whomever/wherever that may be.

There is no "Oday" in public, so it must be gift wrapped specially for you.

You'd have to provide a few tons more information for any useful help, but Uruguay is always a good answer.

Best answer?

Eesh...your original post and your follow-ups didn't have much info and Catworld tried to help by providing good tips and information to follow.

Did you run that IP through 'whois'? If so, you'll see it is one out of a block of IP's owned by a ISP. You could contact their administrator...but more than likely the IP in their block might have been a relay and not the original source.

As for your original question, can email header UIDs be spoofed, yes they can. Spammers can edit anything in the header.

could you show me how it can be spoofed ? because you just stated it, i'm forced to believe?

this question is outdated, i've found perl process, that was running as root, and it was sending the spam i mentioned above. confirmed to be root.

sorry, should i kill myself ? i'm for years into security but what's going on have no sense for me or i'm going nuts. perhaps you even have right but - as before - obviously i'm not able to verify that.

if i own some machine with root access i would send spam as regular user (nobody/mailnull, etc) to keep the root access with me, in contradiction: if i have just regular user access i would do everything i can so the attack would look like coming from root, so the one who's investigating it focus on something else than he should.


yes i asked 'whois' for that ip and sent email to the abuse with no reply so far, still waiting but not expecting any answer as none returned to me yet.

wish i could, i have more info but sharing it is not possible at this point.

Uruguay? are you ok there ?

You've indicated a root compromise. If the machine is compromised, after all you said you've found the UID 0 perl process, then asking questions about spoofing contradicts with "i know this all, have dealt with such cases so many times before that i can't even count it all" your (OK, perceived) level of practical security knowledge (unless you're applying some alien methodology). If you are not willing to share details except "i've found perl process, that was running as root" then there is nothing in this thread that helps us help you (and I do not count speculating wildly as helpful).

At this point you have a choice and you can turn this whole thing around. But that means no more interpretations, no more descriptions of what you think you are seeing and no more wild goose chases but providing detailed (anonymized) information like log excerpts, tool output and reporting from running checks from the CERT Intruder Detection Checklist that may help us help you. Please keep that in mind.

1 members found this post helpful.

Sorry, I'm too cryptic sometimes. What I was trying to put across is you should consider the "social engineering" aspect.

For instance, if you are running this for a widget manufacturer, you may suspect other widget manufacturers to be more interested in you than any random attacker.

Add to that a possible exacerbating situation; did your company just fire somebody that has enough knowledge of your systems to compromise them, or at least enlist someone else to do so?

Conversely, did your company just head-hunt someone from the competition, that would scare them enough to take illicit measures to try to figure out what's going on? (look at the recent Oracle/HP executive debacle)

The mailings may not be the actual goal of the attacks, in fact it may well be an inverse honey pot. You'd find this unwarranted email traffic, eliminate it, watch for a while and see that it's indeed gone, and think you've dealt with the compromise, when actually a rootkit or back door remains.

One thing I said is better stated thusly: did the entity you work for, or who's servers these are, or anyone associated with the reason for these systems existing say or do anything that angered someone else? Or perhaps 'intrigued' someone else to where they'd target you?

EG maybe the CEO said something cryptic about a release in the next quarter and that shareholder are going to be very happy about it... such that someone in the industry (whatever it is) would take any measure to find out what's going on.

I suggest you look into potential reasons for all this, because it does seem you are being directly, specifically targeted, rather than suffering from your typical skript-kiddie garbage.

You didn't specifically say whether you ran wireshark and iptstate on the outside interface. You'd have the info the abuse handler at the remote ISP would need to sniff out a more specific source... which I will tell you he/she will hope is outside of their domain, eg the IP is either spoofed or is merely a relay.

As for Uruguay, that was a highly angular non-sequitur. My wife understands me but that's about as far as that goes. But based on what you have provided by way of information, I still say "Uruguay" is as valid an answer as any other.

Was SELinux in enforcing mode on affected machines ? After reading all comments of original poster I would say either there is social engineering/someone fired recently, or unskilled server maintenence

Cheers