Server compromised (exim headers) ?
 

Welcome everybody (this is my first post) !

Today i noticed that there was spam sent from my server (centos5 + cpanel)
from headers it looks like the spam was sent as root:

X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12]

all packages on this server are up to date, including kernel.

Is it possible to send e-mail message so these headers will show that the Originator's UID/GID is 0/0 ? but as _regular_ user ?

if it's possible then there is still hope that the server is not hacked with root prvileges, but if it's not possible, that would indicate hacked server with root privs...


i have tried to send e-mail from regular user's account but when doing so, in headers i see:

X-AntiAbuse: Originator/Caller UID/GID - [558 555] / [47 12]

Welcome to LQ, hope you like it here.


As far as I've read adding the X-AntiAbuse may be configured through a web-based panel but are actually set by the MTA. I doubt anyone would want to spoof or explicitly set it to UID == 0.
You could:
- generate detailed listings of processes, open files, users and network connections (host, pastebin or attach as plain text?),
- stop your web server and MTA while investigating,
- check user login records (just in case),
- run all system and daemon logs through Logwatch without service exclusions and using "--Archives" (generate leads),
- look for any network-using processes that run as UID root or
- have odd CWDs like /var/www or
- have odd process names (like your httpd is /usr/sbin/httpd but you see "/usr/local/bin/apache -DSSL"),
- search your docroot(s) for any mailforms,
- check if any web log, forum, web mail, statistics or other applications you run in your web stack have known LFI/RFI/other net-attackable vulnerabilities.

Thank you for your reply.

but is there confirmed information that this spoof is doable ? even in theory ?

i'm going to use your hints, hope to find some more info, thanks for it.

Possible 0day attack ?
 

Greetings,

this is my fourth server affected in past 15 days (Centos5 + cpanel), as the result, spam is being send by the root user:

headers:
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.XXX.XXX
X-AntiAbuse: Original Domain - tmail.com
X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12]
X-AntiAbuse: Sender Address Domain - server.XXX.XXX

two example exim's log lines:
2010-10-14 16:29:51 [24688] 1P6Pkd-0006QC-8o <= root@server.XXX.XXX U=root P=local S=635 T="Hi Joe Smith. It's Glenda. Wanna date?" from <root@server.XXX.XXX> for at18sharks@aim.com
2010-10-14 16:29:51 [24691] 1P6Pkd-0006QF-9B <= root@server.XXX.XXX U=root P=local S=659 T="Hello Pothead666. It's Charlene. Wanna date?" from <root@server.XXX.XXX> for bielfeedback666@yahoo.com


as far as i know, this header can't be faked by regular user:
X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12]
so it must be really root that sends this spam. all Centos packages are up to date and ksplice (rebotlees kernel updates) are in effect (if that have any matter).

for me it's odd that someone (or some scanner) exploit a server with root privileges just to send spam.


have anyone exprienced this problem ? or have any ideas or anything ? thanks upfront.

Good heavens! 4 different machines? Are they all in the same physical location? Have you checked to see if they are rooted? What mail server are you using?

It's not clear from the information you posted; are these mails being sent via your mail server, or are these system emails from the root user destined for the terminal?

If the machines are rooted (a root kit installed) or even wormed the first thing I would suspect, unfortunately, is ksplice. When they announced it's release I looked into it and had doubts as to it's security.

Before anyone here can suggest possible mitigation we'd need to know the answers to the above.

It's unusual for a root system account to be compromised. Usually it's the server app, rarely is anything in the underlying system hit. It can be done, but normally through the server(s) being exposed. I would be quite disturbed if the server was healthy but the root account has been cracked.

What other servers and/or services are open on these machines? Please tell me not telnet...

I'm going to merge this thread with your previous one to keep discussions in one place.

Yes, all are in the same physical location, but the DC technicans are 100% trusted, so this is not an option. I have double checked each server, and found nothing. All i have are these exim AntiAbuse headers that says the spam is being sent by root.

yes, each server have it's own mailserver on localhost

i've contacted the ksplice team:
"I don't believe we're aware of any known exploits going around." is their reply. Obviously i can't verify that info ;-)

i hope i provided you the informations you wanted, if you need more let me know.

agree, unless some kernel/system service/cpanel 0day gone wild..

named, snmp, mysqld, httpd, exim, named, cupsd, sshd, pureftpd - all are latest centos5's packages versions

no telnet :)

Those are all exposed to the internet? Any of them could be a vector for gaining root access on the host, particularly cupsd and ftp.

Is the server hosting a SQL database that's accessible from the outside? If it's only a backend to something apache is doing you should close off the port.

Have you checked logs to see if any untoward burst of activity occurred seeking to connect to any of those services? You might want to run snort on the thing for a while and see if whoever comes back, once you shut off the mass mailings.

I'd run wireshark on the internet facing adapter to see which, if any, port or ports are the vector of attack.

If the body of the mails that are going out are not files residing on the hard drive then the content of each has to be coming in through an exploited channel along with the mail command that's sending them. That ought to be very easy to see with wireshark.

Another tool I can't live without is iptstate. I believe there's a .deb for that. It's a console tool that shows all active connections in the heart of the IP stack, the addresses connected and on which port.

My first go-to tool when I log in to a remote server for a hands-on session I run iptstate. I can tell right off if someone unauthorized is present, that snort or the mail subsys might not be working etc.

Lastly, ftp. Is this for admins? Or is there a public use? I always shudder when there is a requirement for ftp to be facing the public. It's hard as heck to secure, harder to move off standard ports and there's no getting around passwords... in-the-clear usually, the script kiddies have a field day with dictionary attacks on ftp.

If it's strictly for admins to update content, I'd use rsync over ssh, or even consider sshfs. (I have set up quite a few systems to utilize sshfs, never seen anyone poking at it, and I almost always move ssh off port 22 to thwart the automated mayhem)

The only time I ever expose cups to the internet is on a honeypot.

BTW do you run X on this machine? That is a major vector for compromise. I have never run X (let alone a desktop environment) on a public-facing server. Someone may have been able to access one of your services via 'normal,' apparently legitimate means then compromised the system via X.

catworld, i am sure you want to deliver best answer, and your hints and points are valuable, but i know this all, have dealt with such cases so many times before that i can't even count it all. i have investigated all these services that are open to the public and if even one of them would be affected, i would figure that out, keep on mind i posted here after trying just about _everything_ that was possible to do in the production environment.

the original question was if anyone from the huge LQ community is aware of such problem - the spammer who gets root on latest centos5 machines, just to send spam. everything that is needed for someone to recognize such attack is already posted before, this must be 0day and i even caught the attacker's IP, perl process responsible for spam, debugging the process and attaching gdb to the pid gave me nothing but ptrace: Operation not permitted. Well, even if i have the source code of program that is sending spam i will find, hm, how he is sending it? that's useless, i'm trying to search for way he exploited it at first place, this is most important, and i believe this will hit others soon, not only me.

do you recognize this IP ? : 65.110.42.90
the 6th machine that got owned, recorded this IP and that's all we have for now, except the AntiAbuse headers, btw the perl process was running as root, so it's confirmed now to be root exploit.


sure there is a chance for password leak from third party or even sabotage possibility, but at this point i simply don't believe on that. i believe it's 0day, and someone must know something.

whatever it is, thank you, catworld, for trying to help. appreciate.

awaiting some news, hopefully.

so not knowing who/where you are, the real answer must be you have responsibility for protecting something someone who can't be denied wants, or someone who is paying you for the same pissed off the wrong people, whomever/wherever that may be.

There is no "Oday" in public, so it must be gift wrapped specially for you.

You'd have to provide a few tons more information for any useful help, but Uruguay is always a good answer.

Best answer?

Eesh...your original post and your follow-ups didn't have much info and Catworld tried to help by providing good tips and information to follow.

Did you run that IP through 'whois'? If so, you'll see it is one out of a block of IP's owned by a ISP. You could contact their administrator...but more than likely the IP in their block might have been a relay and not the original source.

As for your original question, can email header UIDs be spoofed, yes they can. Spammers can edit anything in the header.

could you show me how it can be spoofed ? because you just stated it, i'm forced to believe?

this question is outdated, i've found perl process, that was running as root, and it was sending the spam i mentioned above. confirmed to be root.

sorry, should i kill myself ? i'm for years into security but what's going on have no sense for me or i'm going nuts. perhaps you even have right but - as before - obviously i'm not able to verify that.

if i own some machine with root access i would send spam as regular user (nobody/mailnull, etc) to keep the root access with me, in contradiction: if i have just regular user access i would do everything i can so the attack would look like coming from root, so the one who's investigating it focus on something else than he should.


yes i asked 'whois' for that ip and sent email to the abuse with no reply so far, still waiting but not expecting any answer as none returned to me yet.

wish i could, i have more info but sharing it is not possible at this point.

Uruguay? are you ok there ?

You've indicated a root compromise. If the machine is compromised, after all you said you've found the UID 0 perl process, then asking questions about spoofing contradicts with "i know this all, have dealt with such cases so many times before that i can't even count it all" your (OK, perceived) level of practical security knowledge (unless you're applying some alien methodology). If you are not willing to share details except "i've found perl process, that was running as root" then there is nothing in this thread that helps us help you (and I do not count speculating wildly as helpful).

At this point you have a choice and you can turn this whole thing around. But that means no more interpretations, no more descriptions of what you think you are seeing and no more wild goose chases but providing detailed (anonymized) information like log excerpts, tool output and reporting from running checks from the CERT Intruder Detection Checklist that may help us help you. Please keep that in mind.

Sorry, I'm too cryptic sometimes. What I was trying to put across is you should consider the "social engineering" aspect.

For instance, if you are running this for a widget manufacturer, you may suspect other widget manufacturers to be more interested in you than any random attacker.

Add to that a possible exacerbating situation; did your company just fire somebody that has enough knowledge of your systems to compromise them, or at least enlist someone else to do so?

Conversely, did your company just head-hunt someone from the competition, that would scare them enough to take illicit measures to try to figure out what's going on? (look at the recent Oracle/HP executive debacle)

The mailings may not be the actual goal of the attacks, in fact it may well be an inverse honey pot. You'd find this unwarranted email traffic, eliminate it, watch for a while and see that it's indeed gone, and think you've dealt with the compromise, when actually a rootkit or back door remains.

One thing I said is better stated thusly: did the entity you work for, or who's servers these are, or anyone associated with the reason for these systems existing say or do anything that angered someone else? Or perhaps 'intrigued' someone else to where they'd target you?

EG maybe the CEO said something cryptic about a release in the next quarter and that shareholder are going to be very happy about it... such that someone in the industry (whatever it is) would take any measure to find out what's going on.

I suggest you look into potential reasons for all this, because it does seem you are being directly, specifically targeted, rather than suffering from your typical skript-kiddie garbage.

You didn't specifically say whether you ran wireshark and iptstate on the outside interface. You'd have the info the abuse handler at the remote ISP would need to sniff out a more specific source... which I will tell you he/she will hope is outside of their domain, eg the IP is either spoofed or is merely a relay.

As for Uruguay, that was a highly angular non-sequitur. My wife understands me but that's about as far as that goes. But based on what you have provided by way of information, I still say "Uruguay" is as valid an answer as any other. :D

Was SELinux in enforcing mode on affected machines ? After reading all comments of original poster I would say either there is social engineering/someone fired recently, or unskilled server maintenence

Cheers

First I would suggest to walk through the system maintenance step by step to ensure that the system is updated. There are some quirks regarding cPanel that this should be checked several times a year. Think of it as being about the same maintenance schedule as you would change the oil on your car.

WHM 11 cPanel - HowTo - General Maintenance
https://sites.google.com/site/zenars...al-maintenance

I am not so certain that enabling SELinux would be the answer as it is still too experimental and there have even been recent exploits that target SELinux.

A vulnerability in SELinux
http://marc.info/?l=selinux&m=105490085132101&w=2

As the spam is originating from the root user, they may have exploited some of the system notification scripts.

You would want to search against the Exim identifier to get more details on what all happened. Of course you may have to modify the command a little if the log files have already rotated.

Just don't...
 

// Apologies to the OP for this OT post but this needs to be corrected.

Sure SELinux was a b*tch to handle in the old days but it has improved over the past ten years. Hearsay and personal experience older than four years are no valid basis for making claims on. Your claim is unfounded, fuels misinformation and creates FUD and the case you refer to is stale (2003). Here's some Real Life SELinux mitigation cases:
2006 CVS-2006-3626 local privilege escalation stopped by targeted policy
2007 CVE-2007-2446, CVE-2007-2447 Samba heap overflow and arbitrary code execution (RHSA-2007-0354)
2007 CVE-2007-5208 hplip arbitrary command execution (CVE-2007-5208)
2007 CVE-2007-3304 Apache httpd 1.3.37, 2.0.59, and 2.2.4 Prefork local denial of service (RHSA-2007:0556-2)
2008 APR Flash bytecode exploit (matasano@wayback, PDF)
2008 CVE-2008-0003 OpenPegasus CIM arbitrary code execution (RHSA-2008:0002-7)
So please either do your research or keep yourself from posting things like that.

:):) super, I really had not will to provide more information and oponent information about SELinux back from 2003 :)

Thx

sorry for off-topic
:)

hello guys, i would like to close this thread if possible, as i've solved the problem.

there was data leak, it's all i can say about it, most important is, it's stopped now.


thank you all for help and sensible replies.

catworld's 10-16-10 03:35 PM reply was very close to the truth.

cheers!

Glad you got it sorted out. You might want to mark the thread as solved so others know.

Some folks look at long standing unsolved threads to see what's up. Others might hit the thread searching for a similar problem.

I'm currently having the same issue. Everything is patched, cpanel updated, exim updated. Same exact problem, same exact spam. What did you end up doing to fix this?

Have you looked into the 'social engineering' aspect? When the software is or at least appears to be running normally the first thing I look into is who's had their hands on the hardware. While I'm considering that I'll check for any newly reported exploits and also plan a check for the possibility of a rootkit. (requires downtime, not always the most welcome request)

Exploiting these kinds of things isn't the hardest task, but it's by no means something a script-kiddie bot is going to accomplish, so long as you are up to date and firewalls are well established. It's not a bad starting point to consider either someone has targeted you directly from the outside (trade secrets?) or from the inside. (espionage for same, or disgruntled (ex)employee?) If you see no untoward external traffic making it's way in via logs, it's a pretty safe bet someone has messed it up from the inside, whether intentionally or not..

Only I have access to this server, it is a colocated server so support staff of the datacenter does not have access (I have IP kvm). It's an openvz node with about 40 vz's. Several of the VZ's are penetrated each night around 4-5 am. So far the VZs have little in common, some have exim, some just have sendmail, some have cPanel and some have directadmin. The node has all the latest updates and runs solusvm + openvz. The VZ's have all the latest updates.

I initially posted this in the following thread last night, but it was split off by a mod: http://www.webhostingtalk.com/showthread.php?t=991510

My problem is identical to his. Same spam, same everything. I straced the process and it doesn't seem to do anything except spam through sendmail. I've checked for installed SSH keys, I've checked for changes to pam, checked /etc/passwd & /etc/shaddow. I also checked spool for any crontabs that have malicious data, nothing found :(. I have ossec running and monitoring those files for changes on the node and a couple of the VZs.

Hi,
Just wondering on this one, since you said it was a "data breach"

Can you be a bit more specific about what the file name was, and or contents of the script you found, etc?

I think it would be very helpful to others if you can provide some file specifics, as well as how you ended up solving the issue (that is formatting server, setting up a blocking command of some sort, etc.).

I read in another post on another forum that this was caused by an auto generating encrypted perl script found in / or /tmp, so curious what you learned.

Best Wishes,
Jim

What version of Exim? There was a serious root vulnerability discovered not long ago:

http://www.exim.org/lurker/message/2...32d4f2.en.html

Exim was patched on my servers. There are vps servers without exim that were also affected. Still no exact explanation for what happened, but os reload seems to have solved the problem. Perhaps it started with exim, but not sure how non-exim systems could have been affected. Exim was patched as I found out about the exploit some time before it was even public. What also concerns me is the fact that I couldn't ever find the method they were regaining access with (ie. the rootkit.)

Since the OP and you have consistently refused to share any useful information, since you have denied yourself any chance of investigation by "fixing" things and since I'm not into guessing any (further) reply that doesn't hold leads, useful information or clues is of no value at all.

Hi Dman1,
Were you able to learn anything more in your review?

I've experienced the same "Wanna Date?" hack and have to say it's quite maddening.

Even had all of the top cPanel support people review and they could not figure out how it was being generated from root (after over 4 hours of review on their part and another 20 hours on ours).

1. The spammer in this situation is somehow injecting the body of the email and list of addresses into the server root and sending them via perl as root.

2. Suffice it to say running every malware scanning script and file checking script known to man has turned up nothing out of the ordinary.

3. Even wrapping sendmail and logging to the nth degree did not help in identifying the source.

4. The hack itself injects up to 70k messages within the span of a few minutes, virtually the same time every day during a 2 hour window of time.

5. Other than the perl process appearing momentarily, watching the process list the microsecond messages are created and injected into the queue turns up no details.

6. Since we know when the spam is being injected we just freeze the queue for a couple hours, then the minute after the email is injected we just delete them all, so spammer is wasting their time and ours-- so this has turned in an academic issue at the moment.


- What we do know is that there is a script somewhere in root facilitating the sending of email through perl and sendmail.
- We know that some script on a site on server is being used to pass the information to the root level script.
- We know the spammer has a test script which emails an email alert to spammer approximately every 4 hours to check is server remains compromised (not a cron):
Subject: Notification # 6274
Body: All OK! ______ //h6vK3pJywC
- We know the script in root has a means of deleting itself, since we found a copy of the script on one occasion, and in later episodes look in the same location during the 5 minute spam injection and no scripts were found in that location or any other afterward.


So, if anyone has further evidence to share that would be helpful.


Best Wishes,
Jim

Hey,

I'm basically stuck at the same point you are at. The perl script definitely runs as root, I can see it sending mail if I attach to it with strace. Standard penetration testing gets me nowhere fast. My problems persisted and I've taken a similar approach to yours. On affected systems I have a script running that kills their process before it can send anything out. Pretty crappy solution, but working for the time being. I was initially concerned that this was a result of exim exploits, but now know that's not true since qmail and sendmail servers were affected as well. I would guess this is a remote root exploit on a common linux service. I have not experienced the secondary notification script you have seen. By default I provision servers and vps that run exim to have a lot of logging enabled to deal with spam complaints and I don't see any mail like that being sent.

I really don't know what else to add.

Just curious as to what ports are open on the effected server?

Also wondering if anyone who faces having to rebuild a server ground-up has considered using tripwire?

http://sourceforge.net/projects/tripwire/

I'm already running OSSEC on a bunch of them, but no useful info. I'll take a look at tripwire for some as well. External/internal nmaps of the server dont yield any unexpected information.

Dman1, let me start by offering you a belated welcome to LQ. I notice that you have appended to an existing thread with a similar problem. Unfortunately, that thread is a poor representation of LQ and especially LQ-Security as it contains a bunch of wild speculation and cryptic suggestions about covert employees. Instead of continuing this one, I would really like to recommend that if you are serious about looking into this problem, that you start your own (new) thread on the subject. This is of course assuming that the re-load you reference in your previous thread did not "resolve" the problem. Note that I put "resolve" in quotes because unless you determine what and how you were seemingly compromised you run a real risk of repeating the problem.

At LQ-sec, we focus on a fact approach to performing an investigation. Typically we start by referencing the CERT Intruder Detection check list. Here is a link. It is an older document, but contains a lot of wisdom that I think would be especially helpful to solving the situation described in this thread.

From the description of the problem, it appears that there is a rogue root level process running on this system. These processes can be difficult, but not impossible to find. Lets recap some action items from the Cert checklist. If and when you do start your own thread, please focus on these items and especially with regards to the output of them. If you need help with any of the steps, we can provide that help. Here are some action items from the checklist to get you started.

0) Consider isolating the machine or machines that are infected with this problem. Allow yourself SSH access from a known clean host only. This will help cut the application off from its home base and increase your chances of finding it.

1) Your log files may contain valuable information. The problem with a root level compromise is that they can be altered, even by the problem program. See if you can use a write-once medium to have a copy of the log entries directed to it. See the following page for a possible suggestion: link.

2) This appears to be a root level compromise. Find all files and folders owned by user/group root and any users that belong to this group. Also find any files with the setuid and setguid bits set. *This one is important and I suspect you will 'see' your target, though you may not recognize it at this step.

3) Using this list, verify each of these files against the version contained in your package libraries. Don't assume that any of them are clean. Verify them with a KNOWN CLEAN tool, such as a binary directly from the repository or one on another system.

4) look at your crontabs, for users and roots. See what programs are running, what connections your system is trying to make. Look at the output of ps, netstat, lsof, etc. (e.g. output of /bin/ps axfwwwe; lsof -Pwln; netstat -antupe; who) *NOTE: you will want to verify the legitimacy of these binaries first as they could cover up a rogue entry).

5) look VERY carefully for hidden files, folders, and things in places like user's folders, /tmp folders, and and other locations that are common for wierd file names. **my suspicion is that you will find something here as your OSSEC is not seeing changes to 'normal' system files. **

@Noway2,

Thanks for the welcome. OSSEC should be finding suid/guid 0 files on the whole file system (i tested this.) I've been checking over logs, enabled lots of extra exim logging. I just tried ordering a VPS (normally i host my own VPS on my own colocated nodes) from softlayer's cloud hosting. Fresh cpanel install and added it as a hostname of one of my domains and put up a few domains on it via cPanel. Looked on it today and it's having the same problem now. This was a completely clean setup at a totally different provider a thousand miles away from my other provider.


2011-01-31 10:03:27 1PjwDu-0003L3-SM <= root@cloud.MYHOSTNAMEGOESHERE.com U=root P=local S=671 T="Hi Joy Michael. This is Diana. Wanna date?" from <root@cloud.MYHOSTNAMEGOESHERE.com> for STRIPPEDEMAILADDR@yahoo.com
2011-01-31 10:03:27 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1PjwDu-0003L3-SM
2011-01-31 10:04:28 1PjwDu-0003L3-SM == STRIPPEDEMAILADDR@yahoo.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host
2011-01-31 11:52:52 1PjwDu-0003L3-SM == STRIPPEDEMAILADDR@yahoo.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host
2011-01-31 11:54:59 1PjwDu-0003L3-SM == STRIPPEDEMAILADDR@yahoo.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host
2011-01-31 15:11:00 cwd=/var/log 3 args: exim -Mrm 1PjwDu-0003L3-SM
2011-01-31 15:11:00 1PjwDu-0003L3-SM removed by root
2011-01-31 15:11:00 1PjwDu-0003L3-SM Completed

Also the method being used must be new as there are no more encrypted files in /tmp, and lsof isnt helping me much at this time.

Also I wanted to add that this seems to have stopped happening on my main nodes after some strict firewall rules. I really feel that there is some 0day activity happening. I just can't find any actual rootkit. Here is the nmap output of the new VPS (done externally):

PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
25/tcp open smtp
53/tcp open domain
80/tcp open http
110/tcp open pop3
143/tcp open imap
443/tcp open https
465/tcp open smtps
623/tcp filtered unknown <-- ASF
664/tcp filtered secure-aux-bus <-- ASF
993/tcp open imaps
995/tcp open pop3s
2077/tcp open unknown <-- cPanel Servce
2078/tcp open unknown <-- cPanel Servce
2082/tcp open unknown <-- cPanel Servce
2083/tcp open unknown <-- cPanel Servce
2086/tcp open unknown <-- cPanel Servce
2087/tcp open unknown <-- cPanel Servce
2095/tcp open unknown <-- cPanel Servce
2096/tcp open unknown <-- cPanel Servce
3306/tcp open mysql
32883/tcp open unknown <-- SAN Service
38697/tcp open unknown <-- SAN Service
47266/tcp open unknown <-- SAN Service
48000/tcp open unknown <-- SAN Service
48001/tcp open unknown <-- SAN Service

Seems more people are being affected by this:

http://www.google.com/search?hl=en&s...&aqi=&aql=&oq=

It's possible my workstation is being affected and used to gain access. I do keep up with AV updates (avast) and I use ssh keys for everything. I dont even know what the passwords are. If I ever need the root password I just reset it to a 600+bit one.

Those ports are exposed to public?

Yes it would have to be for cPanel, Apache HTTP, Exim MTA, etc.. I could lock MySQL down to localhost I suppose.

What I do know in our instance is that we suspended all accounts on server and the wanna date spam continued, so the updated email list (which changed daily) was not submitted through a script as we originally believed, and surely had to be sent through some root level server exploit.

What is odd, is that this starts as lots of email to yahoo.com then on another day live.com or hotmail.com becomes the culprit.

That said we gave up and just wiped server. I honestly doubt this has anything to do with your workstation (dman1) since we changed root pass several times during the process of review, and no one else was logged in as root throughout.

Somewhat luckily this all happened on a server with just a few accounts, so we were able to play with server a great deal throughout.

Side note, we have a great deal of security expertise, so those of you stating "get a server admin" are sadly mistaken.
Server was fully PCI compliant by McAfee Secure and other PCI compliant company standards.

As for "Noway2" comments-- been there done that.
In our case, we went way beyond your recommendations/comments (having had over 6 different level 2/3 techs review separately, including cPanel's best).

Wish I could provide more info. Our researched ended at the wipe this past weekend.

Best Wishes,
Jim

Wow. This is an interesting problem. I would make a disk image of an infected system and get it into the hands of someone for study. (sort of like keeping small pox around)

It's disconcerting enough to have such a deeply rooted problem, but more so that as many eyes as have looked at this aren't finding the cause.

This would definitely be a case for tripwire, on a honey pot running the same system prior to infection. Maybe some tech school somewhere would be interested.

My thought about the open ports is why not forward most of that through ssh? Of course service ports need to be there, but admin that can be tunneled should be, it would seem.

Of course nobody has a clue yet what port(s) were exploited initially, and unfortunately it's most likely a public service, (and more likely mail) but who knows?

Only tripwire and a lot of logging would have a shot at conclusive proof.

The only IT college prof I know works for a local small town community college, but they do tout themselves as an IT destination. I'll ask him if he's interested in such a scenario or if he knows anyone who is.

I'm of the opinion something like this exploit can't be left an unknown. Even if it ultimately proves to be a small, previously unknown flaw in one of the components it's still a lesson to be learned. (like how to more swiftly find the source, perhaps)

We've been experiencing the same issue on a cPanel DNSONLY machine -- so there is VERY little running to start with.

A large binary encoded file gets dropped in /tmp and suddenly a perl script is running and injecting email into exim/sendmail. I managed to cat the /proc/XXXXX/environ before killing it...

SHELL=/bin/bash
SSH_CLIENT=62.xxx.xxx.102 40338 22
USER=root
MAIL=/var/mail/root
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
PWD=/root
SHLVL=1
HOME=/rootLOGNAME=root
SSH_CONNECTION=62.xxx.xxx.102 40338 x.x.x.x 22
_=/usr/bin/perl


We have no root logins set for the sshd server but this only happens when the firewall is down (apf kept knocking it down before I figured out what was going on). I'm starting to wonder if there's an exploit in the ssh server that we're not aware of. This happened on 2 other cPanel DNSONLY boxes we were running until we firewalled port 22.

Interesting, and that would be quite disturbing if ssh server is cracked.

Do you use password (ssh) log ins? Keys? Both?

If ssh is a problem I wonder if it would get sniffed out on a non-standard port. I always change ssh to something around 7000, but the service will report if the port is included in a scan. This really only stops the automated 'free download' teen/tween crackers from finding it on 22, useful enough to move the port, anyway.

Was firewalling port 22 the only change made? I'm trying to wrap my head around the common elements in this thread, and so far it appears to be (and please correct what is wrong):

CentOS/RHEL 5
cPanel
sshd
Exim

If the sshd idea is on track, one of the questions is going to be how widespread it is. If everyone is using the same sshd from CentOS, then are we looking at a distro-specific problem (like the one a Debian patch caused a few years ago) or is it a problem fundamental to openSSH?

You were saying this was a cPanel DNSONLY machine, and my understanding (which may be wrong) is that you're not likely to be running exim/sendmail on that. Is that correct or if not, what was running there?

OMG... I just checked the same thing and I think we're being affected by the same Netherlands IP range. Furthermore just a few hours after the cloud was created for me at softlayer (on the 17th) that IP range logged into the box:

root@cloud [~]# cat /var/log/secure* | grep 'Accepted' | grep -v '64.218' | grep -v '64.72.'
Jan 17 20:43:11 cloud sshd[28209]: Accepted password for root from 62.212.130.xxx port 60210 ssh2
Jan 17 20:43:14 cloud sshd[30122]: Accepted password for root from 62.212.130.xxx port 57673 ssh2
root@cloud [~]#

I don't see any more connections from it via SSHd though.

That is correct - the only change made was to firewall port 22. And the only time they got in, was when it was open.




Exim, itself, is NOT running on the DNSONLY boxes...

the processes are VERY minimal.

cpsrvd
named
mysqld (for cPhulkd)
cPhulkd
sshd

Yup, that's the one...

any big files in /tmp with odd names from the same time period?

There were on other affected boxes, this one there aren't any. Seems their method is changing. I'm dropping that whole range for now via iptables on the softlayer vps and at our router at colocated boxes, but I'll setup another VPS with tripwire that's open at softlayer.

Interesting. Any details you would be willing to share about your ssh install would be appreciated (version, config, policies, etc.).

Apologies if I'm being thick, but how was the spam being sent on these boxes? Is the suspect perl script capable of sending email?

Is mysqld accessible from outside the machine or is it locked down to localhost? I'm still a bit confused by pr1soner's comment that they found a "data breach" and I'm wondering if somehow MySQL is playing a role here.

Have you had a look through what is running now to see if anything odd is either running or listening? You know, output like netstat -pane, lsof -Pwn, ps -axfwwwe.

Yea I've been lsof'ing all night :(. I dont see any more connections, no authorized keys, etc.. I nmaped known affected boxes, etc..

nothing really special on this... Protocol 2, No root logins...
openssh-server-4.3p2-41.el5_5.1


Basically, the perl script was running its own instance of Exim and injecting mail right into the queue from what I could tell. Exim is on the server but not running as a daemon.


mySQL was not available to the outside world.



yes, and there's nothing extra running at all on the machine nor at the time of the drop.

If either of you want extra eyes looking at the log files or the lsof/netstat/ps output, feel free to post them or contact me directly and I can get them put somewhere where some of the more experienced people around here can look. Please note I'm not questioning anybody's abilities, just offering some additional brain cells/input.

I appreciate that... I've been doing this since '93 so no shortage of experience and ability but sometimes even a 2nd set of eyes can see things that I miss. (anyone who figures they're above a 2nd set of eyes in this business and in this situation is best to get out before someone loses a server) :)

Having said that, if they'd leave more of a clue besides what I posted, it would help. I've installed CSF on the boxes in question at this point to see what we can snare. I'll let you know if I get anything else...

ssh passwords? I see mention of v2 but not whether password auth is enabled.