I'm already running OSSEC on a bunch of them, but no useful info. I'll take a look at tripwire for some as well. External/internal nmaps of the server dont yield any unexpected information.

Dman1, let me start by offering you a belated welcome to LQ. I notice that you have appended to an existing thread with a similar problem. Unfortunately, that thread is a poor representation of LQ and especially LQ-Security as it contains a bunch of wild speculation and cryptic suggestions about covert employees. Instead of continuing this one, I would really like to recommend that if you are serious about looking into this problem, that you start your own (new) thread on the subject. This is of course assuming that the re-load you reference in your previous thread did not "resolve" the problem. Note that I put "resolve" in quotes because unless you determine what and how you were seemingly compromised you run a real risk of repeating the problem.

At LQ-sec, we focus on a fact approach to performing an investigation. Typically we start by referencing the CERT Intruder Detection check list. Here is a link. It is an older document, but contains a lot of wisdom that I think would be especially helpful to solving the situation described in this thread.

From the description of the problem, it appears that there is a rogue root level process running on this system. These processes can be difficult, but not impossible to find. Lets recap some action items from the Cert checklist. If and when you do start your own thread, please focus on these items and especially with regards to the output of them. If you need help with any of the steps, we can provide that help. Here are some action items from the checklist to get you started.

0) Consider isolating the machine or machines that are infected with this problem. Allow yourself SSH access from a known clean host only. This will help cut the application off from its home base and increase your chances of finding it.

1) Your log files may contain valuable information. The problem with a root level compromise is that they can be altered, even by the problem program. See if you can use a write-once medium to have a copy of the log entries directed to it. See the following page for a possible suggestion: link.

2) This appears to be a root level compromise. Find all files and folders owned by user/group root and any users that belong to this group. Also find any files with the setuid and setguid bits set. *This one is important and I suspect you will 'see' your target, though you may not recognize it at this step.

3) Using this list, verify each of these files against the version contained in your package libraries. Don't assume that any of them are clean. Verify them with a KNOWN CLEAN tool, such as a binary directly from the repository or one on another system.

4) look at your crontabs, for users and roots. See what programs are running, what connections your system is trying to make. Look at the output of ps, netstat, lsof, etc. (e.g. output of /bin/ps axfwwwe; lsof -Pwln; netstat -antupe; who) *NOTE: you will want to verify the legitimacy of these binaries first as they could cover up a rogue entry).

5) look VERY carefully for hidden files, folders, and things in places like user's folders, /tmp folders, and and other locations that are common for wierd file names. **my suspicion is that you will find something here as your OSSEC is not seeing changes to 'normal' system files. **

2 members found this post helpful.

@Noway2,

Thanks for the welcome. OSSEC should be finding suid/guid 0 files on the whole file system (i tested this.) I've been checking over logs, enabled lots of extra exim logging. I just tried ordering a VPS (normally i host my own VPS on my own colocated nodes) from softlayer's cloud hosting. Fresh cpanel install and added it as a hostname of one of my domains and put up a few domains on it via cPanel. Looked on it today and it's having the same problem now. This was a completely clean setup at a totally different provider a thousand miles away from my other provider.


2011-01-31 10:03:27 1PjwDu-0003L3-SM <= root@cloud.MYHOSTNAMEGOESHERE.com U=root P=local S=671 T="Hi Joy Michael. This is Diana. Wanna date?" from <root@cloud.MYHOSTNAMEGOESHERE.com> for STRIPPEDEMAILADDR@yahoo.com
2011-01-31 10:03:27 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1PjwDu-0003L3-SM
2011-01-31 10:04:28 1PjwDu-0003L3-SM == STRIPPEDEMAILADDR@yahoo.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host
2011-01-31 11:52:52 1PjwDu-0003L3-SM == STRIPPEDEMAILADDR@yahoo.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host
2011-01-31 11:54:59 1PjwDu-0003L3-SM == STRIPPEDEMAILADDR@yahoo.com R=lookuphost T=remote_smtp defer (-53): retry time not reached for any host
2011-01-31 15:11:00 cwd=/var/log 3 args: exim -Mrm 1PjwDu-0003L3-SM
2011-01-31 15:11:00 1PjwDu-0003L3-SM removed by root
2011-01-31 15:11:00 1PjwDu-0003L3-SM Completed

Also the method being used must be new as there are no more encrypted files in /tmp, and lsof isnt helping me much at this time.

Also I wanted to add that this seems to have stopped happening on my main nodes after some strict firewall rules. I really feel that there is some 0day activity happening. I just can't find any actual rootkit. Here is the nmap output of the new VPS (done externally):

PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
25/tcp open smtp
53/tcp open domain
80/tcp open http
110/tcp open pop3
143/tcp open imap
443/tcp open https
465/tcp open smtps
623/tcp filtered unknown <-- ASF
664/tcp filtered secure-aux-bus <-- ASF
993/tcp open imaps
995/tcp open pop3s
2077/tcp open unknown <-- cPanel Servce
2078/tcp open unknown <-- cPanel Servce
2082/tcp open unknown <-- cPanel Servce
2083/tcp open unknown <-- cPanel Servce
2086/tcp open unknown <-- cPanel Servce
2087/tcp open unknown <-- cPanel Servce
2095/tcp open unknown <-- cPanel Servce
2096/tcp open unknown <-- cPanel Servce
3306/tcp open mysql
32883/tcp open unknown <-- SAN Service
38697/tcp open unknown <-- SAN Service
47266/tcp open unknown <-- SAN Service
48000/tcp open unknown <-- SAN Service
48001/tcp open unknown <-- SAN Service

Seems more people are being affected by this:

http://www.google.com/search?hl=en&s...&aqi=&aql=&oq=

It's possible my workstation is being affected and used to gain access. I do keep up with AV updates (avast) and I use ssh keys for everything. I dont even know what the passwords are. If I ever need the root password I just reset it to a 600+bit one.

Those ports are exposed to public?

Yes it would have to be for cPanel, Apache HTTP, Exim MTA, etc.. I could lock MySQL down to localhost I suppose.

What I do know in our instance is that we suspended all accounts on server and the wanna date spam continued, so the updated email list (which changed daily) was not submitted through a script as we originally believed, and surely had to be sent through some root level server exploit.

What is odd, is that this starts as lots of email to yahoo.com then on another day live.com or hotmail.com becomes the culprit.

That said we gave up and just wiped server. I honestly doubt this has anything to do with your workstation (dman1) since we changed root pass several times during the process of review, and no one else was logged in as root throughout.

Somewhat luckily this all happened on a server with just a few accounts, so we were able to play with server a great deal throughout.

Side note, we have a great deal of security expertise, so those of you stating "get a server admin" are sadly mistaken.
Server was fully PCI compliant by McAfee Secure and other PCI compliant company standards.

As for "Noway2" comments-- been there done that.
In our case, we went way beyond your recommendations/comments (having had over 6 different level 2/3 techs review separately, including cPanel's best).

Wish I could provide more info. Our researched ended at the wipe this past weekend.

Best Wishes,
Jim

Wow. This is an interesting problem. I would make a disk image of an infected system and get it into the hands of someone for study. (sort of like keeping small pox around)

It's disconcerting enough to have such a deeply rooted problem, but more so that as many eyes as have looked at this aren't finding the cause.

This would definitely be a case for tripwire, on a honey pot running the same system prior to infection. Maybe some tech school somewhere would be interested.

My thought about the open ports is why not forward most of that through ssh? Of course service ports need to be there, but admin that can be tunneled should be, it would seem.

Of course nobody has a clue yet what port(s) were exploited initially, and unfortunately it's most likely a public service, (and more likely mail) but who knows?

Only tripwire and a lot of logging would have a shot at conclusive proof.

The only IT college prof I know works for a local small town community college, but they do tout themselves as an IT destination. I'll ask him if he's interested in such a scenario or if he knows anyone who is.

I'm of the opinion something like this exploit can't be left an unknown. Even if it ultimately proves to be a small, previously unknown flaw in one of the components it's still a lesson to be learned. (like how to more swiftly find the source, perhaps)

We've been experiencing the same issue on a cPanel DNSONLY machine -- so there is VERY little running to start with.

A large binary encoded file gets dropped in /tmp and suddenly a perl script is running and injecting email into exim/sendmail. I managed to cat the /proc/XXXXX/environ before killing it...

SHELL=/bin/bash
SSH_CLIENT=62.xxx.xxx.102 40338 22
USER=root
MAIL=/var/mail/root
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
PWD=/root
SHLVL=1
HOME=/rootLOGNAME=root
SSH_CONNECTION=62.xxx.xxx.102 40338 x.x.x.x 22
_=/usr/bin/perl


We have no root logins set for the sshd server but this only happens when the firewall is down (apf kept knocking it down before I figured out what was going on). I'm starting to wonder if there's an exploit in the ssh server that we're not aware of. This happened on 2 other cPanel DNSONLY boxes we were running until we firewalled port 22.

Interesting, and that would be quite disturbing if ssh server is cracked.

Do you use password (ssh) log ins? Keys? Both?

If ssh is a problem I wonder if it would get sniffed out on a non-standard port. I always change ssh to something around 7000, but the service will report if the port is included in a scan. This really only stops the automated 'free download' teen/tween crackers from finding it on 22, useful enough to move the port, anyway.

Was firewalling port 22 the only change made? I'm trying to wrap my head around the common elements in this thread, and so far it appears to be (and please correct what is wrong):

CentOS/RHEL 5
cPanel
sshd
Exim

If the sshd idea is on track, one of the questions is going to be how widespread it is. If everyone is using the same sshd from CentOS, then are we looking at a distro-specific problem (like the one a Debian patch caused a few years ago) or is it a problem fundamental to openSSH?

You were saying this was a cPanel DNSONLY machine, and my understanding (which may be wrong) is that you're not likely to be running exim/sendmail on that. Is that correct or if not, what was running there?

OMG... I just checked the same thing and I think we're being affected by the same Netherlands IP range. Furthermore just a few hours after the cloud was created for me at softlayer (on the 17th) that IP range logged into the box:

root@cloud [~]# cat /var/log/secure* | grep 'Accepted' | grep -v '64.218' | grep -v '64.72.'
Jan 17 20:43:11 cloud sshd[28209]: Accepted password for root from 62.212.130.xxx port 60210 ssh2
Jan 17 20:43:14 cloud sshd[30122]: Accepted password for root from 62.212.130.xxx port 57673 ssh2
root@cloud [~]#

I don't see any more connections from it via SSHd though.

That is correct - the only change made was to firewall port 22. And the only time they got in, was when it was open.




Exim, itself, is NOT running on the DNSONLY boxes...

the processes are VERY minimal.

cpsrvd
named
mysqld (for cPhulkd)
cPhulkd
sshd

Yup, that's the one...

any big files in /tmp with odd names from the same time period?

There were on other affected boxes, this one there aren't any. Seems their method is changing. I'm dropping that whole range for now via iptables on the softlayer vps and at our router at colocated boxes, but I'll setup another VPS with tripwire that's open at softlayer.

Interesting. Any details you would be willing to share about your ssh install would be appreciated (version, config, policies, etc.).

Apologies if I'm being thick, but how was the spam being sent on these boxes? Is the suspect perl script capable of sending email?

Is mysqld accessible from outside the machine or is it locked down to localhost? I'm still a bit confused by pr1soner's comment that they found a "data breach" and I'm wondering if somehow MySQL is playing a role here.

Have you had a look through what is running now to see if anything odd is either running or listening? You know, output like netstat -pane, lsof -Pwn, ps -axfwwwe.