LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 02-03-2011, 07:06 PM   #61
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582

If I may step in for a moment here, and I'm looking towards Hangdog42 for input here as he handles this problem with you all, I would like to suggest we keep issues separated. While I agree mitigation is essential so far this thread has yielded no solution that is backed by any tangible evidence. I'm not saying anyone shouldn't offer suggestions but rather than see this thread turn into a lengthy, in-depth discussion of that I'd rather see us focus on getting a grip on the cause.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 02-04-2011, 07:18 AM   #62
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 418Reputation: 418Reputation: 418Reputation: 418Reputation: 418
OK, if we're going to make any progress at all on this, I think we need to nail down the facts we're dealing with. That means that dman1 and tailtwister are going to need to post some details (and pr1soner if for some reason he is still paying attention to this). I'm sorry this may be a bit redundant with the earlier bits of the thread, but I'm suspecting we're starting to lose facts in the noise. Again, I'm not trying to tell people how to do their jobs, I'm just trying to organize the thinking with a different set of eyes. I know from hard experience that if you stare at a problem long enough, you can stop seeing the problem. What I'm thinking is:

CentOS version and install environment (hardware or VM and if VM kind and version)
Public network facing services including versions, configurations and particularly what users they are being run under
I know that SSH and cPanel are common elements here so particular attention (but not exclusive attention) to them. I think for SSH it would be good to run down what accounts have access and how, and how strong passwords are enforced (or not)

In addition, I think it would be a good idea to have some details about the operational state of the machines. So I think the outputs of lsof -Pwn, netstat -pane and ps -axfwwwe would be good. I know these are too big to post, so contact me and I'll get them someplace visible.

I know through process of elimination, SSH is the main suspect, however this is bugging me largely because if it was an SSH exploit, I would expect a LOT more chatter out there. Also, I think we've kind of given cPanel a free pass in this because of the result from tailtwister that blocking port 22 stopped the problem.

@tvcnet - Thanks for the idea, however I think that puts us in the place of playing catch-up. All the bad guys have to do is change "wanna date" to "want to date" and we're right back where we started. Filters definitely have their place, but I think we're better off trying to nail the underlying cause here.
 
Old 02-05-2011, 12:50 AM   #63
dman1
LQ Newbie
 
Registered: Dec 2010
Posts: 16

Rep: Reputation: 0
Hey guys. We had rolling blackouts here in Houston which knocked out my ISP's local distribution hardware for over 48 hours. Just got back on the net at home, been supper busy at the office.

So far all of the servers that have been affected on my end are VPS. I also have about 50-60 actual hardware servers some of which are nodes. Not all of the boxes were cPanel. Not all of the boxes were exim. I just did a review, there were some directadmin boxes and some cpanel and one webmin which i think was running sendmail. The operating systems are all up to date and have yum-cron running doing daily updates. On some boxes I utilize ksplice for reboot-less kernel upgrades.

It's all CentOS 5.5 latest everything, huge random generated passwords, ssh keys used for actual logins, each server or node has it's own ssh key, not a standard key accross all servers. All of the containers affected on nodes I own were OpenVZ. My XEN container at softlayer was affected. I use solusvm (latest) to control my XEN and OpenVZ nodes. I did recently find a pretty nasty root exploit which can DEFINITELY be used for local privilege escalation by a local user account. It can also be potentially used for remote privilege escalation if there is any way to execute system/exec/back ticks/etc via XSS, insecure script upload or may other attacks scenarios. I mentioned this to the solus people, but they didn't seem concerned. They just said not to let anyone else have any access to the nodes (ie. user level accounts) and mentioned that the security fail is required for users using old versions of vzctl. It is a really really bad exploit. I fixed it on my end just fine since i don't use an old vzctl version. I dont understand why they don't push out a vzctl update (even if optional) with an update, or why they dont check ur vzctl version upon install/update and fix this insecurity if ur current.
 
Old 02-05-2011, 08:08 AM   #64
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 418Reputation: 418Reputation: 418Reputation: 418Reputation: 418
Thanks dman1, that gives us a bit more to work with and hopefully tailtwister can build on this. So based on what you've posted, it looks like common elements in the compromised machines are:

CentOS 5.5
Solusvm
openSSH (by the way, do you allow direct root access via SSH?)

Possibly related: OpenVZ and Xen. Since these are such different technologies, it seems unlikely that they would share a common weakness, However the fact that virtualization is common, there is definitely a question about the host. Has any investigation been done into those or has investigation been limited to the guests?

Quote:
Originally Posted by dman1
I did recently find a pretty nasty root exploit which can DEFINITELY be used for local privilege escalation by a local user account. It can also be potentially used for remote privilege escalation if there is any way to execute system/exec/back ticks/etc via XSS, insecure script upload or may other attacks scenarios.
Can I ask what remedies were taken after this discovery (and what the exploit was)? Also, were the infected machines exploited in any way other than the install of the exploit itself? Is it possible this was the initial breach that allowed them enough access to compromise other machines?

Sorry, I missed in your initial post that solusvm was in use across compromised machines. Neither pr1soner nor tailtwister mention it, but it would be interesting to hear if either use it. Given their attitude about security, this seems a potentially good candidate.
 
Old 02-08-2011, 09:32 AM   #65
tailtwister
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
Sorry guys - life took some ugly turns on our family last week so any moment I wasn't dealing with servers, I was away from computers.

We've added csf to all of our servers as a result of the last invasion. Port 22 is blocked to all but known IPs. Only 3 accounts have shell access on any given server (all administrators). Keys are used (big, random passwords were also usable).

The first time this happened was within a couple of hours of installing a fresh server and before making any account changes, etc.

Each incident has resulted in a large, randomly named bundle (owned by root) being found in /tmp and a queue full of mail files. Sometimes the application is still running and injecting, sometimes it has run its course and there is no sign of it. Twice I've done a re-install from scratch to the server in case something was left behind (scans showed nothing out of the ordinary).

Since adding csf (like the previous method of blocking port 22), no further incidents.


Common on all servers:

- CentOS 5.5
- cPanel DNSonly
- Virtuozzo
- only Exim running

The only extra information I've been able to capture was posted earlier in this thread.
 
Old 02-09-2011, 07:18 AM   #66
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 418Reputation: 418Reputation: 418Reputation: 418Reputation: 418
OK, I'd like to toss out a slightly different idea here. Both tailtwister and dman1 are using virtualization, and it turns out they are using essentially the same software to create VPS. If I understand Virtuozzo and OpenVZ, they are identical or at least share a huge amount of the code base. So during the investigation, did you look into the host being compromised or just the guests?

The thing that bugs me about SSH being the vector is that if there were a bug in openSSH, I would have expected a lot more people to get hit by this, and from what I've seen, there isn't a lot of chatter out there about this (I'm very willing to admit I've missed something though). So maybe instead of an SSH problem, we're seeing a problem with the virtualization software, or some combination of virtualization with CentOS creating a vulnerability.
 
Old 02-09-2011, 01:41 PM   #67
tvcnet
LQ Newbie
 
Registered: Jan 2011
Posts: 7

Rep: Reputation: 0
Thumbs up

Hi folks,
Having worked with a number of servers involving this hack as part of our company's security service, it' fairly clear to our staff here that updating exim on the compromised cpanel servers resolved the issue.

From what we can tell after fixing several servers with this hack is that the following were not contributors:
- SSH or changing the SSH port
- whether server was VPS or not

Solution in each case was #4 in my post on January 4th: #55

Once you update Exim I believe you'll find the hack will stop.

If a finger can be pointed I suppose it's at cPanel for not alerting their client's to the threat in a more clear and direct manner. This is a very uncommon occurrence and somewhat out of character for cPanel.

I reported the situation over a month ago, which did result in them releasing a second update most recently but far to quietly for my tastes. IMHO - This should have been a raise the flag and ring the bell alert from cPanel.

Instead cPanel chose to quietly include the exim update in versions of cpanel which many would not have known to force an update too respectively.

Bottom line:
If you are hit by the wanna date spam hack, update Exim to the latest version available.
 
Old 02-09-2011, 03:07 PM   #68
dman1
LQ Newbie
 
Registered: Dec 2010
Posts: 16

Rep: Reputation: 0
Quote:
Originally Posted by Hangdog42 View Post
Can I ask what remedies were taken after this discovery (and what the exploit was)? Also, were the infected machines exploited in any way other than the install of the exploit itself? Is it possible this was the initial breach that allowed them enough access to compromise other machines?

Sorry, I missed in your initial post that solusvm was in use across compromised machines. Neither pr1soner nor tailtwister mention it, but it would be interesting to hear if either use it. Given their attitude about security, this seems a potentially good candidate.
SolusVM installs a binary that it uses with world writable permissions and suid . Works fine with a proper permission/ownership setup on my servers. They claim some old version of vzctl requires world writable permissions on their binary.

I don't use SolusVM on all boxes, the softlayer xen I dont think uses solus (I doubt they use it.)
 
Old 02-09-2011, 03:08 PM   #69
dman1
LQ Newbie
 
Registered: Dec 2010
Posts: 16

Rep: Reputation: 0
Quote:
Originally Posted by tvcnet View Post
Bottom line:
If you are hit by the wanna date spam hack, update Exim to the latest version available.
Servers that have exim + cpanel are running the latest exim updated via a forced run of /scripts/eximup. cPanel claims to have patched the exim bug when it was made public. I saw a sendmail server affected as well. I think it was running webmin or something.


Found another one. This container was affected without cpanel and without exim:

[root@420 log]# ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2160 664 ? Ss Jan10 0:02 init [3]
root 1999 0.0 0.1 2248 552 ? S<s Jan10 0:00 /sbin/udevd -d
root 3413 0.0 0.2 7208 1064 ? Ss Jan10 0:13 /usr/sbin/sshd
root 3852 0.6 0.5 14992 2756 ? Ss 07:09 0:00 \_ sshd: unknown [priv]
sshd 3854 0.0 0.2 8552 1352 ? S 07:09 0:00 \_ sshd: unknown [net]
root 3466 0.0 0.1 2836 872 ? Ss Jan10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 10121 0.0 0.3 9304 1884 ? Ss Jan10 0:19 sendmail: accepting connections
smmsp 13833 0.0 0.2 8256 1480 ? Ss Jan10 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 13847 0.0 1.3 17988 6960 ? Ss Jan10 0:01 /usr/sbin/httpd
apache 25964 0.0 0.7 18120 3820 ? S Feb06 0:00 \_ /usr/sbin/httpd
apache 9747 0.0 0.7 18120 3820 ? S Feb06 0:00 \_ /usr/sbin/httpd
apache 5975 0.0 0.7 18120 3820 ? S Feb07 0:00 \_ /usr/sbin/httpd
apache 5990 0.0 0.7 18120 3820 ? S Feb07 0:00 \_ /usr/sbin/httpd
apache 5991 0.0 0.7 18120 3820 ? S Feb07 0:00 \_ /usr/sbin/httpd
root 13855 0.0 0.2 4492 1104 ? Ss Jan10 0:00 crond
root 13864 0.0 0.1 5680 708 ? Ss Jan10 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root 13866 0.0 0.0 5680 440 ? S Jan10 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
root 3469 0.0 0.0 1996 304 ? Ss 07:09 0:00 vzctl: pts/0
root 3470 0.0 0.2 3716 1500 pts/0 Ss 07:09 0:00 \_ -bash
root 3882 0.0 0.1 2532 840 pts/0 R+ 07:09 0:00 \_ ps faux
[root@420 log]#


[root@420 /]# ll /tmp/
total 300860
-rw-r--r-- 1 root root 9177717 Jan 12 04:59 1DW4zEQspT
-rw-r--r-- 1 root root 12800732 Jan 21 00:55 A85IwI0UlL
-rw-r--r-- 1 root root 9273695 Jan 29 14:23 MRPeHfNRch
[root@420 /]#

They aren't all CentOS either. I found a Debian one affected today too.


Found this:

vz:/var/spool/mqueue# cat Qfp0V53S91031904
V8
T1296450208
K1296881870
N719
P64740385
I0/66/51773457
MDeferred
Fwbs
$_localhost.localdomain [127.0.0.1]
$rESMTP
$svz.XXXXXXXXXXXXXXXXXX.org
${daemon_flags}
${if_addr}127.0.0.1
S<root@vz.XXXXXXXXXXXXXXXXXX.org>
MDeferred
rRFC822; frnklnjac7@aol.com
RPFD:<frnklnjac7@aol.com>
H?P?Return-Path: <g>
H??Received: from vz.XXXXXXXXXXXXXXXXXX.org (localhost.localdomain [127.0.0.1])
by vz.XXXXXXXXXXXXXXXXXX.org (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p0V53S91031904
for <frnklnjac7@aol.com>; Mon, 31 Jan 2011 08:03:28 +0300
H?x?Full-Name: root
H??Received: (from root@localhost)
by vz.XXXXXXXXXXXXXXXXXX.org (8.14.3/8.14.3/Submit) id p0V53SxX031903;
Mon, 31 Jan 2011 08:03:28 +0300
H??Date: Mon, 31 Jan 2011 08:03:28 +0300
H??Message-Id: <201101310503.p0V53SxX031903@vz.XXXXXXXXXXXXXXXXXX.org>
H??From: gMu9clCrTM@exclusiveperks.com
H??Subject: Notification # 6097
H??To: frnklnjac7@aol.com
H??Mime-Version: 1.0
H??Content-type: text/plain
All OK!
______
//gMu9clCrTM
.
vz:/var/spool/mqueue#


Did a search on "frnklnjac7@aol.com" found this neat thread here:

http://www.linuxquestions.org/questi...server-854002/

Last edited by dman1; 02-10-2011 at 07:21 AM.
 
Old 02-10-2011, 07:51 AM   #70
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
dman1, CYP contact me by email to discuss and arrange drop-off for the /tmp files, complete listings (the "/var/tmp/log.txt" file) of running '(ps axfwwwe 2>&1; netstat -anpe 2>&1; lsof -Pwln 2>&1; who 2>&1; last 2>&1; rpm -Vva 2>&1|grep -v "^\.\{8\}"; logwatch --numeric --detail 5 --service all --range All --archives --print 2>&1; ) > /var/tmp/log.txt;' ?
 
Old 02-12-2011, 12:43 PM   #71
dman1
LQ Newbie
 
Registered: Dec 2010
Posts: 16

Rep: Reputation: 0
Sent you an email
 
Old 02-16-2011, 08:40 PM   #72
pr1soner
LQ Newbie
 
Registered: May 2010
Location: earth
Distribution: slackware
Posts: 10

Original Poster
Rep: Reputation: 1
hey guys, i've resolved the issue after investigating the problem, however, because the problem is sensitive, i will let the interested peoples know how to solve it only per PM request, i won't disclose it on public because the probability that the spammer itself may read it is high. i hope the moderators understand that.

i would sit quiet about it after i solved the case but i want this particular spammer to loose his incomes and credibility.

Best regards.

Last edited by pr1soner; 02-16-2011 at 08:41 PM. Reason: grammar typos.
 
Old 02-17-2011, 04:40 PM   #73
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Quote:
Originally Posted by dman1 View Post
Sent you an email
Please respond to my reply, TIA.
 
Old 02-17-2011, 04:41 PM   #74
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Quote:
Originally Posted by pr1soner View Post
i will let the interested peoples know how to solve it only per PM request
I sent you an email. Please respond, TIA.
 
Old 02-17-2011, 06:57 PM   #75
catworld
Member
 
Registered: Nov 2004
Location: Horseheads, New York
Distribution: Mandriva 2010.1 / KDE 4.5.2, Slax, Knoppix, Backtrack & etc...
Posts: 198

Rep: Reputation: 36
Quote:
Originally Posted by unSpawn View Post
I sent you an email. Please respond, TIA.
per pm req plz


Blog Entries: 40

Rep: Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655

Quote:
Originally Posted by pr1soner View Post
i will let the interested peoples know how to solve it only per PM request
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
server compromised? eco Linux - Security 3 09-03-2010 11:58 AM
my server has been compromised, what next? Kropotkin Linux - Security 15 08-27-2009 06:15 AM
Server Compromised? lss1 Linux - Security 7 12-16-2005 12:49 AM
Server Compromised? stlyz3 Linux - Security 6 09-07-2005 04:28 PM
Server was compromised, need help Asiana Linux - Security 3 06-02-2004 12:39 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 04:37 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration