LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 09-10-2010, 03:13 PM   #31
nomb
Member
 
Registered: Jan 2006
Distribution: Debian Testing
Posts: 675

Rep: Reputation: 58

I still feel like we are missing a piece of the puzzle. I can't seem to go from A to B to C to D etc.

I have only been glancing the thread because there are always a lot of helpful people for these things but from what's presented I'm gathering:

server compromised (somehow)
openssh trojan installed (needed root)
changes in sshd, sshd_config (needed root)
found /dev/httpd <- containing ssh logins (needed root)

I can't go from the compromise to the trojan install. If joomla was exploited it should have been in the web user. If mysql was exploited it should have been in the mysql user. Where is the privilege escalation?

I'm guessing there is no hids (aide, tripwire) to check for file modifications against a known good ro copy?

nomb
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-10-2010, 03:41 PM   #32
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
Quote:
I can't go from the compromise to the trojan install. If joomla was exploited it should have been in the web user. If mysql was exploited it should have been in the mysql user. Where is the privilege escalation?
You're right, initially that is probably where they were. At this point there is no hard evidence on the exploit used, so this is all guesswork, but it may have been something along the lines of this advisory. If I'm understanding it right, and I may not be, they got access as mysql or apache and then could use the PAM/openssh exploit to escalate to root. The other thing to keep in mind is that the server was not up-to-date with patches, so the escalation may have used something besides openssh and we just don't have any evidence for it. While it would be nice to do some forensics on the box, given that there is clear evidence of a compromise, it is probably better to get it off the network and worry about forensics later.

Quote:
I'm guessing there is no hids (aide, tripwire) to check for file modifications against a known good ro copy?
There has been no suggestion that something like that exists. The rpm -Vv command kind of heads in that direction in that it verifies the running system against what is supposed to be installed. If you look at the output it is very clear that sshd_config and sshd are not what they should have been. It is certainly conceivable that biotones did change sshd_config from the default, but the fact that sshd itself is wrong is a major, major red flag.
 
Old 09-10-2010, 04:17 PM   #33
nomb
Member
 
Registered: Jan 2006
Distribution: Debian Testing
Posts: 675

Rep: Reputation: 58
Quote:
Originally Posted by Hangdog42 View Post
If I'm understanding it right, and I may not be, they got access as mysql or apache and then could use the PAM/openssh exploit to escalate to root.
That is also what I got from reading it.

Quote:
Originally Posted by Hangdog42 View Post
The other thing to keep in mind is that the server was not up-to-date with patches, so the escalation may have used something besides openssh and we just don't have any evidence for it.
I was kinda of headed in that direction also. Yes it wasn't up-to-date but it was CentOS 5. Or wasn't it RHEL 5. Either way that pam exploit is from 2006 and affected 3. I couldn't imagine they wouldn't have fixed it in two releases. But I guess there is always going to be more so it could be something similar.[/QUOTE]

Quote:
Originally Posted by Hangdog42 View Post
While it would be nice to do some forensics on the box, given that there is clear evidence of a compromise, it is probably better to get it off the network and worry about forensics later.
Very true. Damage control is far more important. But as you said there is always later. An image or something of the sorts could be taken to examine at a later time. It's important to make sure that the boxes aren't re-exploited. And while stopping the initial exploit would have definitely improved the chances of stopping the box from being compromised. Stopping the privileged escalation could have very well had a similar effect.

Quote:
Originally Posted by Hangdog42 View Post
There has been no suggestion that something like that exists. The rpm -Vv command kind of heads in that direction in that it verifies the running system against what is supposed to be installed. If you look at the output it is very clear that sshd_config and sshd are not what they should have been. It is certainly conceivable that biotones did change sshd_config from the default, but the fact that sshd itself is wrong is a major, major red flag.
Yes that was obvious. Which is what led me to suggest pulling out a known good ro copy of a hids run, and test it against the running system now to see what else was possibly modified. Being that now the logins were attained, it's possible that the other boxes where just logged into directly, and perhaps other means where used to maintain access.

But that depends on the admin have already been using a hids which if I had to guess I would guess not. You could test all of the rpms but that wouldn't really be complete.

Anyway just suggestions.

nomb
 
Old 09-11-2010, 07:32 AM   #34
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
Quote:
Originally Posted by nomb
Very true. Damage control is far more important. But as you said there is always later.
I'm going to qualify this just a little bit. In this case, damage control became the highest priority because there was pretty convincing evidence that the machine had been pretty thoroughly cracked. What we're trying to discourage here (and I know you're not advocating it) is the knee-jerk "Oh, you think you've been compromised, well then nuke from orbit and re-install". Biotones deserves some kudos for taking the advice that was offered, developing the evidence and then (hopefully!) taking action based on the facts. You're right that the openssh exploit is an old one, and hopefully we'll get a chance to do some more forensics to figure out if it was that or something else.

Quote:
Originally Posted by nomb
Yes that was obvious. Which is what led me to suggest pulling out a known good ro copy of a hids run, and test it against the running system now to see what else was possibly modified. Being that now the logins were attained, it's possible that the other boxes where just logged into directly, and perhaps other means where used to maintain access.
It would be very nice to have HIDS data. At this point we're not even sure if we can trust the output of the various standard commands. Since root access was obtained, it is possible that packages besides ssh were replaced with modified versions designed to cover evidence of the crack. Doing an extensive rpm -Vv would potentially be enlightening.
 
1 members found this post helpful.
Old 09-11-2010, 07:45 AM   #35
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
...in addition to what's already been posted (Hangdog42 being the primary incident handler for this thread):
Quote:
Originally Posted by biotones View Post
Ill do it ASAP
I'm not saying you're not cooperating but it would help if you were verbose, exact and complete when responding. Doing so makes the whole process more efficient and may help reduce work done in the assessment phase. Right now we're missing out on a large chunk of information. Wrt adjacent servers the Operating System or which internal or external customers you cater to does not really matter: what matters is making certain integrity is maintained.

In your next reply, and let's hope that doesn't have to wait until monday, please re-read posts in this thread and confirm things wrt previously asked questions, please post a list of actions you took so far and detailed listings of "evidence" and log excerpts where applicable. Wrt providing information try to think of it as helping us help you.
 
Old 09-11-2010, 09:09 AM   #36
abefroman
Senior Member
 
Registered: Feb 2004
Location: lost+found
Distribution: CentOS
Posts: 1,430

Rep: Reputation: 55
Quote:
Originally Posted by Hangdog42 View Post
Doing an extensive rpm -Vv would potentially be enlightening.
What do you mean?

Do a rpm -Vv for all packages?

What do you look for in the output?
 
Old 09-11-2010, 09:57 AM   #37
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
Quote:
Originally Posted by abefroman View Post
What do you mean?

Do a rpm -Vv for all packages?

What do you look for in the output?

Yeah, that is pretty much what I mean, although you could probably limit it to packages with system-level commands. If you look at the output biotones posted:

Code:
# rpm -Vv openssh-server
........ c /etc/pam.d/sshd
........   /etc/rc.d/init.d/sshd
........   /etc/ssh
S.5....T c /etc/ssh/sshd_config
........   /usr/libexec/openssh/sftp-server
S.5....T   /usr/sbin/sshd
........ d /usr/share/man/man5/sshd_config.5.gz
........ d /usr/share/man/man8/sftp-server.8.gz
........ d /usr/share/man/man8/sshd.8.gz
........   /var/empty/sshd

Notice the S,5 and T in the output. According to the rpm man page, that means the file size (S), md5sum (5) and mTime (T) differ from what should be there based on the internal rpm database. While sshd_config might be changed by an admin, the fact that the sshd binary appears to be changed is really worrying.

Since this is pretty good evidence that openssh was altered, that raises the question if anything else critical was altered. Can we really trust the output from lsof, ps and netstat? They could have been replaced with cracked versions altered to hide the crackers activities.
 
2 members found this post helpful.
Old 09-12-2010, 08:04 AM   #38
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
@biotones

In looking at the lsof output you sent me, I see that all of the lighttpd processes are owned by root. Now while I don't have experience with lighttpd, I do know that it behaves like Apache in that it starts as root so it can grab system resources and then drops to an unprivileged user for operation. On my Apache server, lsof shows a mix of root and that apache unprivileged user. Do you have lighttpd configured to drop to an unprivileged user or is it running as root all the time?
 
Old 09-14-2010, 11:51 AM   #39
nomb
Member
 
Registered: Jan 2006
Distribution: Debian Testing
Posts: 675

Rep: Reputation: 58
Quote:
Originally Posted by Hangdog42 View Post
Since this is pretty good evidence that openssh was altered, that raises the question if anything else critical was altered. Can we really trust the output from lsof, ps and netstat? They could have been replaced with cracked versions altered to hide the crackers activities.
Exactly what I was getting at.
 
Old 09-14-2010, 05:24 PM   #40
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Since the OP hasn't responded in four days I have emailed him. Let's see if he does.
Please be reminded that solving an incident can only be done with input from the OP.
 
Old 09-15-2010, 03:50 AM   #41
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:


It appears that you're now saying that the system isn't critical, or maybe that it's less critical than initially thought.

To be honest, this is a bit upsetting. There are several threads every week that state a system has been compromised and that assistance is needed to resolve the issue. When advise is given, one of the first responses to almost every single thread is that the system is critical and it can't be removed from the network. You see, its a catch-22 type thing...if you're not willing to follow that advise, then you're probably not in as dire a situation as the thread subject line and post content suggests (which usually reads, "system compromised...help!!!!").

As unSpawn mentioned, hiding the issue from your customers will seriously cause problems later...trust is a HARD thing to earn with customers, and much harder to regain if lost. Again, there's also the issue of your compromised machine attacking other machines. IMO, it is very hard to argue the fact that a system is critical. If it were that critical, you'd probably have already had the proper security layers in place (and wouldn't be accessing a machine via SSH as root).

I'm betting that this machine is still online. That creates an ethical problem, IMO.
The server is indeed critical due it containing adds and video feed that is required in my main sites. The only way is by migrating all the content to another clean server. However i've no resource on doing it. Sorry for the misunderstanding.
 
Old 09-15-2010, 04:01 AM   #42
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Sorry guys for letting you waiting. I cant put the lighttp & sql down due data is still required by my main webserver. We are on the way to get a new server to replace this one. (this will take some times). How ever I'll will update whatever action that I've done. Thank you.
 
Old 09-15-2010, 04:47 AM   #43
abefroman
Senior Member
 
Registered: Feb 2004
Location: lost+found
Distribution: CentOS
Posts: 1,430

Rep: Reputation: 55
Is he saying he is keeping the compromised server online?
 
Old 09-15-2010, 05:01 AM   #44
jmc1987
Member
 
Registered: Sep 2009
Location: Oklahoma
Distribution: Debian, CentOS, windows 7/10
Posts: 879

Rep: Reputation: 113Reputation: 113
biotones im not a super knowledgeable at linux but one thing I can tell you is leaving the server online is of the worst you can do. Shut it down even if it means down time for your business or clients. I don't know what is on the server exactly but you could be feeding the attacker all the information of your business, clients etc.

If you can't transfer to a clean server then maybe try downloading a full backup of the server and then have them wipe the server harden it and reupload files. Then load your back up on a isolated system home pc with out network access or whatever to do further investigation.

If you do go this route be sure to use different strong keys, passwords, etc that are not the same as the previous server.

Last edited by jmc1987; 09-15-2010 at 05:04 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Website discrimination against linux or my computer invaded? dgoddard Linux - Newbie 10 08-23-2009 10:24 PM
LXer: Internet Café Invaded by Linux Desktop (Philippines) LXer Syndicated Linux News 0 11-20-2008 01:20 PM
Computers have been invaded by idiots iwasapenguin General 19 09-08-2007 10:09 AM
html page invaded by evil image! wucan General 6 10-25-2006 12:15 AM
SSH FreeNX server am I being invaded? dasbooter Linux - Security 6 04-26-2006 04:30 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 08:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration