LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 09-06-2010, 11:32 PM   #1
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Rep: Reputation: 7
My Server was invaded


Hi all,
Iam currently having a security issue due to a uninvited intrusion happen few days back. I've been checking the server for any possible backdoor but i could not find any. anyway i've finally found a weird SSH login logger which will log any successful SSH connection"id & password" (scary). The file was located under /dev/ and named "httpd". Im totally blank now. Dont know where to start troubleshooting. Please help me on this issue. Thank You in advance.

OS: CENTOS 5 i386
Kernel: 2.6.18-8.el5PAE

Last edited by biotones; 09-06-2010 at 11:38 PM.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-07-2010, 01:01 AM   #2
marafa
LQ Newbie
 
Registered: Sep 2008
Posts: 26

Rep: Reputation: 1
you start by taking that server offline

next, you use a live cd to analyse the server and to search for any other applications. (you may have found an application that logs successful ssh connection id and password but you havent figured out the HOW it arrived there!). fedora has a security spin for forensics. there are others you can find on distrowatch.

this of course doesnt sound practical, especially if this is a mission critical server, so you will need to replace the server while you do your forensics. failing that, in case this is a webhost, you are going to make your own tough decisions.
 
0 members found this post helpful.
Old 09-07-2010, 02:56 AM   #3
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by marafa View Post
you start by taking that server offline

next, you use a live cd to analyse the server and to search for any other applications. (you may have found an application that logs successful ssh connection id and password but you havent figured out the HOW it arrived there!). fedora has a security spin for forensics. there are others you can find on distrowatch.

this of course doesnt sound practical, especially if this is a mission critical server, so you will need to replace the server while you do your forensics. failing that, in case this is a webhost, you are going to make your own tough decisions.
Dude thx 4 the reply, how ever this is indeed a critical web hosting server. The server still running due the other servers required files access on it. How do detect any script or software that doing the logging?
 
Old 09-07-2010, 03:01 AM   #4
giammy
Member
 
Registered: Feb 2010
Posts: 36

Rep: Reputation: 20
hi,

well: the better thing will be to recover an old backup you know to be clean.
Otherwise: server reinstallation, and data transfer being careful to what you transfer.

Obviously, this will be effective only after you found the original problem which let the intruder in!

bye
giammy
 
0 members found this post helpful.
Old 09-07-2010, 03:11 AM   #5
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by giammy View Post
hi,

well: the better thing will be to recover an old backup you know to be clean.
Otherwise: server reinstallation, and data transfer being careful to what you transfer.

Obviously, this will be effective only after you found the original problem which let the intruder in!

bye
giammy
Thx for the reply Giaamy,
Im doing a forensic task now. The sever was 1st invade by mySQL injection. It was running joomla and now i've remove the joomla main folder. Is there any forensic step is recon for this?
 
Old 09-07-2010, 04:11 AM   #6
giammy
Member
 
Registered: Feb 2010
Posts: 36

Rep: Reputation: 20
Quote:
Originally Posted by biotones View Post
Thx for the reply Giaamy,
Im doing a forensic task now. The sever was 1st invade by mySQL injection. It was running joomla and now i've remove the joomla main folder. Is there any forensic step is recon for this?
sorry, i'm not a forensic specialist.
Anyway, they probably used joomla to access the system: I think that removing joomla just removed the door they used: they probably installed some software in the system.

bye
giammy
 
0 members found this post helpful.
Old 09-07-2010, 06:23 AM   #7
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
The best thing for you to do is stop the activity with the server. If at all possible, isolate it from the network, but if it isn't too late do not reboot it as this may cause a loss of information. Next, start reading from this thread: http://www.linuxquestions.org/questi...erences-45261/ One of the things you should do is familiarize yourself with the CERT intruder detection checklist, perhaps run some of these tests on another known clean system to get an idea of what the results look like.

The activity, especially a strange file that logs ids and passwords, looks suspicious and you will need / want to take appropriate action. There are others on this forum that are very knowledgeable in this regard and will have questions for you and ask you to perform checks. Until then please don't disturb things any more than is possible.
 
2 members found this post helpful.
Old 09-07-2010, 07:16 AM   #8
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 418Reputation: 418Reputation: 418Reputation: 418Reputation: 418
Quote:
Originally Posted by giammy View Post
hi,

well: the better thing will be to recover an old backup you know to be clean.
Otherwise: server reinstallation, and data transfer being careful to what you transfer.

giammy

Please be aware that this forum works on a fact-based basis and offering this specific advice is frowned upon. As you noted, unless you've done an investigation, simply re-installing may be putting the original weakness back in place. Noway2 has the approach we're after.

Quote:
Originally Posted by biotones
Im doing a forensic task now. The sever was 1st invade by mySQL injection. It was running joomla and now i've remove the joomla main folder. Is there any forensic step is recon for this?
Can I ask what the basis for these decisions was? If you've got some evidence, we would love to see it so we can help.

Other things to look at:

The outputs of lsof -Pwn, netstat -pane and ps -axfwwwe.

Any suspicious log entries

A good resource for investigating is the CERT Checklist.

It would also help to have a better description of the server. What we're after is things like the state of patching, what services are being run, is it a VM guest and the intended use of the machine. Knowing about installed software like Joomla and its patch state would be good. Knowing a bit about its network environment may also help.


Also, since you suspect an SSH breach, if you can't pull the network cable, you should put a firewall in place that only allows SSH access from trusted IP addresses.
 
4 members found this post helpful.
Old 09-07-2010, 07:30 PM   #9
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
Quote:
Originally Posted by Hangdog42 View Post
Please be aware that this forum works on a fact-based basis and offering this specific advice is frowned upon. As you noted, unless you've done an investigation, simply re-installing may be putting the original weakness back in place.
I agree, however, a wipe & reload (even with the original vulnerability) is better than leaving the box online. Wipe & Reload might also bring in newer software, leading to a net security improvement.

(Forensic analysis & lockdown are best of course, but not everyone has the resources.)
 
Old 09-08-2010, 07:02 AM   #10
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 418Reputation: 418Reputation: 418Reputation: 418Reputation: 418
Quote:
Originally Posted by Matir
I agree, however, a wipe & reload (even with the original vulnerability) is better than leaving the box online. Wipe & Reload might also bring in newer software, leading to a net security improvement.
Certainly it is better than leaving an infected box online if there is absolutely no other option, but all too frequently wipe and reload is offered as the first thing to do, which is never right in my opinion. As for the forensic analysis, the kinds of stuff we ask for here tend to be fairly simple. If someone can't work their way through the CERT checklist or post the output of a few commands, they probably shouldn't be running a server.

I guess my concern is that the owner of infected boxes actually learn a little something about how to stop cracks in the first place, or detect them if they can't. I don't think a wipe and reload teaches them anything near enough about the security aspects of being responsible for a server.
 
1 members found this post helpful.
Old 09-08-2010, 07:35 AM   #11
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
Please be aware that this forum works on a fact-based basis and offering this specific advice is frowned upon. As you noted, unless you've done an investigation, simply re-installing may be putting the original weakness back in place. Noway2 has the approach we're after.



Can I ask what the basis for these decisions was? If you've got some evidence, we would love to see it so we can help.

Other things to look at:

The outputs of lsof -Pwn, netstat -pane and ps -axfwwwe.

Any suspicious log entries

A good resource for investigating is the CERT Checklist.

It would also help to have a better description of the server. What we're after is things like the state of patching, what services are being run, is it a VM guest and the intended use of the machine. Knowing about installed software like Joomla and its patch state would be good. Knowing a bit about its network environment may also help.


Also, since you suspect an SSH breach, if you can't pull the network cable, you should put a firewall in place that only allows SSH access from trusted IP addresses.
Thx alot guys, I'll upload the requested file asap. Im actually away from office due some urgent matter. Sorry for the late reply.
 
Old 09-08-2010, 10:16 AM   #12
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Hi Hangdog,
There's no VM running. I Believe the server is far from latest patches. I dont believe its a SSH breach because I've limit the ssh connection only from few IPs. BTW i've run the command and theres nothing on logging the ssh connection. pif...
 
Old 09-08-2010, 10:28 AM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Quote:
Originally Posted by Matir View Post
I agree, however, a wipe & reload (even with the original vulnerability) is better than leaving the box online.
While he's pretty much capable of putting the idea forward I'll still wedge in a reply to say I agree with Hangdog42 completely.

This is not about "net security improvement" or allocating resources but about the right approach to handling incidents. A "wipe and reinstall" may be a verdict, a conclusion, but never the starting point. LQ members who by reflex post "wipe and reinstall" (and often never return to the thread) should not post that but go read incident handling threads here and the standard docs. Also this is not up for discussion. This is how we want incident handling to be done at LQ. Any member who still thinks "wipe and reinstall" is the way to go will find enough fora on the 'net where they won't care for more than that. Not here.
 
1 members found this post helpful.
Old 09-08-2010, 12:05 PM   #14
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 418Reputation: 418Reputation: 418Reputation: 418Reputation: 418
Quote:
Originally Posted by biotones View Post
Hi Hangdog,
There's no VM running. I Believe the server is far from latest patches.
I dont believe its a SSH breach because I've limit the ssh connection only from few IPs. BTW i've run the command and theres nothing on logging the ssh connection. pif...
Could you explain this last bit in a bit more depth? Are you saying that the logging command you found isn't actually logging? That might be a good thing, but it still doesn't explain how it got there and unless you do a bit more investigating, you don't know if other compromises have happened.
 
Old 09-08-2010, 04:10 PM   #15
nomb
Member
 
Registered: Jan 2006
Distribution: Debian Testing
Posts: 675

Rep: Reputation: 58
Hangdog,

Kind of a semi-on-topic response being I really didn't quite understand your last post either; but I was kind of expecting you to post the output from those commands for the rest of us to look at?

Also, if you didn't delete /dev/httpd which is the suspected ssh logger, you can use it's timestamps to perhaps help narrow down in the logs where to start looking?

nomb
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Website discrimination against linux or my computer invaded? dgoddard Linux - Newbie 10 08-23-2009 10:24 PM
LXer: Internet Café Invaded by Linux Desktop (Philippines) LXer Syndicated Linux News 0 11-20-2008 01:20 PM
Computers have been invaded by idiots iwasapenguin General 19 09-08-2007 10:09 AM
html page invaded by evil image! wucan General 6 10-25-2006 12:15 AM
SSH FreeNX server am I being invaded? dasbooter Linux - Security 6 04-26-2006 04:30 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 06:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration