LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 11-03-2009, 10:30 AM   #16
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15

Quote:
Originally Posted by mase View Post
Problem is: An attacker might have had access to a root shell on your server, that means he could have done anything.

And it is easier to just reinstall from fresh media
than trying to track down what might have been changed,
especially on a running server.
You are of course right, mase... There's no question he DID have root access. But I see no way to figure out how or when he got it. The files involved in this attack date clear back to 2006. My server didn't even come into existence until 2008. So, it's pretty much impossible to determine when the initial penetration actually occured.

That means if I do nothing else for the next 60 days but rebuild the server and all 25 of the sites hosted there and do everything possible to tighten security, he could STILL be back 5 minutes after the server is up and running on the net again. I hate to admit that. But it's true.

Under those circumstances, what's the incentive and where's the justification to rebuild? I personally watched a huge multi-million dollar corporate webhosting firm with thousands of active servers deliberately and systematically ignore (and even hide) widespread server vulnerabilities and penetrations for years because there was NO way to remove them without admitting to thousands of clients that their server and network security sucked dead bears and wasn't worth the cost of the gunpowder it would take to blow it straight to hell.

That's why I went out on my own. I figured I couldn't do any worse than those guys were doing while I was paying them to do much better.

Last edited by websissy; 11-03-2009 at 12:15 PM.
 
Old 11-03-2009, 10:40 AM   #17
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Wim Sturkenboom View Post
@mase
Easier yes, and the attacker will, with the same ease, get in again if you don't know how he/she got in.
You're right, Wim. But let's face it we humans are eternal optimists. That's why we keep getting up every time we fall. ;-)
 
Old 11-03-2009, 10:45 AM   #18
Jim Bengtson
Member
 
Registered: Feb 2009
Location: Iowa
Distribution: Ubuntu 9.10
Posts: 164

Rep: Reputation: 38
He may well get in again. The only system that is 100% secure is not plugged in, encased in cement, and dropped to the bottom of the Marianas Trench...and then only because it's such a bother to go down and retrieve it.

So you can't make it 100% secure and still use it. But you can make the intruder respect you...
 
Old 11-03-2009, 12:03 PM   #19
mase
LQ Newbie
 
Registered: Jul 2008
Posts: 22

Rep: Reputation: 15
Quote:
Originally Posted by websissy View Post
You are of course right, mase... There's no question he DID have root access. But I see no way to figure out how or when he got it. The files involved in this attack date clear back to 2006. My server didn't even come into existence until 2008. So, it's pretty much impossible to determine when the initial penetration actually occured.

That means if I do nothing else for the next 60 days but rebuild the server and all 25 of the sites hosted there and do everything possible to tighten security, he could STILL be back 5 minutes after the server is up and running on the net again. I hate to admit that. But it's true.

Under those circumstances, what's the incentive and where's the justification to rebuild? I personally watched a huge multi-million dollar corporate webhosting firm with thousands of active servers deliberately and systematically ignore (and even hide) widespread server penetrations for years because there was NO way to remove them without admitting to thousands of clients that their server and network security wasn't worth the cost of the gunpowder it would take to blow it to hell.

That's why I went out on my own. I figured I couldn't do any worse than those guys were doing while I was paying them to do much better.
Well it is pretty clear I think that he got
in through a security hole, because you didn't upgrade.
(rather than maybe a configuration problem)
And since he got root rights and you can track the debian security
mailing list back and search for privilege escalation bugs as well as
bugs in one of your services,
that should be enough to know what it was.
I think there where only a few bugs of that kind.
And that's why I strongly believe, that installing a fresh and fully updated system will make you secure again.

Here is the mailing list, one of the thinks you should do is
subscribe to it:

http://lists.debian.org/debian-security/


Also you have to differentiate, up until this point the attack might
as well have been scripted. A script that runs somewhere with no user attention that automatically infects hosts with a specific security hole and installs shv5. At that point in time you might as well be able
to completely clean the infection.

But if an attacker actually accessed your server then you don't know what happened since he could have done anything. And in that case you will stay infected and none of your programs / services can be trusted anymore.
 
Old 11-03-2009, 12:06 PM   #20
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Jim Bengtson View Post
And that's only the ones you know about. I don't think you have any alternative but a full wipe and reinstall. This time, after you do the install but before you open it for business, try running Tripwire (or commercial version here) to make a baseline record of what your server looks like (file names, sizes, etc.). Then you can take regular snapshots and compare the current data to the original data...any files that have been altered will be listed, and if you can't explain the discrepancy then you know there may be a hacker at work.
Thanks, Jim. The reference to Tripwire is helpful. I'm also hopeful about COPS (ftp://ftp.cerias.purdue.edu/pub/tool...ps.1.02.README) and Tiger (http://packages.debian.org/lenny/tiger and http://www-arc.com/tara/) both of which I discovered at the very useful www.yolinux.com (see: http://www.yolinux.com/TUTORIALS/Lin...rityTools.html).

Tripwire has been added to my server security evaluation list.

Thanks.

Last edited by websissy; 11-03-2009 at 12:10 PM.
 
Old 11-03-2009, 12:11 PM   #21
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Jim Bengtson View Post
...you can't make it 100% secure and still use it. But you can make the intruder respect you...
Amen to that, Jim!

Last edited by websissy; 11-03-2009 at 12:13 PM.
 
Old 11-03-2009, 12:25 PM   #22
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by mase View Post
Well it is pretty clear I think that he got
in through a security hole, because you didn't upgrade.
(rather than maybe a configuration problem)
And since he got root rights and you can track the debian security
mailing list back and search for privilege escalation bugs as well as
bugs in one of your services,
that should be enough to know what it was.
I think there where only a few bugs of that kind.
And that's why I strongly believe, that installing a fresh and fully updated system will make you secure again.

Here is the mailing list, one of the thinks you should do is
subscribe to it:

http://lists.debian.org/debian-security/


Also you have to differentiate, up until this point the attack might
as well have been scripted. A script that runs somewhere with no user attention that automatically infects hosts with a specific security hole and installs shv5. At that point in time you might as well be able
to completely clean the infection.

But if an attacker actually accessed your server then you don't know what happened since he could have done anything. And in that case you will stay infected and none of your programs / services can be trusted anymore.
Once again, you're right. Blunt as a blackjack, perhaps... but right! With THAT said, I think I'll go have a left over lollipop and flip a coin... heads, we rebuild... tails we have another lollipop...

Last edited by websissy; 11-03-2009 at 12:39 PM.
 
Old 11-03-2009, 03:59 PM   #23
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Don't!

Quote:
Originally Posted by websissy View Post
If it has been there for months (I suspect it may have been), then rather than take 25 client sites down for weeks while I rebuild, I might conclude it's best to leave the compromised system running and keep client sites running (..) my small business customers really do NEED their sites to be up and accessible on the web. So, my goal here would be to avoid putting them completely out of the web business while I tackle this rebuild.
Before helping you with suggestions on how to assess the situation better I think I should first be allowed to change your perspective a bit here. Be warned this may come across as me lecturing you. That is not the case. Please try to see through that and just focus on the aspects I point at. I'd like to present you with three arguments that should lead to mitigating the situation as I asked you to consider before, looking at the situation from the vantage point of:

0. You
Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Trustworthy, stable and secure enough for many companies and institutions to rely on for critical processes. What most people forget is that running a GNU/Linux machine is also a responsibility. What you have is a root compromise. This simply means the machine is no longer under your control. You are responsible for mitigating that situation swiftly and decisively.

1. Us
Again: you have a root compromise. Without you controlling the machine it may be or may have been used to attack other machines over the 'net. As such the machine not only poses a threat to you but also to us.

2. Your business
Best practices state a compromised machine remains untrusted and must not be used. In a root compromise scenario any communications may be sniffed and any data on the server may have been siphoned off. This includes not only databases containing commercial and private information but also personal or company account details which may be sold or abused or otherwise used to damage business. If you run a company then you (may) have (paying) clients who trust you. Not timely alerting them of the compromise, not allowing them the opportunity to stop using the server or take measures will kill your business once the word gets out because trust will be lost. You know very well that trust is easier lost than regained.


If after reading this you feel that you still have good reasons to keep the server in use as-is, postpone any mitigating measures or otherwise stalling seems sensible please let me know.
 
Old 11-03-2009, 04:26 PM   #24
Jim Bengtson
Member
 
Registered: Feb 2009
Location: Iowa
Distribution: Ubuntu 9.10
Posts: 164

Rep: Reputation: 38
Quote:
What you have is a root compromise. This simply means the machine is no longer under your control.
More to the point, the OP is responsible for what his server does, even if the hacker initiates it. How understanding will his customers be if the server gets blackballed because it was used in a DDOS attack, or as a launching point on an attack on some other server?

Imagine the lawyers getting involved...?
 
Old 11-03-2009, 04:33 PM   #25
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Quote:
Originally Posted by websissy View Post
For the record, I did confirm direct replacement or impacts on ls, ps, top, dir, ifconfig, ttymon, ttyload, netstat, pstree, lsof, find, md5sum and other system components.
Thanks. (Not to promote things too much but for the upcoming release of Rootkit Hunter 1.3.6 I'm revisiting all known rootkits and adding in missing details. I've always thought of doing that as just making RKH as comprehensive as possible (as rootkit compromises have nearly died out), but your compromise shows me this wasn't a waste of effort. If you want to run RKH from CVS with all the new details you can get it with 'mkdir /tmp/rkh && cd /tmp/rkh && cvs -Q -z3 -dserver:anonymous@rkhunter.cvs.sourceforge.net:/cvsroot/rkhunter co -P rkhunter'.)


Quote:
Originally Posted by websissy View Post
can you please explain what you're referring to when you mention SysV startup files? (..) Are you referring to changes in the /etc/init.d/ startup files? If so, I'd say there's no question THAT could have happened.
Root compromise: unless you have safe and independent means of verifying integrity (Aide, Samhain or even tripwire, package management) all bets are off.


Quote:
Originally Posted by websissy View Post
From what I saw, I'd say it's likely both the openssh and openssl packages on my server have also been compromised by this rootkit. In fact, I'm convinced shv5 contains files designed to replace the server's public (and private?) ssh and ssl key files.
SHV5 installs SSHd and keys but not with the default names. SHV4 does but places it in /lib/security/.config/ssh/.
 
Old 11-03-2009, 04:49 PM   #26
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Quote:
Originally Posted by mase View Post
I think
This is the Linux Security forum. When doing proper incident response we should react to facts, not fiction. (For those who have done actual forensic investigations you know what I mean, you know what happens if an investigator starts to think or think up things. He gets biassed, starts to interprete things, is prone to dismissing clues or tries to match up available evidence to what he wants to see.) Please just be careful.


Quote:
Originally Posted by mase View Post
you can track the debian security mailing list back and search for privilege escalation bugs
That's what I thought of as well. Unpatched Debian has seen its fair share of SSL/SSH problems in the past 1-2 years.


Quote:
Originally Posted by mase View Post
And that's why I strongly believe, that installing a fresh and fully updated system will make you secure again.
No it won't. Recovery takes more than that ranging from ensuring adjacent systems are clean to changing all auth to properly hardening. And especially in his case most of the default hardening won't even depend on the outcome of an investigation.


Quote:
Originally Posted by mase View Post
At that point in time you might as well be able to completely clean the infection.
Even if you have a complete timeline of events and a full understanding of the compromise that is really, really bad advice. Please don't.
 
Old 11-03-2009, 06:05 PM   #27
mase
LQ Newbie
 
Registered: Jul 2008
Posts: 22

Rep: Reputation: 15
Quote:
Originally Posted by unSpawn View Post
This is the Linux Security forum. When doing proper incident response we should react to facts, not fiction. (For those who have done actual forensic investigations you know what I mean, you know what happens if an investigator starts to think or think up things. He gets biassed, starts to interprete things, is prone to dismissing clues or tries to match up available evidence to what he wants to see.) Please just be careful.

No it won't. Recovery takes more than that ranging from ensuring adjacent systems are clean to changing all auth to properly hardening. And especially in his case most of the default hardening won't even depend on the outcome of an investigation.
A system installed from scratch and updated without any services or anything yet configured is secure as in there won't be any traces of the rootkit left that's what I mean.

How to harden the system would have been another topic in itself.

Quote:
Originally Posted by unSpawn View Post
Even if you have a complete timeline of events and a full understanding of the compromise that is really, really bad advice. Please don't.
I didn't advice him to do it, I just told him that there is a possibility. He can't do what I said anyway since he has no means
of knowing if the attacker actually accessed the server.
That's why my advice would be to reinstall from scratch as I've told
him already.
 
Old 11-04-2009, 12:10 AM   #28
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn View Post
Before helping you with suggestions on how to assess the situation better I think I should first be allowed to change your perspective a bit here. Be warned this may come across as me lecturing you. That is not the case. Please try to see through that and just focus on the aspects I point at. I'd like to present you with three arguments that should lead to mitigating the situation as I asked you to consider before, looking at the situation from the vantage point of:

0. You
Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Trustworthy, stable and secure enough for many companies and institutions to rely on for critical processes. What most people forget is that running a GNU/Linux machine is also a responsibility. What you have is a root compromise. This simply means the machine is no longer under your control. You are responsible for mitigating that situation swiftly and decisively.

1. Us
Again: you have a root compromise. Without you controlling the machine it may be or may have been used to attack other machines over the 'net. As such the machine not only poses a threat to you but also to us.

2. Your business
Best practices state a compromised machine remains untrusted and must not be used. In a root compromise scenario any communications may be sniffed and any data on the server may have been siphoned off. This includes not only databases containing commercial and private information but also personal or company account details which may be sold or abused or otherwise used to damage business. If you run a company then you (may) have (paying) clients who trust you. Not timely alerting them of the compromise, not allowing them the opportunity to stop using the server or take measures will kill your business once the word gets out because trust will be lost. You know very well that trust is easier lost than regained.


If after reading this you feel that you still have good reasons to keep the server in use as-is, postpone any mitigating measures or otherwise stalling seems sensible please let me know.
I didn't feel lectured at all, Unspawn. Indeed, by the time you had made this post I was already in the process of developing a plan for how to move my existing web clients off this server while I rebuild it and then move them back again after that's done. In fact, when I spotted and read this post I was already talking with my clients about what a full-scale server evacuation and a rebuild and move-back strategy. So far, everyone has agreed to stick with me and has accepted my plan.

Frankly, I knew all along what the decision would probably be. But before I tossed out both the baby and the bathwater in a mad rush to get off the server, I needed to think the problem through and come up with a short a plan for how to make it work.

Thanks for the comments and feedback. You were right on the money.
 
Old 11-04-2009, 12:22 AM   #29
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn View Post
Thanks. (Not to promote things too much but for the upcoming release of Rootkit Hunter 1.3.6 I'm revisiting all known rootkits and adding in missing details. I've always thought of doing that as just making RKH as comprehensive as possible (as rootkit compromises have nearly died out), but your compromise shows me this wasn't a waste of effort. If you want to run RKH from CVS with all the new details you can get it with 'mkdir /tmp/rkh && cd /tmp/rkh && cvs -Q -z3 -dserver:anonymous@rkhunter.cvs.sourceforge.net:/cvsroot/rkhunter co -P rkhunter'.)


Root compromise: unless you have safe and independent means of verifying integrity (Aide, Samhain or even tripwire, package management) all bets are off.


SHV5 installs SSHd and keys but not with the default names. SHV4 does but places it in /lib/security/.config/ssh/.
chkrootkit and rkhunter at least hinted at the presence of both the SHV4 and SHV5 rootkits. From what I saw, I believe that's right. I've concluded both rootkits are present on my server. I've been in the profession a long time. I studied the impact of these 2 beasts fairly carefully. There are several relatively easy ways to identify them and I'd be willing to bet the files and directories they create are standard in name too. So, yes, I could probably offer some ideas about what to look for.

But if I posted those details here, they might well change tommorrow...
 
Old 11-04-2009, 01:01 AM   #30
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Jim Bengtson View Post
More to the point, the OP is responsible for what his server does, even if the hacker initiates it. How understanding will his customers be if the server gets blackballed because it was used in a DDOS attack, or as a launching point on an attack on some other server?

Imagine the lawyers getting involved...?
Screw the damned lawyers... Frankly, Jim, while you and I may take this seriously so do the huge hosting companies and their entire staffs from the CEO to the lowest support techs who collectively conspire to ignore, obfuscate and look the other way when their own dedicated and shared hosting servers and their vast corporate revenues are threatened.

Let's face it. I may be morally obligated to try to find a solution that works for my hosting customers; but no government agency is going to fund me and there are no prison sentences for small web hosts who choose to look the other way as they drive by either. The fact is there are hundreds (if not thousands) of compromised servers and millions of PCs around the world that have been similarly compromised by rootkits or zombieware. And ALL could be aimed at federal, state and local governments, police and media sites and financial institutions around the world and there'd be NOTHING short of pulling the plug on every computer on the planet that anyone could do to stop it. If you don't believe that's true, ask the financial institutions of Uzbekistan that were attacked and brought to their knees 3 years ago for over 2 weeks after the government approved the removal of a Soviet era Russian statue from an obscure municpal park or plaza.

There's no question the same thing could happen in America and if it did, it would eventually be proved that legions of zombies recruited from our own pc and server infrastructure had been used to mount such an attack. We've certainly seen lots of evidence for years now that those of us on the internet's front lines have been fighting the initial skirmishes of a cyber war. Do you see Homeland Security doing anything about it? Hell no! They'd much rather grope their way through suitcases filled with the panties of dangerous-looking grandmas at airports instead!

We've been lucky. So far we haven't experienced a large enough, long enough or devastating enough cyber-attack to provoke an arrogant and tech-naive president like George Dubyah - who had a John Wayne complex and advanced testosterone poisoning - into declaring hacking to be an act of war and starting a global war on hackers yet. But God knows that day could come tomorrow!

Meanwhile, I plan to keep slugging away here. Fighting off the bad guys as long as this tired old bod draws breath. I may take a cyber bullet or an rpg or two, but I'm a tough old buzzard. And the truth is I'd MUCH rather die at the front, wearing my boots and fighting a cyber war than to do it in bed in a nursing home somewhere!

Who knows? Maybe I'll be the first to be posthumously awarded the Congressional Medal of Cyber-Honor! Come on you ayrab-commie-hacker-bastards... come right-on down here to my old adobe computer room with the 18" thick walls and dig me out!! Ratatatatatat!! KABOOM!!!

Last edited by websissy; 11-04-2009 at 01:23 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
think i got hacked phatbastard Linux - Security 7 12-01-2007 01:36 PM
hacked? WRSpithead Linux - Security 2 08-30-2006 03:21 AM
Was my PC hacked? the-iguana Linux - Security 4 04-07-2006 08:38 AM
Have I been hacked? af_dave Linux - Security 3 07-14-2004 02:02 PM
HELP I think i got hacked spank Linux - Newbie 5 03-24-2004 08:59 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 04:38 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration