LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices



Reply
 
Search this Thread
Old 01-26-2013, 05:13 AM   #1
alistair_sunde
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Rep: Reputation: 0
compromised system: to reinstall or not


Hi, our server at school was compromised through a ssh dictionary attack and an irc daemon installed. The daemon has been removed,ssh and the firewall repaired/strenghtened, the compromised user account deleted and the files removed and as far as I can see, the the root account has not been compromised. The server does not contain any sensitive information or files that are vital. The server now appears stable. My question is whether I should undertake a time-comsuming reinstall or wait and see. I presume the text book answer is a reinstall, however given that the server does not contain sensitive information and that the system is stable and working, I do not feel it warrents the time spent.
 
Old 01-26-2013, 06:29 AM   #2
pingu
Senior Member
 
Registered: Jul 2004
Location: Skuttunge SWEDEN
Distribution: Debian preferably
Posts: 1,318

Rep: Reputation: 124Reputation: 124
One question: How was the irc daemon installed, how do you know the root account is not compromised?

Well, if you don't reinstall you need to monitor the server closely.
Consider using log monitoring tools like octopussy, splunk etc. Also monitor network traffic with appropriate tools.

The firewall is strengthened - good, what do you have now?
You do have a dedicated firewall I hope? And you're using an IDS?
 
1 members found this post helpful.
Old 01-26-2013, 07:03 AM   #3
alistair_sunde
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
to get an internal mail system working I created users with the same usernames and passwords forgetting to reconfigure sshd_config allowusers settings making me vulnerable to a dictionary attack (even though the firewall only allows 2 login attempts every 3 minutes). Also the visible scripts found are of a generic nature suggesting that I was hacked and got programs installed by a computer. There is no evidence suggesting that the computer continued trying to get the root password once inside the system - though I am no expert.

I don't know that the root account has not been compromised, but everything suspicious has been run under the compromised user's account according to logfiles, tcpdump and wireshark. I have strengthened the firewall by only allowing login attempts from a designated ip and designated user.

We are not using IDS, but that sounds like a good idea - can you recommend a system/progam for ubuntu? also octopussy or splunk sounds great.
 
Old 01-26-2013, 07:15 AM   #4
pingu
Senior Member
 
Registered: Jul 2004
Location: Skuttunge SWEDEN
Distribution: Debian preferably
Posts: 1,318

Rep: Reputation: 124Reputation: 124
Ok I see how the intruder got in now.
Those users you created, they didn't have sudo-rights I hope? If so, and root password is good, I'd say it's not likely there's any danger in not reinstalling.

I get the feeling you don't have a dedicated firewall? But then, it's a server at school so there should be - or is i just that you personally don't have access to it?
If you really don't have a dedicated firewall then I say get one!
My personal preference is pfSense - BSD-based but with a very good web interface.

The only IDS I've used is snort, it does a good job.
All my firewalls are pfSense, and Snort can be run easily on it.
 
Old 01-26-2013, 07:41 AM   #5
alistair_sunde
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
The users created did not have sudo rights and all of them are now deleted. We don't have a dedicated firewall on a seperate computer -dmz and all that. The compromised server runs a filewall/iptables amoung other services for clients based on a 2 nic setup (well 4 nic's in fact). As the ssh daemon is the only service we provide to the outside, I find setting up a dmz a bit overkill.

Getting time to do everything is a bit of a problem as I am responsible for the whole ict system with over 200 computers/devices with an 8% position dedicated to these tasks.
 
Old 01-26-2013, 10:54 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,766
Blog Entries: 54

Rep: Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976
The preferred approach to (perceived) compromises of security in this forum is to deal with facts, analyze them and give advice based on that. A structured approach to handling incidents serves both "victim" and incident handler as guideline ensuring all aspects are addressed efficiently and in order.

Provided facts:
Quote:
Originally Posted by alistair_sunde View Post
- We don't have a dedicated firewall on a seperate computer
- We are not using IDS
- The compromised server runs (..) iptables
- the ssh daemon is the only service we provide to the outside,
- the firewall only allows 2 login attempts every 3 minutes
- I created users with the same usernames and passwords forgetting to reconfigure sshd_config allowusers settings
- our server at school was compromised through a ssh dictionary attack
- The users created did not have sudo rights
- The fact that a dictionary attack yielded access means:
0) pubkey auth was not used: review your knowledge of and adherence to security best practices,
1) that either syslogged warnings were ignored or no reporting is in place: see for example Logwatch and
2) that no mitigating measures were in place: see fail2ban.
- Using a production server to test functionality is not a best practice: use a staging server (VM?) as testbed instead.


Mitigation:
Quote:
Originally Posted by alistair_sunde View Post
- I have strengthened the firewall by only allowing login attempts from a designated ip and designated user.
- The daemon has been removed,ssh and the firewall repaired/strenghtened, the compromised user account deleted and the files removed
- Missing here is:
0) alerting everyone with an account on the machine to change their passwords regardless,
1) changing critical accounts passwords,
2) implementing security best practices as mentioned above,
3) the option to restore from a known secure and safe backup.


Assumptions / analysis:
Quote:
Originally Posted by alistair_sunde View Post
- The server now appears stable.
- The server does not contain any sensitive information or files that are vital.
- an irc daemon installed.
- as far as I can see, the the root account has not been compromised.
- the visible scripts found are of a generic nature
- I was hacked and got programs installed by a computer.
- There is no evidence suggesting that the computer continued trying to get the root password once inside the system
- I don't know that the root account has not been compromised,
- everything suspicious has been run under the compromised user's account according to logfiles, tcpdump and wireshark.
- Deleting "evidence" without listing details and making backups first is a reflex that unfortunately new as well as seasoned admins succumb to. This does not help analyze the situation much, especially since Linux installations generally speaking do not come with much of a (comprehensive) audit trail out-of-the-box anyway. Without "evidence" a knowledgeable handler can, to some extent, rely on theoretical knowledge of the field and practical experience with prior intrusions. Even then, but this goes especially for those who are not well-versed in analysis, the danger of pitfalls like hypothesizing and speculation remains.
- The common MO of most of these crews is to only and efficiently hunt for low-hanging fruit. Often those machines are used to scan for other systems. That they are limited in their interests and time spent per system however should not be mistaken as "proof" the system is safe. Only proper analysis can confirm that it is.
- What is missing here is:
0) a time line indicating when brute forcing accounts started, when it yielded access and when you started mitigation,
1) how accounts, shell history, files, auth records and daemon / syslog entries involved in the compromise were analyzed,
2) a complete integrity check of the machine plus any adjacent ones.


Questions:
Quote:
Originally Posted by alistair_sunde View Post
My question is whether I should undertake a time-comsuming reinstall
- Be aware that the questions you posed are not guided solely with security practices in mind but are, understandably though, colored by ulterior motives. Having minimal time to spend is not a reason to exclude any measures. Instead it means any investment of time must be efficient and effective: yield the largest possible ROI.

While I sympathize with your time and other constraints my advice would be to restore from a backup. Having one that is known to predate the intrusion and is known to be untainted gives you a valid reason to forego proper analysis on. Should you not have one then an installation from scratch ensures security and trustworthiness. After all running GNU/Linux is about performance, protecting assets and providing services in a continuous, stable and secure way.
 
Old 01-26-2013, 11:42 AM   #7
alistair_sunde
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
thank you for the feedback.
I might add that I am the only user with access to the machine and as regards backups, I do take them regulary. However the last one was done after the machine was infected. I will have to check an off-site disk to see if I have an older one. Much of the machine can be restored using the cfengine pullserver, but a lot of tinkering and customisation would be lost.

The server was attacked on the 10th of January and I became aware of the attack on the 23rd. Evidence was secured and the most critical mitigation measures (deleting user accounts, tightened ssh) were taken immediately.

Advice given here will be implemented in the coming week and I will consider doing a reinstall when things are not as hectic as they are now at work (either from the pull server or from cfengine). As you point out without installling from stratch one cannot be entirely certain that the system is not compromised and, at some time in the future, could be brought down again with more serious consequences.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Has my system been compromised? towheedm Linux - Security 14 01-09-2013 01:34 PM
Has my System been Compromised? dman777 Linux - Security 2 05-17-2011 09:07 PM
Has my system been compromised? Drfarfrompuken Linux - Security 3 05-18-2007 06:58 PM
Help! My system's been compromised.... DaVenom Linux - Security 1 11-12-2004 03:49 PM
System compromised BruceCadieux Linux - Security 20 09-29-2003 09:24 PM


All times are GMT -5. The time now is 12:59 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration