Hi, our server at school was compromised through a ssh dictionary attack and an irc daemon installed. The daemon has been removed,ssh and the firewall repaired/strenghtened, the compromised user account deleted and the files removed and as far as I can see, the the root account has not been compromised. The server does not contain any sensitive information or files that are vital. The server now appears stable. My question is whether I should undertake a time-comsuming reinstall or wait and see. I presume the text book answer is a reinstall, however given that the server does not contain sensitive information and that the system is stable and working, I do not feel it warrents the time spent.

One question: How was the irc daemon installed, how do you know the root account is not compromised?

Well, if you don't reinstall you need to monitor the server closely.
Consider using log monitoring tools like octopussy, splunk etc. Also monitor network traffic with appropriate tools.

The firewall is strengthened - good, what do you have now?
You do have a dedicated firewall I hope? And you're using an IDS?

1 members found this post helpful.

to get an internal mail system working I created users with the same usernames and passwords forgetting to reconfigure sshd_config allowusers settings making me vulnerable to a dictionary attack (even though the firewall only allows 2 login attempts every 3 minutes). Also the visible scripts found are of a generic nature suggesting that I was hacked and got programs installed by a computer. There is no evidence suggesting that the computer continued trying to get the root password once inside the system - though I am no expert.

I don't know that the root account has not been compromised, but everything suspicious has been run under the compromised user's account according to logfiles, tcpdump and wireshark. I have strengthened the firewall by only allowing login attempts from a designated ip and designated user.

We are not using IDS, but that sounds like a good idea - can you recommend a system/progam for ubuntu? also octopussy or splunk sounds great.

Ok I see how the intruder got in now.
Those users you created, they didn't have sudo-rights I hope? If so, and root password is good, I'd say it's not likely there's any danger in not reinstalling.

I get the feeling you don't have a dedicated firewall? But then, it's a server at school so there should be - or is i just that you personally don't have access to it?
If you really don't have a dedicated firewall then I say get one!
My personal preference is pfSense - BSD-based but with a very good web interface.

The only IDS I've used is snort, it does a good job.
All my firewalls are pfSense, and Snort can be run easily on it.

The users created did not have sudo rights and all of them are now deleted. We don't have a dedicated firewall on a seperate computer -dmz and all that. The compromised server runs a filewall/iptables amoung other services for clients based on a 2 nic setup (well 4 nic's in fact). As the ssh daemon is the only service we provide to the outside, I find setting up a dmz a bit overkill.

Getting time to do everything is a bit of a problem as I am responsible for the whole ict system with over 200 computers/devices with an 8% position dedicated to these tasks.

The preferred approach to (perceived) compromises of security in this forum is to deal with facts, analyze them and give advice based on that. A structured approach to handling incidents serves both "victim" and incident handler as guideline ensuring all aspects are addressed efficiently and in order.

Provided facts:
- The fact that a dictionary attack yielded access means:
0) pubkey auth was not used: review your knowledge of and adherence to security best practices,
1) that either syslogged warnings were ignored or no reporting is in place: see for example Logwatch and
2) that no mitigating measures were in place: see fail2ban.
- Using a production server to test functionality is not a best practice: use a staging server (VM?) as testbed instead.


Mitigation:
- Missing here is:
0) alerting everyone with an account on the machine to change their passwords regardless,
1) changing critical accounts passwords,
2) implementing security best practices as mentioned above,
3) the option to restore from a known secure and safe backup.


Assumptions / analysis:
- Deleting "evidence" without listing details and making backups first is a reflex that unfortunately new as well as seasoned admins succumb to. This does not help analyze the situation much, especially since Linux installations generally speaking do not come with much of a (comprehensive) audit trail out-of-the-box anyway. Without "evidence" a knowledgeable handler can, to some extent, rely on theoretical knowledge of the field and practical experience with prior intrusions. Even then, but this goes especially for those who are not well-versed in analysis, the danger of pitfalls like hypothesizing and speculation remains.
- The common MO of most of these crews is to only and efficiently hunt for low-hanging fruit. Often those machines are used to scan for other systems. That they are limited in their interests and time spent per system however should not be mistaken as "proof" the system is safe. Only proper analysis can confirm that it is.
- What is missing here is:
0) a time line indicating when brute forcing accounts started, when it yielded access and when you started mitigation,
1) how accounts, shell history, files, auth records and daemon / syslog entries involved in the compromise were analyzed,
2) a complete integrity check of the machine plus any adjacent ones.


Questions:
- Be aware that the questions you posed are not guided solely with security practices in mind but are, understandably though, colored by ulterior motives. Having minimal time to spend is not a reason to exclude any measures. Instead it means any investment of time must be efficient and effective: yield the largest possible ROI.

While I sympathize with your time and other constraints my advice would be to restore from a backup. Having one that is known to predate the intrusion and is known to be untainted gives you a valid reason to forego proper analysis on. Should you not have one then an installation from scratch ensures security and trustworthiness. After all running GNU/Linux is about performance, protecting assets and providing services in a continuous, stable and secure way.

thank you for the feedback.
I might add that I am the only user with access to the machine and as regards backups, I do take them regulary. However the last one was done after the machine was infected. I will have to check an off-site disk to see if I have an older one. Much of the machine can be restored using the cfengine pullserver, but a lot of tinkering and customisation would be lost.

The server was attacked on the 10th of January and I became aware of the attack on the 23rd. Evidence was secured and the most critical mitigation measures (deleting user accounts, tightened ssh) were taken immediately.

Advice given here will be implemented in the coming week and I will consider doing a reinstall when things are not as hectic as they are now at work (either from the pull server or from cfengine). As you point out without installling from stratch one cannot be entirely certain that the system is not compromised and, at some time in the future, could be brought down again with more serious consequences.