What to do when someone compromises your system ?
 

The other day we were at work and the connection got really slow, so i logged in to the server to see who was eating all the bandwith and accidentally found out user miguel logged in. It turned out miguel was logging in from Rumania and he was installing an IRC bot (to control the server remotely i found out later).

I am not in charge of maintaining the server, but i do like tweaking the systems i use. In windows i would know what files to edit, where to look, what services to disable, etc, in order to kick miguel's butt far away from the server. But i have been happily using linux for only 11 months and my weapons are not so sharp.

The server is running a Debian Sarge with a 2.4.24 kernel, and here is what i did:


#w gave me his user name, terminal, and ip.

#ps -ax | grep sshd gave me the process number for his ssh connection

#kill -9 {process} kicked him out of the system

I deleted his user and some other users we didn´t create, changed all the remaining user's passwords to more difficult ones (i´m sure that´s how he got in).

I checked what ports where open with #nmap localhost, looked for the most strange services on etc/rc0~5.d/ and disabled them.

I tried to check his bash history but there was nothing there. After that I was told to run rkhunter to check if the sistem was compromised, everything was ok.

---

So here are the questions:

1- What else should i have done ?

2- If a user logs in, how can i know what he is doing ? w gives little information.

3- .bash_history is not written until the user logs off, correct ? does linux keep a temporal file i can read while the user is logged in ?

4- Which files tell linux what to load on start up, which services will be running, etc.

5- Which are linux default services, a nmap to the server from outside makes me think there can be a hole or two or three or...

21/tcp open ftp
22/tcp open ssh
110/tcp open pop3
111/tcp open rpcbind
113/tcp open auth
137/tcp filtered netbios-ns
138/tcp filtered netbios-dgm
139/tcp filtered netbios-ssn
143/tcp open imap
199/tcp open smux
265/tcp filtered maybeFW1
354/tcp filtered bh611
445/tcp open microsoft-ds
548/tcp open afpovertcp
631/tcp open ipp
646/tcp filtered unknown
993/tcp open imaps
1720/tcp filtered H.323/Q.931
1723/tcp open pptp
5432/tcp open postgres
5800/tcp open vnc-http
5900/tcp open vnc
27002/tcp filtered flexlm2

---


I would be glad to read and learn if you can point me in the right direction.

Thanks in advance.

The conventional wisdom says that a complete reinstall is the only way to be sure. You have no way to know what else he has changed.

Best of luck...

David

at a minimum you want to chkrootkit
http://www.chkrootkit.org/

Everything of consequence goes into /var/log/messages. If a regular user su's or sudo's, it will be logged and you can see what they did.

You need to shut down a bunch of those services and firewall your system. For example, VNC open is always a bad idea on any public server. Anyone needing VNC, it is through redirection through SSH. FTP needs to be shutdown immediatly. SCP (the secure copy over SSH) replaces FTP. (Again, unless you are a public FTP server).

I would make sure that ssh as root is disabled. I always do this, and then when it is a server I control, I move su from a path directory to a hidden directory somewhere on the system. If they can't su, they can't do a lot. I also change su to be runable from only one specific user.

I think a re-install depends on how critical the server is and what it is doing. A print server, monitor it and install intrusion detection. An FTP/SFTP server or email server or anything critical...you get the idea.

Head to Debian headquarters, service configuration and firewall configuration should be right there.

RolledOat

How did you get so many services anyway? I see FTP, E-mail, a database, what I think is a samba server and more. And of course if you're attacker is creating his own users he managed to get root access somehow, I think a clean install would be best.

Genrally when installing debian I use the netinstall CD and select no packages, then I apt-get what I need.

P.S. try useing write to ask you're intruder how he got in, maby he'll tell you.

Sysadmin (using write)> How did you get in??

Intruder> Using an old version of write.


:p

I'm not totally familiar with Debian, but usually services on startup are stored as executable scripts in /etc/rc.d and look like "rc.sshd" or "rc.samba"

Try making some of those non-executable and, while the system is off, invoking a command like "/etc/rc.d/rc.samba stop" to shut down a service.

They're right, in that you can't really know what this guy did there, but a total reinstall can be a pain. Try shutting down some services. What is the server for anyway?

You might also try "killall vncserver" or something like that.

What else should i have done?
This being a business situation what you should have done (not "else") is alert someone that *is* responsable for the server. If you are not responsable then taking over control means you *become* responsable for the outcome, good or bad, and deleting an account is a good way to destroy information.

The first thing to do is make a log entry and note down information like time, machine name, commands performed and any information about the breach followed by alerting responsable personnel. If no responsable personnel can be contacted within a reasonable timeframe then how you act on it depends on your assessment of the situation. Does the server play a critical role in business? Does the situation cause a critical amount of damage? Does the situation allow the intruder to use the server as a springboard to other systems? If you have no idea you should first read some guidelines like the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html. The simplest way to mitigate the situation is logging as much information remotely as possible followed by a controlled shutdown of the server. While shutting down the server is bad for business it keeps the intruder from destroying "evidence" and keeps legal users from accessing the server if they could not be notified their account data was (possibly) compromised. At this point responsable personnel should take over. If no responsable personnel can be contacted within a reasonable timeframe you should then start by checking adjacent network devices for anomalies and preserve as much logging as possible.


I think a re-install depends on how critical the server is and what it is doing.
AFAIK repartitioning, reformatting and reinstalling from scratch depends on the severity of the compromise.


P.S. try useing write to ask you're intruder how he got in, maby he'll tell you.
Nice. Definately a good way to warn the user you're on to him.


They're right, in that you can't really know what this guy did there, but a total reinstall can be a pain.
Notice this is a business server and AFAIK businesses have no concept of subjective reasoning like "pain" is. What makes business respond is loss of operability, financial loss and loss of customer trust.

Thanks for the replies.

The server works as a proxy and samba for a windows network. There is a lot of information in that server and i would say it is of critial importance, but there is no real concern on security in the company (until something worse happens i suppose).

The person responsable for the server is not capable of securing his own machine, less he is of taking care of the server. I am not by far an expert in security, but as they say here, in the land of the blind the one eyed is king :P.

The contents of the user's folder were copied to another location prior to deleting his account. Inside his folder i found instalation files for psybnc (an IRC client/server), and a cron.d file pointing to this script

---

#!/bin/sh
if test -r /home/miguel/.rut.go.ro/psybnc.pid; then
pid=$(cat /home/miguel/.rut.go.ro/psybnc.pid)
if $(kill -CHLD $pid >/dev/null 2>&1)
then
exit 0
fi
fi
cd /home/miguel/.rut.go.ro
./start &>/dev/null

----

There are a lot of services running on the system, but i am not familiar with linux services and i don´t know which ones are normal or needed. The server has been administered by several people with poor linux or security, knowledge or concern.

If you tell me what information to post maybe you can give an idea of which services to shutdown.


P.S. i am reading the documents at www.cert.org, thanks for the link.

You should probably read the advice posted in the "unload suckit problem" thread (and actually take it seriously). Unfortunately, if the machine was root compromised there's really no way to recover in a completely safe way other than to reformat and reinstall. You can always boot off of a LiveCD to take critical data off the system before reformatting, but be careful you don't leave any nasty surprises left by the attacker.

I certainly hope that you notified that individual who is responsible for the server about the situation and what you've done. Having two people working at cross-purposes even if they are both trying to be helpful leads to mistakes being made. Keeping a log, as mentioned above, is an excellent idea.

If your company does not have the in-house skill to safely resolve the situation then you should consider hiring and outside consultant who does and retain them or pay for training for staff members so they can deal with these situations in the future. Whatever you do, do not brush this off. A proxy/file server is a trusted machine. Once compromised the attacker can use it to attempt to gain access to other machines on your local network. Furthermore, the attacker can use it to attempt attacks against other machines on the Internet. Depending on your exact situation, your firm may be held civilly or criminally liable for damages or denial of service caused by this (I Am Not A Lawyer, however, so don't construe any of this as legal advice -- just what I've read/experienced in my experience as a *nix admin).

A proxy server and SAMBA machine does not need FTP, VNC, etc. You need the samba port, the proxy ports and SSH open but (but ssh ONLY from the internal network). Everything else needs to be firewalled off from the outside world. Everything else is simply a security risk waiting to be accessed. Postgress open, IPP open, pptp open. Start with a firewall with ALL ports closed to the outside world, then allow SAMBA and the proxy ports only. Selectively open anything that is also required. Who from the outside world needs access to VNC?

I would not let this go with your management, at least you are asking the right questions. If I was doing it, I would start with a new server install, as mentioned, minimal install then add what you need fully firewalled, then start moving only the data and data only and when done/tested, replace the old one and blow it away.

RolledOat

look in the packlist for security tools. there are a LOT of them. track and trace your target. confirm and study the target. then plan your attack. I forgot where I got my firewall cracker from. If you attack right, there will be NOTHING left very soon. the only perfect defense is the perfect offense.

@operator10001:
track and trace your target. confirm and study the target.
With any action you take you will most likely be sending your IP to the remote network alerting the legal owner of the box and his/her ISP somebody is attacking a system on their network and alerting the cracker somebody wants to "play". If you manage to trace back activities to a remote box but do not have access to it there is nothing left to "study" and your trace goes cold there. Crackers do not usually operate from home and their intermediates are disposable, so asserting anyway (without proper evidence) the remote machine is "the crackers home" will prove to be wrong. What you will be "attacking" is a remote machine, legally owned by somebody (just like yours), which got cracked (just like yours). Nice catch is that if you do get access to the remote machine you're committing a crime yourself because because the owner of the box has not given you permission to access it.


then plan your attack.
More importantly there is no talk of "attacking". What you'll be doing is retalliating. Not only does the above show it's timeconsuming (without guarantee of resolving) and exposing yourself too, attacking can also be an crime according to local Law. IMNSHO by retalliating you waste time and effort opting for a childish "eye for an eye" attitude towards things instead of investing time and effort in priority number one: mitigating, rebuilding and hardening the box.


In short, this is "advice" which no security-conscious person should give and I request you to not post that kind of "advice" in our Linux Security forum. With all due respect, and such.



@landuchi:
Since you show no intention to mitigate and correct the situation according to what I see as standard procedure (see btmiller's reply) I will be closing this thread RSN, leaving the necessary reading material:
- Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
- Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html
- LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261
If you do intend to correct the situation reply what steps you took already and ask the right questions.

sorry to be offensive, i am an anarchist and i am blackhat. you seem to believe that it cant be done but, it has been done before.
operator

you seem to believe that it cant be done but, it has been done before.
LOL, I do not dispute it can be done. I just filled in some caveats (risks, MO, priorities) and pointed out why your "advice" is not "standard procedure" one should follow.


i am an anarchist and i am blackhat.
I try my best to be as impartial as I can so I do not care if you say you worship cAos or are deep into BO-fu: good for you! What I do care about is providing information that will help the OP and others that read the thread mitigate risks and recover from a situation like this, not find information with hidden risks or that will make him a target (again). Be aware LQ does not tolerate and does not want to be affiliated with anything that remotely looks like black hat ops in any way for obvious reasons. Next time please try to give advice that is helpful for the OP.

ROFL!

Anarchist: What people who find the idea of personal responsibility uncomfortable call themselves.
Black Hat: What skiddies call themselves.


BTW using a botnet to DDoS someone does not count as "hacking", and could you please explain to me what a "firewall cracker" does? Yeah, I thought so.