LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Closed Thread
 
Search this Thread
Old 12-02-2006, 10:52 PM   #1
landuchi
Member
 
Registered: Oct 2004
Distribution: Debian, Ubuntu
Posts: 74

Rep: Reputation: 15
Question What to do when someone compromises your system ?


The other day we were at work and the connection got really slow, so i logged in to the server to see who was eating all the bandwith and accidentally found out user miguel logged in. It turned out miguel was logging in from Rumania and he was installing an IRC bot (to control the server remotely i found out later).

I am not in charge of maintaining the server, but i do like tweaking the systems i use. In windows i would know what files to edit, where to look, what services to disable, etc, in order to kick miguel's butt far away from the server. But i have been happily using linux for only 11 months and my weapons are not so sharp.

The server is running a Debian Sarge with a 2.4.24 kernel, and here is what i did:


#w gave me his user name, terminal, and ip.

#ps -ax | grep sshd gave me the process number for his ssh connection

#kill -9 {process} kicked him out of the system

I deleted his user and some other users we didnīt create, changed all the remaining user's passwords to more difficult ones (iīm sure thatīs how he got in).

I checked what ports where open with #nmap localhost, looked for the most strange services on etc/rc0~5.d/ and disabled them.

I tried to check his bash history but there was nothing there. After that I was told to run rkhunter to check if the sistem was compromised, everything was ok.

---

So here are the questions:

1- What else should i have done ?

2- If a user logs in, how can i know what he is doing ? w gives little information.

3- .bash_history is not written until the user logs off, correct ? does linux keep a temporal file i can read while the user is logged in ?

4- Which files tell linux what to load on start up, which services will be running, etc.

5- Which are linux default services, a nmap to the server from outside makes me think there can be a hole or two or three or...

21/tcp open ftp
22/tcp open ssh
110/tcp open pop3
111/tcp open rpcbind
113/tcp open auth
137/tcp filtered netbios-ns
138/tcp filtered netbios-dgm
139/tcp filtered netbios-ssn
143/tcp open imap
199/tcp open smux
265/tcp filtered maybeFW1
354/tcp filtered bh611
445/tcp open microsoft-ds
548/tcp open afpovertcp
631/tcp open ipp
646/tcp filtered unknown
993/tcp open imaps
1720/tcp filtered H.323/Q.931
1723/tcp open pptp
5432/tcp open postgres
5800/tcp open vnc-http
5900/tcp open vnc
27002/tcp filtered flexlm2

---


I would be glad to read and learn if you can point me in the right direction.

Thanks in advance.

Last edited by landuchi; 12-02-2006 at 10:55 PM.
 
Old 12-03-2006, 12:57 AM   #2
budword
Member
 
Registered: Apr 2003
Location: Wisconsin
Distribution: Switched to regualr Ubuntu, because I don't like KDE4, at all. Looks like vista on crack.....
Posts: 675

Rep: Reputation: 31
The conventional wisdom says that a complete reinstall is the only way to be sure. You have no way to know what else he has changed.

Best of luck...

David
 
Old 12-03-2006, 01:21 AM   #3
RolledOat
Member
 
Registered: Feb 2003
Location: San Antonio
Distribution: Suse 9.0 Professional
Posts: 843

Rep: Reputation: 30
at a minimum you want to chkrootkit
http://www.chkrootkit.org/

Everything of consequence goes into /var/log/messages. If a regular user su's or sudo's, it will be logged and you can see what they did.

You need to shut down a bunch of those services and firewall your system. For example, VNC open is always a bad idea on any public server. Anyone needing VNC, it is through redirection through SSH. FTP needs to be shutdown immediatly. SCP (the secure copy over SSH) replaces FTP. (Again, unless you are a public FTP server).

I would make sure that ssh as root is disabled. I always do this, and then when it is a server I control, I move su from a path directory to a hidden directory somewhere on the system. If they can't su, they can't do a lot. I also change su to be runable from only one specific user.

I think a re-install depends on how critical the server is and what it is doing. A print server, monitor it and install intrusion detection. An FTP/SFTP server or email server or anything critical...you get the idea.

Head to Debian headquarters, service configuration and firewall configuration should be right there.

RolledOat
 
Old 12-03-2006, 03:03 AM   #4
Tortanick
Member
 
Registered: Jul 2006
Distribution: Debian Testing
Posts: 299

Rep: Reputation: 30
How did you get so many services anyway? I see FTP, E-mail, a database, what I think is a samba server and more. And of course if you're attacker is creating his own users he managed to get root access somehow, I think a clean install would be best.

Genrally when installing debian I use the netinstall CD and select no packages, then I apt-get what I need.

P.S. try useing write to ask you're intruder how he got in, maby he'll tell you.
 
Old 12-03-2006, 08:40 AM   #5
jayjwa
Member
 
Registered: Jul 2003
Location: NY
Distribution: None (src & compile)
Posts: 253

Rep: Reputation: 36
Quote:
Originally Posted by Tortanick
P.S. try useing write to ask you're intruder how he got in, maby he'll tell you.

Sysadmin (using write)> How did you get in??

Intruder> Using an old version of write.


 
Old 12-03-2006, 09:17 AM   #6
Tralce
Member
 
Registered: Nov 2006
Location: Maine
Distribution: Debian, Ubuntu, Gentoo
Posts: 109

Rep: Reputation: 15
I'm not totally familiar with Debian, but usually services on startup are stored as executable scripts in /etc/rc.d and look like "rc.sshd" or "rc.samba"

Try making some of those non-executable and, while the system is off, invoking a command like "/etc/rc.d/rc.samba stop" to shut down a service.

They're right, in that you can't really know what this guy did there, but a total reinstall can be a pain. Try shutting down some services. What is the server for anyway?

You might also try "killall vncserver" or something like that.

Last edited by Tralce; 12-03-2006 at 09:18 AM.
 
Old 12-03-2006, 10:55 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
What else should i have done?
This being a business situation what you should have done (not "else") is alert someone that *is* responsable for the server. If you are not responsable then taking over control means you *become* responsable for the outcome, good or bad, and deleting an account is a good way to destroy information.

The first thing to do is make a log entry and note down information like time, machine name, commands performed and any information about the breach followed by alerting responsable personnel. If no responsable personnel can be contacted within a reasonable timeframe then how you act on it depends on your assessment of the situation. Does the server play a critical role in business? Does the situation cause a critical amount of damage? Does the situation allow the intruder to use the server as a springboard to other systems? If you have no idea you should first read some guidelines like the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html. The simplest way to mitigate the situation is logging as much information remotely as possible followed by a controlled shutdown of the server. While shutting down the server is bad for business it keeps the intruder from destroying "evidence" and keeps legal users from accessing the server if they could not be notified their account data was (possibly) compromised. At this point responsable personnel should take over. If no responsable personnel can be contacted within a reasonable timeframe you should then start by checking adjacent network devices for anomalies and preserve as much logging as possible.


I think a re-install depends on how critical the server is and what it is doing.
AFAIK repartitioning, reformatting and reinstalling from scratch depends on the severity of the compromise.


P.S. try useing write to ask you're intruder how he got in, maby he'll tell you.
Nice. Definately a good way to warn the user you're on to him.


They're right, in that you can't really know what this guy did there, but a total reinstall can be a pain.
Notice this is a business server and AFAIK businesses have no concept of subjective reasoning like "pain" is. What makes business respond is loss of operability, financial loss and loss of customer trust.
 
Old 12-03-2006, 03:48 PM   #8
landuchi
Member
 
Registered: Oct 2004
Distribution: Debian, Ubuntu
Posts: 74

Original Poster
Rep: Reputation: 15
Thanks for the replies.

The server works as a proxy and samba for a windows network. There is a lot of information in that server and i would say it is of critial importance, but there is no real concern on security in the company (until something worse happens i suppose).

The person responsable for the server is not capable of securing his own machine, less he is of taking care of the server. I am not by far an expert in security, but as they say here, in the land of the blind the one eyed is king :P.

The contents of the user's folder were copied to another location prior to deleting his account. Inside his folder i found instalation files for psybnc (an IRC client/server), and a cron.d file pointing to this script

---

#!/bin/sh
if test -r /home/miguel/.rut.go.ro/psybnc.pid; then
pid=$(cat /home/miguel/.rut.go.ro/psybnc.pid)
if $(kill -CHLD $pid >/dev/null 2>&1)
then
exit 0
fi
fi
cd /home/miguel/.rut.go.ro
./start &>/dev/null

----

There are a lot of services running on the system, but i am not familiar with linux services and i donīt know which ones are normal or needed. The server has been administered by several people with poor linux or security, knowledge or concern.

If you tell me what information to post maybe you can give an idea of which services to shutdown.


P.S. i am reading the documents at www.cert.org, thanks for the link.

Last edited by landuchi; 12-03-2006 at 03:49 PM.
 
Old 12-03-2006, 05:14 PM   #9
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,058

Rep: Reputation: 295Reputation: 295Reputation: 295
You should probably read the advice posted in the "unload suckit problem" thread (and actually take it seriously). Unfortunately, if the machine was root compromised there's really no way to recover in a completely safe way other than to reformat and reinstall. You can always boot off of a LiveCD to take critical data off the system before reformatting, but be careful you don't leave any nasty surprises left by the attacker.

I certainly hope that you notified that individual who is responsible for the server about the situation and what you've done. Having two people working at cross-purposes even if they are both trying to be helpful leads to mistakes being made. Keeping a log, as mentioned above, is an excellent idea.

If your company does not have the in-house skill to safely resolve the situation then you should consider hiring and outside consultant who does and retain them or pay for training for staff members so they can deal with these situations in the future. Whatever you do, do not brush this off. A proxy/file server is a trusted machine. Once compromised the attacker can use it to attempt to gain access to other machines on your local network. Furthermore, the attacker can use it to attempt attacks against other machines on the Internet. Depending on your exact situation, your firm may be held civilly or criminally liable for damages or denial of service caused by this (I Am Not A Lawyer, however, so don't construe any of this as legal advice -- just what I've read/experienced in my experience as a *nix admin).
 
Old 12-03-2006, 05:15 PM   #10
RolledOat
Member
 
Registered: Feb 2003
Location: San Antonio
Distribution: Suse 9.0 Professional
Posts: 843

Rep: Reputation: 30
A proxy server and SAMBA machine does not need FTP, VNC, etc. You need the samba port, the proxy ports and SSH open but (but ssh ONLY from the internal network). Everything else needs to be firewalled off from the outside world. Everything else is simply a security risk waiting to be accessed. Postgress open, IPP open, pptp open. Start with a firewall with ALL ports closed to the outside world, then allow SAMBA and the proxy ports only. Selectively open anything that is also required. Who from the outside world needs access to VNC?

I would not let this go with your management, at least you are asking the right questions. If I was doing it, I would start with a new server install, as mentioned, minimal install then add what you need fully firewalled, then start moving only the data and data only and when done/tested, replace the old one and blow it away.

RolledOat
 
Old 12-04-2006, 09:42 PM   #11
operator10001
Member
 
Registered: Mar 2006
Distribution: debian sarge
Posts: 222

Rep: Reputation: 30
look in the packlist for security tools. there are a LOT of them. track and trace your target. confirm and study the target. then plan your attack. I forgot where I got my firewall cracker from. If you attack right, there will be NOTHING left very soon. the only perfect defense is the perfect offense.
 
Old 12-05-2006, 06:49 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
@operator10001:
track and trace your target. confirm and study the target.
With any action you take you will most likely be sending your IP to the remote network alerting the legal owner of the box and his/her ISP somebody is attacking a system on their network and alerting the cracker somebody wants to "play". If you manage to trace back activities to a remote box but do not have access to it there is nothing left to "study" and your trace goes cold there. Crackers do not usually operate from home and their intermediates are disposable, so asserting anyway (without proper evidence) the remote machine is "the crackers home" will prove to be wrong. What you will be "attacking" is a remote machine, legally owned by somebody (just like yours), which got cracked (just like yours). Nice catch is that if you do get access to the remote machine you're committing a crime yourself because because the owner of the box has not given you permission to access it.


then plan your attack.
More importantly there is no talk of "attacking". What you'll be doing is retalliating. Not only does the above show it's timeconsuming (without guarantee of resolving) and exposing yourself too, attacking can also be an crime according to local Law. IMNSHO by retalliating you waste time and effort opting for a childish "eye for an eye" attitude towards things instead of investing time and effort in priority number one: mitigating, rebuilding and hardening the box.


In short, this is "advice" which no security-conscious person should give and I request you to not post that kind of "advice" in our Linux Security forum. With all due respect, and such.



@landuchi:
Since you show no intention to mitigate and correct the situation according to what I see as standard procedure (see btmiller's reply) I will be closing this thread RSN, leaving the necessary reading material:
- Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
- Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html
- LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261
If you do intend to correct the situation reply what steps you took already and ask the right questions.

Last edited by unSpawn; 12-05-2006 at 07:00 AM. Reason: //((nudge++))
 
Old 12-05-2006, 11:13 AM   #13
operator10001
Member
 
Registered: Mar 2006
Distribution: debian sarge
Posts: 222

Rep: Reputation: 30
sorry to be offensive, i am an anarchist and i am blackhat. you seem to believe that it cant be done but, it has been done before.
operator
 
Old 12-05-2006, 12:06 PM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
you seem to believe that it cant be done but, it has been done before.
LOL, I do not dispute it can be done. I just filled in some caveats (risks, MO, priorities) and pointed out why your "advice" is not "standard procedure" one should follow.


i am an anarchist and i am blackhat.
I try my best to be as impartial as I can so I do not care if you say you worship cAos or are deep into BO-fu: good for you! What I do care about is providing information that will help the OP and others that read the thread mitigate risks and recover from a situation like this, not find information with hidden risks or that will make him a target (again). Be aware LQ does not tolerate and does not want to be affiliated with anything that remotely looks like black hat ops in any way for obvious reasons. Next time please try to give advice that is helpful for the OP.
 
Old 12-05-2006, 12:18 PM   #15
chort
Senior Member
 
Registered: Jul 2003
Location: Silicon Valley, USA
Distribution: OpenBSD 4.6, OS X 10.6.2, CentOS 4 & 5
Posts: 3,660

Rep: Reputation: 69
Quote:
Originally Posted by operator10001
sorry to be offensive, i am an anarchist and i am blackhat. you seem to believe that it cant be done but, it has been done before.
operator
ROFL!

Anarchist: What people who find the idea of personal responsibility uncomfortable call themselves.
Black Hat: What skiddies call themselves.


BTW using a botnet to DDoS someone does not count as "hacking", and could you please explain to me what a "firewall cracker" does? Yeah, I thought so.
 
  


Closed Thread

Tags
security, services


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
An error Occurred during the file system check. Dropping you to shell; the system wil aneikei Linux - Newbie 3 02-11-2010 07:38 PM
How to find out if my system is i386, i486, or i586? Is there a system info command? mlsbraves Linux - General 6 02-17-2009 04:08 PM
Fedora system stuck at system message bus service upon bootup guy_ripper Linux - Software 4 02-16-2009 01:54 PM
KDE sound system doesn't play system notifications in SUSE with OSS driver mike1111 Linux - Software 0 07-17-2006 12:00 AM
LXer: Move an entire file system on a live Unix system LXer Syndicated Linux News 0 07-08-2006 05:33 PM


All times are GMT -5. The time now is 02:32 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration