LinuxQuestions.org - my server has been compromised, what next?

- Linux - Security (https://www.linuxquestions.org/questions/linux-security-4/)

- - my server has been compromised, what next? (https://www.linuxquestions.org/questions/linux-security-4/my-server-has-been-compromised-what-next-750090/)

my server has been compromised, what next?

Hi all,

I have an old P4 which serves as a home gateway/webserver/WAP.

Yesterday morning, I noticed a perl process using up excessive CPU time. After some investigation, I have determined that it is highly persistent, ie, it starts up after I kill it, even after rebooting.

This script runs as a webserver process (user www):

Code:

$ ps aux | grep www

www      116 100.0  0.7  5864  3588  ??  R    11:53AM  8:10.33 /usr/bin/web/httpd (perl5.8.9)

www      113  0.0  0.0    0    0  ??  Z    11:53AM  0:00.18 <defunct>

Interestingly enough, this file /usr/bin/web/httpd does not exist on my system.

From this:

Code:

$ sudo lsof | grep perl

[...]

perl5.8.9  4272    www    3u    IPv4 0xc33cf000        0t0    TCP

gw:51295->94.102.51.57:afs3-fileserver (ESTABLISHED)

it would appear to be talking to an IRC server at 94.102.51.57 on port 7000.

I am currently using the following bash command to keep it down while I decide what to do next:

Code:

$ while x=1 ; do sudo killall -9 perl5.8.9 && echo "killed..." ; sleep 15; done

Most likely I need to do a fresh reinstall. But obviously I need to find out where the vulnerability is so that it does not get reinstalled.

I am very much a parttime sysAdmin, and I have no experience with any kind of security forensics, hence I'd be most grateful for suggestions to point me in the right direction.

The server runs FreeBSD 7-stable i386, and has been kept fully up-to-date with security patches.

Thanks for any ideas.

You should pull the network cable.

In short what you should do before doing anything else is do nothing but read. The CERT Intruder Detection Checklist (http://web.archive.org/web/200801092...checklist.html) may be outdated but still provides a good checklist if you have none.

Before you start your investigation it is advisable to
0) save a complete detailed listing of users, network connections, processes, open files and
1) raise the firewall to only allow traffic from and to your (management) IP (range) to mitigate damage and
2) save (copies of) logs and then stop services that are not vital for your investigation including databases, web, mail and other servers: basically all you need is SSH.

Because the process runs as user "www" the first question you want to ask yourself is who has, or rather: what has, allowed this process to run. To come up with an answer you will need to
3) verify the integrity of all directories and files owned (writable to) by user "www" ('find / -user www -ls'), including temporary directories and files in users homes,
4) (visually) inspect files and cronjobs that can not be verified using your distributions package manager or filesystem integrity checker and then
5) inspect auth databases, system, firewall and application logs.

Please be verbose in your reporting (the more information the better) and please ask question before you do anything if unsure / unclear.

Quote: