Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I have an old P4 which serves as a home gateway/webserver/WAP.
Yesterday morning, I noticed a perl process using up excessive CPU time. After some investigation, I have determined that it is highly persistent, ie, it starts up after I kill it, even after rebooting.
This script runs as a webserver process (user www):
Code:
$ ps aux | grep www
www 116 100.0 0.7 5864 3588 ?? R 11:53AM 8:10.33 /usr/bin/web/httpd (perl5.8.9)
www 113 0.0 0.0 0 0 ?? Z 11:53AM 0:00.18 <defunct>
Interestingly enough, this file /usr/bin/web/httpd does not exist on my system.
it would appear to be talking to an IRC server at 94.102.51.57 on port 7000.
I am currently using the following bash command to keep it down while I decide what to do next:
Code:
$ while x=1 ; do sudo killall -9 perl5.8.9 && echo "killed..." ; sleep 15; done
Most likely I need to do a fresh reinstall. But obviously I need to find out where the vulnerability is so that it does not get reinstalled.
I am very much a parttime sysAdmin, and I have no experience with any kind of security forensics, hence I'd be most grateful for suggestions to point me in the right direction.
The server runs FreeBSD 7-stable i386, and has been kept fully up-to-date with security patches.
In short what you should do before doing anything else is do nothing but read. The CERT Intruder Detection Checklist (http://web.archive.org/web/200801092...checklist.html) may be outdated but still provides a good checklist if you have none.
Before you start your investigation it is advisable to
0) save a complete detailed listing of users, network connections, processes, open files and
1) raise the firewall to only allow traffic from and to your (management) IP (range) to mitigate damage and
2) save (copies of) logs and then stop services that are not vital for your investigation including databases, web, mail and other servers: basically all you need is SSH.
Because the process runs as user "www" the first question you want to ask yourself is who has, or rather: what has, allowed this process to run. To come up with an answer you will need to
3) verify the integrity of all directories and files owned (writable to) by user "www" ('find / -user www -ls'), including temporary directories and files in users homes,
4) (visually) inspect files and cronjobs that can not be verified using your distributions package manager or filesystem integrity checker and then
5) inspect auth databases, system, firewall and application logs.
Please be verbose in your reporting (the more information the better) and please ask question before you do anything if unsure / unclear.
Quote:
Originally Posted by Kropotkin
I have an old P4 which serves as a home gateway/webserver/WAP.
What products+versions exactly does it serve/run over HTTP? Who has access to the machine? Who has an account on the machine?
Quote:
Originally Posted by Kropotkin
After some investigation, I have determined that it is highly persistent, ie, it starts up after I kill it, even after rebooting.
Might be piggybacking onto another process or run as cronjob. Investigating all files owned by user "www" ('find / -user www -ls'), files in users homes, temporary directories is in order.
Quote:
Originally Posted by Kropotkin
This script runs as a webserver process (user www) Interestingly enough, this file /usr/bin/web/httpd does not exist on my system.
Might be a changed argv[0].
Quote:
Originally Posted by Kropotkin
sudo lsof | grep perl
Since you have the process' PID you can 'sudo lsof -p $PID' for details.
Quote:
Originally Posted by Kropotkin
it would appear to be talking to an IRC server at 94.102.51.57 on port 7000.
Raise firewall.
Quote:
Originally Posted by Kropotkin
Most likely I need to do a fresh reinstall. But obviously I need to find out where the vulnerability is so that it does not get reinstalled.
Since the process runs as user "www" lets see if we can verify integrity of the system before jumping to conclusions.
Quote:
Originally Posted by repo
You should pull the network cable.
Nice but that won't work for a remote machine. Please post a response in the way we may expect from senior LQ members with incident response knowledge or please abstain from posting a response. (If you have any questions about this you're invited to take it up with me by email.)
Last edited by unSpawn; 08-25-2009 at 08:38 AM.
Reason: //more *is* more
3) verify the integrity of all directories and files owned (writable to) by user "www" ('find / -user www -ls'), including temporary directories and files in users homes,
Before you start your investigation it is advisable to
0) save a complete detailed listing of users, network connections, processes, open files and
1) raise the firewall to only allow traffic from and to your (management) IP (range) to mitigate damage and
2) save (copies of) logs and then stop services that are not vital for your investigation including databases, web, mail and other servers: basically all you need is SSH.
One problem: this box serves as my mailserver and also router. (I am beginning to see there might be a problem with this.)
I use PF as firewall. Essentially I block all incoming traffic unless explicitly allowed. Everything allowed out.
Code:
pass out quick on $ext_if proto 41
pass out quick on gif0 inet6
pass in quick on gif0 inet6 proto icmp6
block in log
Obviously I need to quickly block everything out, except that which I need to be able to access the web. Not sure how to proceed here.
For the time being, I am killing the script every 15 seconds, as I indicated above.
Quote:
What products+versions exactly does it serve/run over HTTP? Who has access to the machine? Who has an account on the machine?
I am running lighttpd-1.4.20
the box is sitting next to my desk; only one user, me.
Quote:
Might be piggybacking onto another process or run as cronjob. Investigating all files owned by user "www" ('find / -user www -ls'), files in users homes, temporary directories is in order.
done, see previous
Quote:
Since you have the process' PID you can 'sudo lsof -p $PID' for details.
* CYP contact me by email? I'd like a (tarball) copy of those contents.
Quote:
Originally Posted by Kropotkin
One problem: this box serves as my mailserver and also router. (I am beginning to see there might be a problem with this.)
I use PF as firewall. Essentially I block all incoming traffic unless explicitly allowed. Everything allowed out.
Obviously I need to quickly block everything out, except that which I need to be able to access the web. Not sure how to proceed here.
you've got your priorities skewed. Right now the machine is being (partially) used by others to do their bidding, not yours. Whatever it is they do now or will do later might affect you but has the potential to affect others. So while I understand that email is "nice to have" it is not a valid argument to stop mitigating the current situation. Think "desperate diseases must have desperate cures".
Quote:
Originally Posted by Kropotkin
I am running lighttpd-1.4.20
No, I mean what is publicly accessable or provided over HTTP? Forum software? Webmail? Web statistics software? What?
Quote:
Originally Posted by Kropotkin
How do I disable the cronjob for user www? I can't login as www.
Move the spool file for user "www" out of the cron spool directory or remove it?
Quote:
Originally Posted by Kropotkin
Code:
[root@venus /var/cron/tabs]# ls -l
Code:
total 12
-rw------- 1 root wheel 3440 Aug 25 12:06 colin
-rw------- 1 root wheel 240 Jul 28 23:49 www
Wait. Not good. Your temp files are owned by user "www" group "wheel" but this file (note the timestamp) could only be written to by root.
I suggest you bring the box down now and investigate using a Live CD.
Last edited by unSpawn; 08-25-2009 at 09:45 AM.
Reason: //more *is* more
Wait. Not good. Your temp files are owned by user "www" group "wheel" but this file (note the timestamp) could only be written to by root.
I am sure you know waaay more about this stuff than I do, but are you sure of this?
The file "colin" was also written by root. It contains the cron jobs I have configured in my non-root account by that name. I don't use root to edit that file, just crontab -e. Is there not some way the script could have executed that as user "www" as well?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
