I did a ps aux to kill some processes and I noticed some strange things started by Apache:

root 18805 0.0 0.0 3716 764 ? S Aug19 0:23 /usr/sbin/sshd
apache 9421 0.0 0.0 10896 364 ? S Aug19 0:00 ./elf
apache 10954 0.0 0.0 134080 420 ? S Aug19 0:00 ./elf
apache 16275 0.0 0.0 1468 268 ? S Aug19 0:00 ./bindz
apache 16302 0.0 0.0 1468 280 ? S Aug19 0:00 ./bindz
apache 16317 0.0 0.0 2188 796 ttyp0 S Aug19 0:00 sh -i
apache 16349 0.0 0.0 1468 284 ? S Aug19 0:00 ./bindz
apache 16367 0.0 0.0 2180 796 ttyp1 S Aug19 0:00 sh -i

It's already been running for 6 days, but I sure as heck didnt start those processes. And it seems that root had logged in just before?

Incase you are wondering, yes I do use apache for things.

Anyone have an explanation or have I been compromised? I googled it, and although some of the results mentioned possible backdoor exploit, the actual pages either werent in english or didnt have anything tangeable for me to make developments.

Insight appreciated.

-Chi

Indeed your box seems compromised. "elf" is the exploit (exploiting this AFAIK) and "bindz" is a backdoor (grep -ai backdoor /proc/16275/exe).

I would suggest closing down the box ASAP, notifying whoever has legitimate access to this box to check theirs and change their passwords used and initiate a full mop up: repartition, reformat, reinstall from scratch and harden the box. Any questions please ask.

Here are two docs you should read before continuing, the third one will come in handy in the hardening stage:
Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html
(LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261)

Before you do I would very much like to get my mits on the following if possible, please send me an email where I can get the /mnt/floppy/log:
lsof -n 2>&1 >> /mnt/floppy/log
ps axf 2>&1 >> /mnt/floppy/log
netstat -an 2>&1>> /mnt/floppy/log
lsmod 2>&1>> /mnt/floppy/log
find /tmp /var/tmp -ls 2>&1>> /mnt/floppy/log
find /proc/[0-9]\*/exe | xargs strings -an1 2>&1>> /mnt/floppy/log

If you would like to investigate stuff later on making a dd backup would be advantageous, if you don't at least save you /etc, /var and accessable temp dirs.

Ah crap -_-

/mnt/floppy/log does not exist, so all of those commands produce nothing. This is a webserver so I am going to have to backup alot of stuff. although i am not sure what dd stands for.

I have to goto work now (aaaaahhhh) but when I get back I will have to take a look at all those links and see if I can save myself from disaster.

/mnt/floppy/log does not exist
That was a placeholder thing. Save anywhere you like but preferably on removable media.


This is a webserver so I am going to have to backup alot of stuff.
Doublecheck you're not backing up binaries.


if I can save myself from disaster.
Save you apparently cannot, but keep from happening again, you can.

It's an interesting exploit: I'd look at how the exploit got there in the first place. I can only think of 3 ways on a 'standard' webserver: either a local malicious user, a local user with a really weak password, or you have a remote vulnerability in one of your webpages. Lots of php-based forums are exploitable, so I'd look there.

Definitely take a close look at your apache and ssh logs.

I've been working all day and I was moving as well, just got my computer back up. I am going to look into this further and read the links and do some backups..

..but those apache and ssh logs, where are those at? Also, what is "dd" you mentioned before.

Thanks again for your help.

I've been working all day and I was moving as well, just got my computer back up.
First of all you should not have resurrected the box for examination or making backups unless you're running a Live CD and mount the drives read-only. Resurrecting the box can cause all sorts of trouble like involuntarily executing (rogue) applications that can cause harm to you or other people on teh intarweb. I should have said that before. If you can't run a Live CD then you could take the physical disk to another box. Make sure you mount the drives read-only to not disturb data.


I am going to look into this further and read the links and do some backups..
Like I said before you should read the links before continuing because they contain information that will help you examine the box.


..but those apache and ssh logs, where are those at?
See your syslog.conf and daemon configs. For OpenSSH if logged through Syslog it'll be (/var/log,/var/adm)/secure or equivalent else it's own logs. For Apache it'll be in (/var/log,/var/adm)/httpd or equivalent.


Also, what is "dd" you mentioned before.
"dd" ("man dd") copies data and is commonly used to make "bit for bit" backups of drives or partitions. You will only need this if you want to do a forensics-like investigation later on.


I would like to emphasise again what you should do is:
- read the two docs first,
- boot the box for examination or making backups with a Live CD and mount the drives read-only,
- if you make backups do not back up binaries or binary data (except for formats that can be verified later on like database dumps), and keep your backups separate from regular backups,
* only make "dd" copies if you want to do forensics-like investigation later on. Before you decide that take note that forensics will need a separate box to work on, can have a steep learning curve, and use up considerable amounts of your diskspace and time while no results can be guaranteed (knowledge, tools, interpretation).

When you're done examining the "victim" let us know and we'll talk about reinstallation and hardening.

Okay, I will get back to you on this, I need a few hours or something. And when I said I got my computer back up, I meant my desktop, which isnt the server.

The server is actually still online I havent done anything because there is some stuff on there I want to get back (the server is co-located so my only access is remote) I've been downloading stuff all day. I will be notifying my host very shortly because I fear they are just going to wipe everything.

From what I copied so far to my windoze desktop via ssh and scp, my viruscanner picked up a few things, Linux FakeProcess and a php file in a hidden directory that was deemed malicious. It seems they had used an uploading service to create these files, although php files and directory creation is restricted. But I guess they managed to get around it.

I'll respond later today.

The server is actually still online I havent done anything
This must be one of my worst incident-handling threads ever, I keep forgetting adding stuff... This time it's mitigation. What I should have told you in the first place is to stop all non-critical services (basically you only need ssh to manage stuff and scp files) then make the firewall drop all traffic except for your IP (range). (Then the choice of killing those rogue processes is yours: they could contain malicious mechanisms (that for instance could be activated on (prolonged) network unreach) though that seems unlikely.)


I will be notifying my host very shortly because I fear they are just going to wipe everything.
A good hoster will have picked up anomalous traffic between your box and world already. You could ask them to make a backup, that could speed up getting rid of this situation and rebuilding the box, but I guess that's not a convenience but a money thing...


* If you can still ship me those process details that would be welcome.

Hi sorry. The server has been taken offline. My host is going to handle the backing up and restoration of the server.

I downloaded the logs.. but was very disappointed. No ssh logs were available, and the apache logs didnt go far back enough to find anything useful. The ftp logs did, but they didnt use ftp at all.

You said you were still waiting for those process details. Are you talking about the /mnt/floppy because that did not exist and you didnt mention anything about processes afterwards.

What you said about the iptables and only allowing my range is a very smart thing to do. I suppose next time this occurs I should be better prepared. Better logging and the like.

I have to run off to work now. Wish this post was a bit more indepth. This event had such bad timing because I have been moving, and my internet has been down, and I just havent had the time to react properly.

Thank you for all your help. Is there anything else I should know?

Is there anything else I should know?
Two words: host hardening. When your box is back up come back and see us and we'll talk about it. In the meanwhile you could check out post #1 of the LQ FAQ: Security references. Just don't try to grok it all at once.