Hello everyone,

I am volunteering as an administrator for a small history club forum. We're being hosted on a virtual private server, so all administrative tasks fall on our shoulders. However, none of us are good at Linux, so I apologize for the stupid questions incoming and I hope they'll be easy to answer.

In short, we're being infected. To my knowledge, someone is repeatedly infecting /tmp/ and /var/tmp/ with Perl.Shellbot.B (this one comes in many names), disguised as unix.txt or similar system-looking files. This Shellbot then spawns numerous processes (usr/bin/httpd, usr/sbin/ateam...) under user apache, which bring the CPU to ridiculuos loads and the server has to be restarted. after I kill the processes and delete the files, tmp stays clean for some time, but at some point i noticed a "sh -c cd /tmp/ wget (a perl file in some distant domain)" command in the process list.

so, I read in your forum about securing /tmp/ by moving it to a new filesystem with noexec and other security attributes. i also setup a hourly cronjob to delete all files in /tmp/. also disabled apache and root shell access.

despite that, i got email from my host that my server was used for a DoS attack (once again, must be something in /tmp/). The support admin suggested that we should "inspect our server's connections to IRC servers", but the job is left to us, server-newbie historicians.
and here I come to you - is there any way to:
a) trace the mentioned IRC connections?
b) (if possible) disable those connections?

Linux newbies and history experts will greatly appreciate your help. promise to send you nice photos of World War 2!

------------------------------------------
System details: CentOS Linux 4.7, Linux 2.6.32, Webmin installed 1.441 installed. Hardware - 1GB RAM, plenty of HDD space and Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (1438 MHz).

No apologies necessary: we've all been new to Linux at one point.
OTOH if you don't learn from mistakes, that would be stupid :-]


Here you're treating symptoms and not the cause. The problem often is with software running in the web stack that has either problematic access permissions, home brewn software that doesn't properly escape user input, stale software versions or software that should not be accessible from the 'net at all.


Something like:
...but best start by:
- listing process information - checking your logs for any anomalies or clues (and attaching or posting them),
- checking user homes and any directories any software in the web stack can write to for rogue files.


That's a mortal mistake. Instead of Centos 4 you should be using Centos-5.8 (current AFAIK) and Webmin is at version 1.580. I wonder what more outdated software is accessible from the 'net...

2 members found this post helpful.

Welcome to L

Not the expert here, but there is rkhunter for that sort of time in your life. I'd try that for forensics. (root kit hunter). This command
lets nobody execute files from there. That may cause trouble, so chmod 0766 /tmp might be better.

If you are running things as root, stop. If you have an old system, your servers (anything ending in d, e.g. ircd) might be open to exploits. What's your distribution?

Do you use Perl scripts? If not, why not uninstall Perl?

Also, if you don't need stuff running, stop it. Particularly in /etc/inetd.conf (man inetd.conf)

1 members found this post helpful.

@business_kid: Instead of changing /tmp permissions, another approach is:
And I'd follow unSpawn's very excellent suggestions. (Although it would be nice if he'd use code tags. )

RKH is of limited use in that sort of case (if I described his problem properly). Running RKH does in no way constitute doing forensics.


Services are often started as root and then drop privileges to a lesser-privileged user. Wrt the rest you should read my first reply again.

chmod 0666 or 0766 /tmp will cause all sorts of breakage

well, we've been lazy in our server-side updates, but at least we don't install anything we're not sure about. Is the only way out of this - reinstalling CentOS?

Somehow my iptables don't like this and reply
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `6660:7000'


so I get up, open 'Running Processes' and there it is - "sh wget...", trying to download stuff, and it's child process "php spread.txt", both stuck. (/tmp/ is empty) i hope the process information shows anything:
(both are children of some httpd, otherwise indistuingishable from other httpds)

and a lot of established connections to various ports (see attachment). can you make out anything from that?

as to other logs - I've already checked them, but couldn't see anything suspicious. i'll go through them again later today.
the software is kept to a minimum, and I'd be glad to uninstall anything you suggest. i'm running e107 CMS (quite up-to-date), which does have several 777 directories, but i checked them immediately and didn't see anything.

business_kid and anomie: i left tmp attributes at 777. It's already mounted as a new filesystem with noexec, nosuid and similar limitations

as to perl - i'm always willing to uninstall stuff! e107 CMS needs only PHP, but will Webmin run without Perl?

once again, thanks for the help!

As far as currency of Centos is concerned:

(Ref http://wiki.centos.org/)
(I'm not quite clear if this 7 years from the date of release of the major version, ie, x.0, or whether it is from the release date of major.minor version, but in any case...)

According to distrowatch, release dates for Centos versions were:
So, technically, it looks like any of those versions, with the exception of 2.0 could be currently kept up-to-date. Practically, you wouldn't want to advise someone to go back to 3.9 as a sensible measure, but 4.9, 5.7 and 6.2 could all be possible.

What I think that we have got here is a version that could have been updated, but wasn't rather than something that couldn't have been updated. You may want to have a look at http://wiki.centos.org/FAQ/General?h...t=%28update%29 for the update information.


Personally, I'd always put some kind of rate limiting in logging, just 'coz I'm a bit paranoid about feeling smug about having done the right thing, and finding out later that I've actually created a new vulnerability, because a high load on logging can cause a problem, that wasn't originally present. Not that this is one of the urgent-est things to attend to, however.

1 members found this post helpful.

According to my copy of Iptables tutorial 1.2.2, Oskar Andreasson (...download from frozentux.net...everyone should have a copy...absolutely priceless for occasions like this...not light bedtime reading, but this is no time for light bedtime reading) P190, &ff (table 10-23):
Multiport must be followed by one of
--source-port
--destination-port
--port
where source-port and destination-port do what you'd probably expect* and --port could matches if either source-port or destination-port match. "It (the --port match) can take a maximum of 15 ports and can only be used in conjunction with -p tcp and -p udp. Note that the --port match will only match packets coming in from and going to the same port, for example, port 80 to port 80, port 110 to port 110, and so on."

* If, by 'what you'd expect' you also meant 'has a limit of 15 ports'. There are times when reading the friendly manual is not a sign of weakness.

1 members found this post helpful.

AFAIK The North-American Vendor supports a ten year cycle for major release from the date of the first release. http://wiki.centos.org/FAQ/General#h...91e1dde5b75e6d lists EOL for CentOS-4 as receiving updates until Feb 29, 2012*.


Agree.


Agree, but resource-wise rather disk than CPU IMHO.


Thanks, been posting a bit too quickly.

Looking at your text file of the log output, I see the following connection information:
This is a connection made from the PHP program, launched by a shell interface, from httpd and it is connecting to a typical IRC port range. Looking over your process list, I am not seeing anything particular out of the ordinary for a typical web and email server with a MySQL backend. It does look like material is being uploaded to your /tmp folder and being executed. Here I agree with unSpawn, that setting noexec on this location will be treating the symptom, not addressing the cause of getting the file to upload.

For a point of reference, 69.168.77.37 is a co-lo hosting provider in Utah called Fibernet, and is undoubtedly another "infected" machine of some sort.

What I don't like is the fact that they are running a shell. There are two paths that I would take at this point, one investigate and evaluate your web applications. As unSpawn pointed out previously, home brew PHP (or other web stack code, e.g. PERL) can be a common source of problems as well as outdated CMS and server applications. This would be the first thing to look at. The second thing is that based upon the information, it looks like httpd is an entry vector, that was possibly hijacked into giving a cmd shell, so lets start with that. What version are you running of Apache, PHP, and other web stack applications, e.g. webmin, PHP, MySQL, etc? What I am getting at here is, are you running a version with a known exploit that could account for this?

Best not uninstall or install anything right now. Any (well-meant) advice wrt cleaning up, post-incident hardening and such is rather premature.


Only in that it reinforced what I thought the source could be.


No, we'll get to that after we've tackled your priority #1 problem.


Recursively copy, 'scp -r' or rsync /var/log to another machine, install Logwatch there, move the logs to /var/log (or edit the logwatch.conf) and run 'logwatch --numeric --detail 5 --service all --range All --archives --print' for clues?


"quite up-to-date" doesn't cut it: we need a version as well because e107 CMS before version 1.0.0 (released 28/Dec/2011) has several vulns, see http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=e107 .


Lots of connections to 174.59.20.100 which is listed as a Malware Controller, a connection to 69.168.77.37 (Malware Controller as well), one to 84.32.250.241 (but only for the process calling "index.cgi" so that might be a false positive) and one to 94.125.182.253 (a "Site associated with email abuse or spam, Malware scan and infect source"). Your machine runs MySQL on any interface (should be confined to localhost), POP3 and IMAP (Dovecot does allow you to run SSL-ized services), FTP (plain text auth), SSH, Sendmail, HTTPS (unused?), Sysfence and like you said Webmin (unclear if access is restricted). On top of this all there seem to be no active firewall rules.


I suggest, in the following order, you:
- stop the web server,
- stop and disable any services you don't need during clean-up (MySQL, FTP, Dovecot and Sendmail),
- apply basic firewall rules,
- update what needs to be updated,
- verify all files on your file system,
- harden the machine
and only then expose it to the 'net again.

Look at process 6492 some more.

If you have mod_status install, it will be something like:
Also run:
HTH

1 members found this post helpful.

Thanks for the listing process commands, it is very helpful, I am putting these in my work notes.