IRCD while doing netstat... after being hacked. Need help. = *(

ritec · 11-20-2008, 11:20 AM

A couple of days ago I discovered a hacker had installed pscan2 on my web server. I found the files and deleted then ran chkrootkit and everything seemed to be fine.

I've been paying pretty close attention to the webserver lately and today to my surprise while running netstat I found the following connections:

Code:

tcp        0      0 internal:http           cpe-74-66-2-136.n:52224 ESTABLISHED
tcp        0      0 internal:http           cpe-74-66-2-136.n:52227 ESTABLISHED
tcp        0      0 internal:50462          mesa.az.us.underne:ircd TIME_WAIT
tcp        0      0 internal:http           cpe-74-66-2-136.n:52130 TIME_WAIT
tcp        0      0 internal:44830          quakenet2.underwor:ircd ESTABLISHED
tcp        0      0 internal:http           crawl-66-249-71-1:48998 ESTABLISHED
tcp        0      0 internal:44846          83.140.172.212:ircd     ESTABLISHED
tcp        0      0 internal:http           cpe-74-66-2-136.n:52212 TIME_WAIT
tcp        0      0 internal:http           cpe-74-66-2-136.n:52215 TIME_WAIT
tcp        0      0 web.xxxxxxxxx:ssh 172.16.38.141:1954      ESTABLISHED
tcp        0    132 web.xxxxxxxx:ssh 172.16.38.141:21934     ESTABLISHED
tcp        0      0 internal:http           cpe-74-66-2-136.n:52166 TIME_WAIT
tcp        0   2747 internal:http           us-nj-smtpgw2-1.p:16867 ESTABLISHED
tcp        0      0 internal:38795          efnet.as6453.net:6665   ESTABLISHED
tcp        0      0 internal:50472          195.144.12.5:6669       ESTABLISHED
tcp        0      0 internal:34479          efnet.as6453.net:6665   ESTABLISHED
tcp        0      1 web.xxxxxxxxx:50465 mesa.az.us.underne:ircd SYN_SENT
tcp        0      1 web.xxxxxxx:50466 zagreb.hr.eu.under:9999 SYN_SENT
tcp        0      0 internal:44938          efnet.as6453.net:ircd   ESTABLISHED
tcp        0      1 web.xxxxxxx:50467 irc2.saunalahti.fi:6669 SYN_SENT

As you can see there are a couple of ircd connections going on there. To my knowledge my server has been acting pretty normal lately and I do not see any processes running that should not be running.

But now if I run netstat --inet -ape I get something very scary.

Code:

tcp        0      0 *:32768                 *:*                     LISTEN      rpcuser    1288       962/rpc.statd
tcp        0      0 localhost:32769         *:*                     LISTEN      root       1514       1165/xinetd
tcp        0      0 *:sunrpc                *:*                     LISTEN      root       1229       934/portmap
tcp        0      0 *:http                  *:*                     LISTEN      root       80336      4609/httpd
tcp        0      0 *:x11                   *:*                     LISTEN      root       1858       1431/X
tcp        0      0 *:10000                 *:*                     LISTEN      root       1835       1412/perl
tcp        0      0 *:ssh                   *:*                     LISTEN      root       37239321   5085/sshd
tcp        0      0 localhost:smtp          *:*                     LISTEN      root       1599       1206/sendmail: acce
tcp        0      0 *:https                 *:*                     LISTEN      root       80335      4609/httpd
tcp        0      0 internal:http           crawl-66-249-71-1:46549 ESTABLISHED apache     150408880  15278/httpd
tcp        0      0 internal:51102          mesa.az.us.underne:ircd ESTABLISHED root       150408852  2574/bash
tcp        0      0 internal:44830          quakenet2.underwor:ircd ESTABLISHED 502        149520954  21663/bash
tcp        0      0 internal:44846          83.140.172.212:ircd     ESTABLISHED 502        149521070  21663/bash
tcp        0      0 internal:51112          irc2.saunalahti.fi:6669 ESTABLISHED 502        150408913  21617/bash
tcp        0      0 web.xxxxxx:ssh 172.16.38.141:1954      ESTABLISHED root       149767615  14494/sshd
tcp        0    164 web.xxxxxxxx:ssh 172.16.38.141:21934     ESTABLISHED root       150328903  25887/sshd
tcp        0      0 internal:38795          efnet.as6453.net:6665   ESTABLISHED 502        142612261  21617/bash
tcp        0      0 internal:http           192.195.66.49:42464     TIME_WAIT   root       0          -
tcp        0      0 internal:34479          efnet.as6453.net:6665   ESTABLISHED 502        135393815  21617/bash
tcp        0      1 web.xxxxxxx:51109 zagreb.hr.eu.under:9999 SYN_SENT    root       150408900  2574/bash
tcp        0      0 internal:44986          efnet.as6453.net:ircd   ESTABLISHED root       149522094  2574/bash
tcp        0      0 internal:44938          efnet.as6453.net:ircd   ESTABLISHED root       149521692  2574/bash
tcp        0      0 internal:http           65-85-127-130.cli:26326 TIME_WAIT   root       0          -
tcp        0      1 web.xxxxxxxx:51110 irc2.saunalahti.fi:6669 SYN_SENT    root       150408901  2574/bash
tcp        0      0 internal:http           65-85-127-130.cli:26334 TIME_WAIT   root       0          -
tcp        0      0 internal:51111          10.0.0.5:32600          FIN_WAIT2   root       0          -
tcp        0      0 internal:51107          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51105          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51104          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51114          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51113          10.0.0.5:32600          ESTABLISHED apache     150408921  15135/httpd
tcp        0      0 internal:51094          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51092          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51103          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51097          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:51096          10.0.0.5:32600          TIME_WAIT   root       0          -
tcp        0      0 internal:http           65-85-127-130.cli:26353 ESTABLISHED apache     150408874  15135/httpd
tcp        0      0 internal:44953          83.140.172.211:ircd     ESTABLISHED 502        149521850  21663/bash
tcp        0      1 web.xxxxxxx:51108 Tampa.FL.US.Undern:ircd SYN_SENT    root       150408894  2574/bash
udp        0      0 *:32768                 *:*                                 rpcuser    1285       962/rpc.statd
udp        0      0 *:10000                 *:*                                 root       1836       1412/perl
udp        0      0 *:33425                 *:*                                 root       4391043    2574/bash
udp        0      0 *:32841                 *:*                                 502        36146258   21617/bash
udp        0      0 *:32844                 *:*                                 502        36152029   21663/bash
udp        0      0 *:60882                 *:*                                 502        35914725   17841/inetd

Any recommendations on what I should do to get rid of this annoying problem?

trickykid · 11-20-2008, 11:35 AM

First unplug your system, it's probably clearly been compromised. Then go scan the Security forum for tips on what to do next as these types of questions are asked all the time. But you're machine is most likely untrustworthy. The longer it's still on the net, the more damage it can do.

pixellany · 11-20-2008, 12:17 PM

Moved: This thread is more suitable in Security and has been moved accordingly to help your thread/question get the exposure it deserves.

ritec · 11-20-2008, 12:55 PM

Thank you for moving the thread and sorry for posting it in the wrong forum.

Anyway so I have looked at the processes behind these tcp/udp connections and they seem to have started a week ago when I had the "cracker" intrusion. That means the cracker, to the best of knowledge, has not started new processes since the intrusion.

Apparently I did get rid of the problem but there where still some extraneous processes he seems to have started. I have been checking my logs by the minute and I did see an attempt from him to come back November the 17th but he was unsuccessful as I had changed his password and deleted the account he used to get in which was some sort of test account I had on my system.

That said, I have also traced the cracker and found a couple of his servers including a web server where he stores a bunch of tools and even some incriminating log files. I am about to report this to the FBI as I saw he is a scammer and had a paypal phishing file amongst a lot more incriminating information.

Any advise? It's my first time dealing with something like this. By the way I have an exact snapshot of my server from right after he made his attack.

OlRoy · 11-20-2008, 01:02 PM

Is the web server for personal use, or used by your employer?
Is there personal information on it?
Do you need to collect evidence in a forensically sound manner for court, or do you just want to get it up and running again?
What services were exposed to the internet?
Were you keeping it patched or not using a strong password?
What have you done so far since detecting it?

ritec · 11-20-2008, 01:34 PM

-It is owned by my employer.
-Luckily no personal information is stored in the server.
-The server is actually up and running and seems to be fine now. I have no interest in going to court about this but I do want to report his hideout to the feds as he seems to have a criminal record in the USA, or so it says on one of the chat logs I found in the server he sues to store his tools.
-What services where exposed? well pretty much just http and ssh.
-He was able to log in guessing a password for a test account... i think it might have been the work of a worm.
-Since detecting it I have cleaned up my servers, got rid of the account he created and the account with the weak password and got rid of an invisible directory he created with a couple tools to scan the web. I found out about the compromised because I was getting Huge traffic logs, and it turned out he had been scanning the internet for more computers with weak ssh passwords.

Thanks for the help.

unixfool · 11-20-2008, 01:40 PM

Wow...he's joining IRC servers as root. Because of that, I would expect even more badness to happen. I'd keep a sharp eye on this machine (or just disconnect the machine from the network or internet).

ritec · 11-20-2008, 01:52 PM

Yeah I think I will quarantine that server and put my backup webserver in place.