How do I remove a trojan core?

dougwo · 02-25-2012, 12:24 PM

I found that someone had compromised one of the wordpress sites on my server. It is the trojan that adds base64 encoded javascript to all of your index.php files. I found the suspect file, deleted it, cleaned all the index.php files, locked down FTP and then, after about 10 minutes, all of the code was added back to the index.php files. And it did it without modifying the timestamp.

I went through the entire process again and rebooted the computer to make sure nothing was running in memory. I ran avg on the entire site and it finds nothing. I run rkhunter and it finds nothing. But the index.php files keep self-modifying so I have some sort of virus running in a process somewhere. Any idea how to find it? I did set the permissions on critical index.php files to 440 so that they cannot self-modify and that seems to help. But I need to find the culprit process and stop it. ps -ax yeilds the following:

Code:

  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:01 /sbin/init
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:00 [migration/0]
    4 ?        S      0:00 [ksoftirqd/0]
    5 ?        S      0:00 [migration/0]
    6 ?        S      0:00 [watchdog/0]
    7 ?        S      0:00 [migration/1]
    8 ?        S      0:00 [migration/1]
    9 ?        S      0:00 [ksoftirqd/1]
   10 ?        S      0:00 [watchdog/1]
   11 ?        S      0:00 [events/0]
   12 ?        S      0:00 [events/1]
   13 ?        S      0:00 [cpuset]
   14 ?        S      0:00 [khelper]
   15 ?        S      0:00 [netns]
   16 ?        S      0:00 [async/mgr]
   17 ?        S      0:00 [pm]
   18 ?        S      0:00 [sync_supers]
   19 ?        S      0:00 [bdi-default]
   20 ?        S      0:00 [kintegrityd/0]
   21 ?        S      0:00 [kintegrityd/1]
   22 ?        S      0:00 [kblockd/0]
   23 ?        S      0:00 [kblockd/1]
   24 ?        S      0:00 [kacpid]
   25 ?        S      0:00 [kacpi_notify]
   26 ?        S      0:00 [kacpi_hotplug]
   27 ?        S      0:00 [ata/0]
   28 ?        S      0:00 [ata/1]
   29 ?        S      0:00 [ata_aux]
   30 ?        S      0:00 [ksuspend_usbd]
   31 ?        S      0:00 [khubd]
   32 ?        S      0:00 [kseriod]
   33 ?        S      0:00 [md/0]
   34 ?        S      0:00 [md/1]
   35 ?        S      0:00 [md_misc/0]
   36 ?        S      0:00 [md_misc/1]
   37 ?        S      0:00 [khungtaskd]
   38 ?        S      0:00 [kswapd0]
   39 ?        SN     0:00 [ksmd]
   40 ?        S      0:00 [aio/0]
   41 ?        S      0:00 [aio/1]
   42 ?        S      0:00 [crypto/0]
   43 ?        S      0:00 [crypto/1]
   48 ?        S      0:00 [kthrotld/0]
   49 ?        S      0:00 [kthrotld/1]
   52 ?        S      0:00 [kpsmoused]
   53 ?        S      0:00 [usbhid_resumer]
   84 ?        S      0:00 [kstriped]
  158 ?        S      0:00 [ttm_swap]
  165 ?        S<     0:00 [kslowd000]
  166 ?        S<     0:00 [kslowd001]
  302 ?        S      0:00 [scsi_eh_0]
  303 ?        S      0:00 [scsi_eh_1]
  306 ?        S      0:00 [scsi_eh_2]
  307 ?        S      0:00 [scsi_eh_3]
  401 ?        S      0:00 [kdmflush]
  403 ?        S      0:00 [kdmflush]
  422 ?        S      0:00 [jbd2/dm-0-8]
  423 ?        S      0:00 [ext4-dio-unwrit]
  424 ?        S      0:00 [ext4-dio-unwrit]
  458 ?        S      0:00 [flush-253:0]
  507 ?        S<s    0:00 /sbin/udevd -d
  631 ?        S      0:00 [edac-poller]
  831 ?        S      0:00 [kdmflush]
  868 ?        S      0:00 [jbd2/sda1-8]
  869 ?        S      0:00 [ext4-dio-unwrit]
  870 ?        S      0:00 [ext4-dio-unwrit]
  871 ?        S      0:00 [jbd2/dm-2-8]
  872 ?        S      0:00 [ext4-dio-unwrit]
  873 ?        S      0:00 [ext4-dio-unwrit]
  996 ?        S      0:00 [kauditd]
 1005 ?        S      0:00 [flush-253:2]
 1290 ?        S<sl   0:00 auditd
 1315 ?        Sl     0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
 1344 ?        Ss     0:00 irqbalance
 1358 ?        Ss     0:00 rpcbind
 1376 ?        Ss     0:00 rpc.statd
 1388 ?        Ss     0:00 mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
 1410 ?        S      0:00 [rpciod/0]
 1411 ?        S      0:00 [rpciod/1]
 1415 ?        Ss     0:00 rpc.idmapd
 1446 ?        Ss     0:00 dbus-daemon --system
 1457 ?        Ss     0:00 cupsd -C /etc/cups/cupsd.conf
 1482 ?        Ss     0:00 /usr/sbin/acpid
 1491 ?        Ss     0:00 hald
 1492 ?        S      0:00 hald-runner
 1520 ?        S      0:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event1
 1539 ?        S      0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
 1552 ?        Ssl    0:00 automount --pid-file /var/run/autofs.pid
 1571 ?        Ss     0:00 /usr/sbin/sshd
 1579 ?        Ss     0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 1604 ?        Sl     0:00 /opt/avg/av/bin//avgd 
 1612 ?        Sl     0:01 /opt/avg/av/bin/avgavid
 1651 ?        S      0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysq
 1779 ?        Sl     0:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/pnweb-01.com.
 1791 ?        Sl     0:00 /opt/avg/av/bin/avgtcpd
 1800 ?        Sl     0:00 /opt/avg/av/bin/avgscand -c 3
 1821 ?        Sl     0:00 /opt/avg/av/bin/avgsched
 1844 ?        Ss     0:00 /usr/sbin/dovecot
 1846 ?        S      0:00 dovecot/anvil
 1847 ?        S      0:00 dovecot/log
 1856 ?        Ss     0:00 proftpd: (accepting connections)
 1869 ?        Ss     0:00 sendmail: accepting connections
 1877 ?        Ss     0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
 1900 ?        Ss     0:00 /usr/sbin/abrtd
 1908 ?        Ss     0:00 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
 1918 ?        S      0:00 /usr/bin/python /usr/bin/denyhosts.py --daemon --config=/etc/denyhosts.conf
 1925 ?        Ss     0:00 /usr/sbin/httpd
 1933 ?        Ss     0:00 crond
 1944 ?        Ss     0:00 /usr/sbin/atd
 1957 tty1     Ss+    0:00 /sbin/mingetty /dev/tty1
 1959 tty2     Ss+    0:00 /sbin/mingetty /dev/tty2
 1961 tty3     Ss+    0:00 /sbin/mingetty /dev/tty3
 1963 tty4     Ss+    0:00 /sbin/mingetty /dev/tty4
 1965 tty5     Ss+    0:00 /sbin/mingetty /dev/tty5
 1967 tty6     Ss+    0:00 /sbin/mingetty /dev/tty6
 1973 ?        S<     0:00 /sbin/udevd -d
 1974 ?        S<     0:00 /sbin/udevd -d
 1975 ?        S      0:01 /usr/sbin/httpd
 1976 ?        S      0:00 /usr/sbin/httpd
 1977 ?        S      0:01 /usr/sbin/httpd
 1978 ?        S      0:00 /usr/sbin/httpd
 1979 ?        S      0:00 /usr/sbin/httpd
 1980 ?        S      0:01 /usr/sbin/httpd
 1981 ?        S      0:01 /usr/sbin/httpd
 1982 ?        S      0:00 /usr/sbin/httpd
 1983 ?        S      0:01 /usr/sbin/httpd
 2173 ?        S      0:00 /usr/sbin/httpd
 2182 ?        S      0:00 /usr/sbin/httpd
 2183 ?        S      0:00 /usr/sbin/httpd
 2184 ?        S      0:00 /usr/sbin/httpd
 2391 ?        S      0:00 dovecot/config
 2392 ?        S      0:00 dovecot/auth
 2394 ?        S      0:00 dovecot/auth -w
 2411 pts/0    Ss     0:00 -bash
 2432 pts/0    S      0:00 su
 2434 ?        S      0:00 /usr/libexec/fprintd
 2435 ?        S      0:00 sendmail: startup with 197.122.66.229
 2436 pts/0    S      0:00 bash

Anything look suspicious? I am a little concerned about the next to last line. Why would 197.122.66.229 be starting sendmail?

Help please!!

ButterflyMelissa · 02-25-2012, 02:13 PM

Hi, just a loose thought, run iotop to see what accesses the drive and how much. As soon as the clean-up is done, "it" should get back to work. At that point, it should consume a lot of disk activity...
Next, look at the /etc/passwd file to see what/who has root privz besides the root. That could be a clue to a possilbe "shadowroot"...
If you can spot where what comes from, start the system with a clean Live CD and snip where possible, Knoppix is a good one, has great tools...
And, by the way, if the IP address sendmail uses is foreign...it stands to reason that's the hacker's contact line...cutting that may alert him/her, so be careful there...
Oh, did you try a netstat of the system itself? Maybe something could come from that angle too...
Shape up the fire wall, after the clean-out, it's useless to do that now, the trojan will simply rewire it anyway.
By the way, the trojan may need root rights, deleting the entry from the passowrd file(s) can help...

Good luck

Thor

dougwo · 02-25-2012, 04:14 PM

From what I've been reading all day, the culprit is a compromised php file somewhere. That file is a page that is triggered remotely by accessing it's URL. It then goes and re-writes the trojan into all of the index.php files. I cleaned everything, cleaned by http access files and then started httpd waiting for it to occur. After about 30 minutes it did so I am now analyzing log fines to see if I can determine which page has the compromised code on it. What a hassle. I don't know what code to look for in the 'parent file.

ButterflyMelissa · 02-25-2012, 04:21 PM

Okay, let's be rational...it is possible the hacker is not even calling up the file anymore...this is automated now (real clever, granted). However, it's got to be something that is allowed to write to the local disk. I suspect that php file to call an external file: the trojan.
Okay, the trojan could be anything, but perl comes to mind. Possibly part of a rootkit (sorry, dude), so, I'd go for a grep-seek for anything a php file calls to the "outside", php has an include syntax, it goes something like this

Quote:

<? include a.file.pl ?>

maybe a grep seek in that direction?

Keeping my ears open...

Thor

dougwo · 02-25-2012, 04:27 PM

Well, I did find some perl files with AVG that had been uploaded to the wordpress directory structure. But I deleted than and completely removed that particular blog and the trojans still came back. Is there a way to log disk writes to a file so I can inspect that?

ButterflyMelissa · 02-25-2012, 04:50 PM

Quote:

Is there a way to log disk writes to a file so I can inspect that?

Dunnow of the top of my head....but, as you clean ship...

wait, maybe we can lure 'im out. Make a folder in the wordpress setup, only give yourself write access, make it world readable. Put a php file in there, fire op iotop and watch. There should be a process knocking at the door of that new folder...

Maybe a hint...

dougwo · 02-25-2012, 04:54 PM

op iotop ?

ButterflyMelissa · 02-25-2012, 04:55 PM

typo, I meant to say "start up iotop"

Oops...

dougwo · 02-25-2012, 05:07 PM

FYI, I found this: http://www.cyberciti.biz/tips/linux-...to-a-file.html

ButterflyMelissa · 02-25-2012, 05:11 PM

Cool, just...what file will you be auditing? There's quite a few out there, I guess...
Unless you want to watch the "baitfile"...then, that's clever...do it!

Let's see where the eagle splats down...

Thor

dougwo · 02-25-2012, 05:13 PM

I am auditing one of the index.php files that I know will have code injected into it in a few minutes... (waits with baited breath)

ButterflyMelissa · 02-25-2012, 05:14 PM

...keeping my fingers crossed here...

dougwo · 02-25-2012, 05:27 PM

meanwhile, back at the ranch, I want to run a script to clean all of my index.php files. I wrote this:

#!/bin/bash
sed -i "s/eval(base64_decode('a-long-string-of-characters-goes-here'));//g" index.php

but it doesn't work. This isn't a very difficult command. What am I doing wrong? I basically just want to strip out that long eval. Do I need to escape some character? Also, how can I get it to recurse though all sub directories in my 'sites' folder?

Thanks if you have an answer. I have looked at various googled-results and all seem confirm my syntax.

ButterflyMelissa · 02-25-2012, 05:41 PM

Okay, maybe this threeliner could help

Quote:

sed s/one/two/ <fromthisfile.txt >tothatfile.txt
mv tohatfile.txt fromthisfile.txt
rm tothatfile.txt

Pour that into a script, and get "one" and "two" substituted by the command line params...

Keeping mileage in account here, it's one-o-clock (in the morning) here so...

Edit...

Quote:

sed s/$1/$2/ <$3 >tothatfile.txt
mv tohatfile.txt $3
rm tothatfile.txt

Try it, letr's assume this is called replc.sh, you could call it up with:

Quote:

./replc.sh why not a_file.php

(mileage, remember)

(turning in for the nite *yawn* )

unSpawn · 02-25-2012, 05:53 PM

- Stop the web server during maintenance,
- Check if your WP version is the latest version. BTW there are no compelling reasons not to run the latest version,
- Check plugin usage. Update plugins if updates are available or disable them if they are no longer updated or maintained (also see this WP/TimThumb thread.),
- Check for any files in the upload directory (for instance files having a .jpg extension doesn't mean they are JPEG files),
- Harden the machine. If you host WP yourself then the OWASP mod_security rules (CVS rules tarball) may help you beef up security.
- If there's anything else to address then would be the time, before you expose the machine to the 'net again.

* Please check prioritisation of tasks at hand. Cleaning up base64-encoded crud may seem the most important to you but it is not. If you do not combat the cause first then you'll be experiencing the symptoms again and again.