Thanks unSpawn. I understand that. I have about 100 websites running on that machine and only a handful are WP but AVG found the trojan scripts attached to one of the WP files. As you can see from my thread with Thor, the focus has been on finding the source. I only asked for help with the SED issue while I was waiting for the audit to trigger.

Sure but so far I've only seen you delete files, restart the web server, reboot the machine, delete some more files and delete a web log. All apparently without permanent results. Maybe it's time for a more structured approach? You said "I have about 100 websites running on that machine and only a handful are WP". So what software is used besides WP? Homebrewn scripts? Anything outdated and potentially vulnerable perchance?

That's my point. I perform those tasks to get a clean slate so I can try to watch the injection as it is happening.

I need to find out why they keep reappearing. The structured approach is I installed the kernel audit program and found something strange. I set the audit to watch the index file in an unused openx installation I have. Then, I cleaned all my files and let time go by. After about 20 minutes, I ran ausearch and got this...

type=PATH msg=audit(02/25/2012 19:04:43.721:2118) : item=0 name=/home/httpd/sites/ads/doc-root/openx/index.php inode=68419586 dev=fd:02 mode=file,644 ouid=gfxweb ogid=gfxweb rdev=00:00
type=CWD msg=audit(02/25/2012 19:04:43.721:2118) : cwd=/home/httpd/sites/mlmblog/doc-root/wp-content/themes/classic
type=SYSCALL msg=audit(02/25/2012 19:04:43.721:2118) : arch=i386 syscall=open success=yes exit=62 a0=1f7a1ec a1=0 a2=1b6 a3=1f7a1ec items=1 ppid=9129 pid=9136 auid=dougw uid=gfxweb gid=gfxweb euid=gfxweb suid=gfxweb fsuid=gfxweb egid=gfxweb sgid=gfxweb fsgid=gfxweb tty=(none) ses=20 comm=httpd exe=/usr/sbin/httpd key=hacked

why would an httpd process coming form a blog site hit the openx index.php file? So I looked in that wp subdirectory and there are no scripts that I can find. Nothing immediately jumps out anyway. I found a stray index.html file with a bunch of javascript in it that I hadn't see before. That was promising but after cleaning it, the other index.php mods came back.

selinux
 

Dougwo,
From the log snippet, it looks like you have SELinux disabled. Maybe you don't have the luxury right now of putting it in enforcing mode, but you could set it to permissive mode, wait a while, then do a "ausearch -m avc -ts this-year" to see what kind of AVC messages turn up. You could also do a "ausearch -m avc -ts this-year | audit2allow" to make some quick inferences about what would NOT be allowed were you in enforcing mode. Later, when you get everything cleaned up, put SELinux in enforcing mode and create policy modules for your needs.

Thanks agentbiuzz. I was just thinking today of enabling SE linux. My first experiments a couple years back were a disaster (had to leave installs in standard locations, etc.) where I like to move things around for security by obscurity.

I think I solved the problem. I found it by running auditctl on a file I knew was going to be targeted. Then, when the log showed that the target file was modified by a wordpress theme directory, I investigated. But there was nothing in that directory that was suspicious. Until I looked in the server logs and found that the culprit was a file called functions.php. But functions.php didn't exist in that directory??? So I looked in my backups (infected ones) and found that there was a file called functions.php in a that same theme directory that had all sorts of javascript in it.

Apparently, my partner had a virus on his PC that exposed his WP password to the culprit. That culprit then logged in as admin to a wordpress site and uploaded a theme -- his own theme with compromised code. Then, he switched to his new theme for just a second that executed the code then switched back. We never knew it happened until after the fact. the code was clever enough to delete functions.php after it ran so we couldn't find it. I suppose it was luck that this morning's backup grabbed a copy.


Hope this little adventure helps someone else int he future! Thanks everyone for their help.

Maybe as a closing remark, there are some vulnerabilities with wordpress...maybe worth a look as well?

Glad it turned out okay. Safety of a system is defined by the safety of its parts, security is everyone's concern.

But, that's just me...

Thor

Fundamentally, you need to make sure that the PHP files are not writable by anyone.

I always set up a private maintenance account, with its own private group, and chown all files to it. No one else anywhere has permission to touch the files.

You must also check for Access Control Lists (ACLs), which, if present, will supersede the simple Unix permissions-mask.

If the site is on a shared host, then realistically any other system user might be doing it. Shared hosts customarily define a ftpusers group that everyone on their planet belongs to . . .

Seconding that, the extenstions are the oldest cloaking technique, the "other one" stumbles over this one all the time. If not sure use "file" to inspect something. Not sure about file? Try this:

to make a "fake" jpg file, that clearly is NOT a picture, now inspect:

and this should come back:

...eh, you already know this, what am I on about? :)

Thor

To follow-on to my suggestion, yes, putting stringent permissions in place does limit what you as an administrator can do to a site. But this is easily handled. Simply construct your site and manage it on an internal, inward-facing-only web site. Then, periodically, "publish it" to the public web site location by means of a script (external to WP). This script must be executed by the one maintenance user-id which does have read/write access; a user-id that is used for no other purpose. This script uses chmod to make the files read/writable to itself, then does an rsync to synchronize the directories, then it executes another chmod to make the files read-only, even to itself, once more.

This script should also perhaps use another rsync (or maybe gzip) to make a backup copy of the site before and after. Each backup that is taken is unique, and it is immediately made read-only.

It is a good idea to set the Unix permissions-mask to "no access" so that only an ACL entry permits access to anything .. and the access that is granted is strictly, "need to know, need to do." For instance, if Apache becomes nobody while running, then nobody can use the directories, and the maintenance userid of course can, but no one else anywhere can even see what the directories contain.

Database content is synchronized in a similar manner, and table permissions are likewise stringently controlled. If WordPress needs to consult a table on the production database, then it cannot modify it. And so on.

This is called the principle of least privilege. Computers suck at saying "yes," but they're excellent at saying "no."