Well, this server of mine has problems. I'm well aware of it. The server is in a Superb data center, so we don't have physical access to the box. That complicates things. Further, it's a very old box, and updates aren't available for it from Ensim. I don't really want to pick and choose services to update because Ensim is a picky bastard, and I can't risk breaking the setup. As such, we are going to be getting a new server to replace it.

I should mention first that RKHunter doesn't pop warnings off for root kits or invalid system components, but I know we have a problem.

A little over a month ago I made a post on here regarding what I thought was a port scan attack. As it turned out, I was hugely mistaken. Someone had exploited the server in some way to get an IRC bot net client on the server, which then downloaded movies to send off.

Now, I fixed the situation to "some" degree. I fixed permissions on various folders that were owned by apache with RWX permissions. However, I couldn't change them all. Some of our web scripts rely on them - such as our WYSIWYG editor. Unfortunately, they keep coming back on the folders that I didn't protect. I don't think they have any serious exploits on the server, but one can never be too sure. I just know that they never do anything severe. They just keep coming back every couple of weeks to somehow install a bot net client, serve for a few hours until I detect it and remove it, and then go away.

I did notice something today, though.
Running ps -aux | grep apache

apache 9124 0.0 0.1 6068 1072 ? S 10:45 0:00 sh -c cd /home/virtual/site8/fst/var/www/html/images/.thumbs/;cd /;find . -perm 777 -user apache
apache 9125 1.8 0.0 5620 780 ? R 10:45 0:04 find . -perm 777 -user apache

So, they were searching for exploitable files/directories with RWX permissions owned by apache. Fine, I can understand that, but where are they executing this from? I've checked to make sure that there aren't any .PHP command loaders laying around. I've found a few of them. They were basically a PHP file that generates an HTML form with a text box for a command to execute or a file box for a file to upload. It then prints out the result of your command upon the next page load.

So I guess I'm left wondering. Can I turn on some kind of global apache monitoring to watch which commands are invoked via the PHP system() command or PHP exec() command? Also, as a precaution, could I monitor all file inclusions? I know that log would be huge with the number of sites we have, but I wouldn't mind sifting logs to determine the cause.

Forgive the elongated response, but ya know, venting happens.

Now, I fixed the situation to "some" degree. I fixed permissions on various folders that were owned by apache with RWX permissions. However, I couldn't change them all. Some of our web scripts rely on them - such as our WYSIWYG editor.
So I get it you didn't ditch "vulnerable" home brewn (if any) or outdated stock software, didn't set up stricter firewall in and egress policies or installed mod_security as I offered in post #7? You see, fixing permissions isn't "fixing" things, it's like addressing a few symptoms, not the cause. Also it seems Ensim is being a blocking factor. If it stands in the way of doing necessary maintenance I'd say get rid of it and find an alternative that doesn't block what's necessary. How much time until the new box is configured, hardened and up and running?


I should mention first that RKHunter doesn't pop warnings off for root kits or invalid system components, but I know we have a problem.
Tried 1.3.0? If it still doesn't light up like a Xmas tree I'd appreciate some more details that could help us provide more coverage. An attachment with full process (axfwwwe) and open files (-n 2>&1) listings for instance.

I don't envy you. Trying to administer a machine exclusively over a network is rough. Can you get the data center's people to do things like put a CD in a CD-ROM drive?

There are so many things that you can do that it is possible to write a book on the subject. Unfortunately the books that I've read appear to be full of stuff that you don't need and deficient in stuff that you do need, like step by step guides.

One thing that you can do is check the necessity of permission settings on directories. Depending on the application you may not need to have read or write access granted to apache on directories where apache is opening a file and if it already knows the names of all of the files that it needs to access. You may be able to reduce the permissions on directories to execute only.

I couldn't tell from your example whether you have the apache user account chroot jailed. The directory structure does remind me of a chroot jail with the home directory in /var/www or something like that. If your apache daemon user account is not chroot jailed then make it so.

You can make container files and mount them with the noexec parameter on /tmp and on /var/tmp. This will reduce the ability of people to copy an executable into the publicly writable directories and then run them. I do the same thing on the /home partition. Note that this does not inhibit people from running a Java executable. I think that's because the Java software is on the root partition and the jar file is treated like a data file. I don't really know. It is possible that other programming languages might also be able to execute a script that is located on a partition that is mounted with the noexec parameter. I haven't done an extensive study of this technique's limits yet. I still implement this technique on my systems.

As far as the IRC bot goes what I would want to do is to put a machine in between your server and the public network. This machine would act as the firewall. This way even though your server is compromised it would require the hacker to break into the second firewall system to use network ports that you have closed via a firewall.

Speaking of this I hope that you at least are using a firewall program on the server. If you close the ports that you don't use then the hacker will have to have administrative privilege to open the IRC port and other ports. Hopefully the hacker only has normal user privileges.

Change the passwords on all of your user accounts on a regular basis and whenever you suspect that your system has been hacked. Make the passwords good ones. Change the passwords on all of the user accounts every day if you have to.

Remove all of the user accounts that you don't use. Be careful. You may have user accounts such as the nobody account that you don't realize are in use.

Make sure that you are not starting application daemons such as a mail transfer agent that you don't use. Keep what you need but get rid of things that you don't need.

Install Trip Wire or something like it. Yes the horse is out of the barn if you have malicious code on your computer but you could at least see what files change from this point onward.

Geez. There are a thousand things that you can do. From your post it seems that you are unwilling to break a running system's configuration. I agree. Try to expedite obtaining the new server. When you are setting it up you can implement lots of security enhancing things like the ones that I've listed.

On the new server you could also implement SELinux. It helps to reduce the scope of what any given user account is able to do.

Maybe some of my ideas will be immediately useful. Otherwise plan ahead as to how you can enhance security on the new system, and try to get it in house, and try to get it as soon as is possible. IMO remote sites are useful as backups. You should have your primary server some place that you can get your hands on it.

Well first off, thanks for the responses.

As for rkhunter 1.3.0 ... No, I hadn't tried it. However, it didn't pop off on anything *too* severe.


[08:25:04] Warning: The command '/usr/bin/GET' has been replaced by a script: /usr/bin/GET: a /usr/bin/perl -w script text
executable
[08:25:05] /usr/bin/groups [ Warning ]
[08:25:05] Warning: The command '/usr/bin/groups' has been replaced by a script: /usr/bin/groups: a /bin/sh script text ex
ecutable
[08:25:06] Warning: The command '/usr/bin/ldd' has been replaced by a script: /usr/bin/ldd: Bourne shell script text execu
table
[08:25:12] Warning: The command '/usr/bin/whatis' has been replaced by a script: /usr/bin/whatis: a /bin/sh script text ex
ecutable
[08:25:13] Warning: The command '/sbin/ifdown' has been replaced by a script: /sbin/ifdown: a /bin/bash script text execut
able
[08:25:13] /sbin/ifup [ Warning ]
[08:25:13] Warning: The command '/sbin/ifup' has been replaced by a script: /sbin/ifup: a /bin/bash script text executable
[08:25:18] Warning: The command '/usr/local/bin/rkhunter' has been replaced and is not a script: /usr/local/bin/rkhunter:
a /bin/sh script text executable


I checked all of the source code of the commands it warned about. They all appear to be fine.


[08:26:45] Checking for enabled xinetd services [ Warning ]
[08:26:45] Warning: Found enabled xinetd service: /etc/xinetd.d/wp_imap
[08:26:45] Warning: Found enabled xinetd service: /etc/xinetd.d/wp_imaps
[08:26:45] Warning: Found enabled xinetd service: /etc/xinetd.d/wp_ipop3
[08:26:45] Warning: Found enabled xinetd service: /etc/xinetd.d/wp_pop3s


The xinetd services that it mentions are used by Ensim's virtual hosting system.


[08:27:21] Checking for promiscuous interfaces [ Warning ]
[08:27:21] Warning: Possible promiscuous interfaces:
[08:27:21] 'ifconfig' command output: UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1


Eth0 is running in promiscuous mode, but I need to verify that Superb didn't do this on purpose. Until then, I don't want to change it.


[08:27:29] Warning: The SSH configuration option 'PermitRootLogin' has not been set.
The default value may be 'yes', to allow root access.


I'm aware the server has root login available. While I know it's not good practice, the password security on the account is very high, and it is changed regularly.


[08:38:47] Checking /dev for suspicious file types [ Warning ]
[08:38:47] Warning: Suspicious files found in /dev:
[08:38:47] /dev/MAKEDEV: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamical
ly linked (uses shared libs), stripped


Not too sure on this one, but from what I've read on Google, it seems ok.


[08:38:49] Checking for hidden files and directories [ Warning ]
[08:38:49] Warning: Hidden directory found: /etc/.java


This is fine.


[08:38:49] Warning: Hidden file found: /usr/share/man/man1/..1.gz: gzip compressed data, from Unix, max compression


This is fine. Just contains a man file.


[08:38:50] Warning: Hidden file found: /usr/bin/.tmppprofp: a php script text executable


Seems benign. I'm not quite sure why it was left there though. It doesn't do anything critical and seems like it was created when the server was initially setup in 2004.


As for the logs of running processes and open files, I'll post a URL with them after I get back from lunch along with a further response to the second post.

I uploaded the logs I mentioned to another server of ours.
http://www.crystaljayne.com/090307-1.tar.gz

As for the second responder, thanks for the lack of envy. It's much appreciated. :] I'll look into all of your suggestions. I've already taken care of a few of them, then there are a few I don't know anything about. I'll do some research.

In the end, I wish we could have our next server in house. However, we don't have the bandwidth in house to serve the demand. Beyond that, we don't have redundant power and network infrastructure to ensure the highest uptime.

I'm still looking into the ability to monitor PHP include()'s, exec()'s, and system()'s. I haven't found anything as of yet, though.