C code question/Server security inquiry (debian/LAMP)

curtisb · 03-23-2011, 09:15 PM

I found this in my server.

Can someone tell me what it does?
Am I correct in thinking it takes an argument and then changed the gid/uid on the file?

Code:

#include <stdio.h>
#include <stdlib.h>
main(int argc, char *argv[])
{
setgid(0);
setuid(0);
system(argv[1]);
return 0;
}

How could someone have put this on my server?

It was owned by apache's www-data user. and a module was loaded in apache and was including malware in my site as people loaded the pages. I am trying now to figure out how it happened.

I found the file by using rkhunter but no rootkits were found. I have checked all the pages on my site for phpshells and cannot find any. so I am not sure how these punks got into my server.

Server details:

Code:

# uname -a
Linux hostname 2.6.26-2-amd64 #1 SMP Thu Nov 25 04:30:55 UTC 2010 x86_64 GNU/Linux

# php -v
PHP 5.2.6-1+lenny9 with Suhosin-Patch 0.9.6.2 (cli) (built: Aug  4 2010 06:06:53) 
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies

# mysql -V
mysql  Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (x86_64) using readline 5.2

# apache2ctl -V
Server version: Apache/2.2.9 (Debian)
Server built:   Dec 11 2010 21:34:00
Server's Module Magic Number: 20051115:15
Server loaded:  APR 1.2.12, APR-Util 1.2.12
Compiled using: APR 1.2.12, APR-Util 1.2.12
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT=""
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"

Any help appreciated.

Thank you,
-Curtis

rtmistler · 03-24-2011, 11:57 AM

Looks like it's code meant to make itself root and then execute the argument that was passed into the program. Dangerous, but I'm not sure it will work if the login name you you execute it from is not allowed to perform those actions.

Noway2 · 03-24-2011, 02:11 PM

First, let me say welcome to LQ, Curtisb.

One question I have is what are the permissions on the web server directories? Normally one would not expect www-data to be able to write to any of these. Was the file located inside the www-data / apache document root or was it outside of it? Are you sure that you did not create it at some point as part of experiment to see if this were possible? I ask because I recall reading an example somewhere recently that discussed writing a C file to set the UID to 0 and see if you get root privilege from it.

Yes, the code tries to set the ID to 0 (root) and then execute whatever was passed to it. It is also C code, which suggests that it would need to be compiled. You may have a binary version executing somewhere. I would think that one of the first things to do upon activating a binary copy of it would be to erase this potentially 'tell tail' sign.

There are a couple of other things I would check, such as the output of netstat -pane to see if you have any open connections that are not explainable. You could also review the CERT check list (here is a link). It provides a solid overview of things you can look for. In this case, it mentions looking for files with uid and guid set that are owned by root (i.e. files that will execute with root permission).

I will also mention that LQ-Security handles investigations seriously with a fact based approach. We tend to ask a lot of questions as ask for the output for commands. Normally, we recommend that you unplug a computer from the network or put a firewall up to isolate it except for a trusted SSH connection.

One possible problem that you face, especially with a root level or possibly root level compromise (and the program looks like an attempt to execute root programs) is that that you are using customized versions of you investigative tools that won't report the suspect connection. Consequently, you can verify your critical application binaries, like netsat, against your package repositories first.

curtisb · 03-24-2011, 07:39 PM

Thank you Noway2,

I ran the netstat -pane as you suggested and can see nothing out of the ordinary.

The file with the code was actually in /dev/shm/ which scares me. Since how it got there is beyond me.

I just went through all the md5sums from the original files and compared them with the repositories and they all seem fine. I also checked apache2, apachectl, mysql and PHP. I then went through all the modules and compared them to a freshly installed server using the same packages and all the md5sums for those matched also.

I see no evidence of shells or even any extra users in the passwd or shadow files.

After further investigation and some alerts from my users it looks as though I have fallen subject to a javascript injection attack. Apache is injecting javascript into existing javascript files 'on the fly' I have seen the infected file by using google developer tools and the downloaded version is almost the same except at the top of the javascript file there is a rogue bit of code. I have moved that file to a different server as a test... and now another .js file has come under attack on THIS server (the other file is no longer infected). Both are clean on the server but is randomly tainted on download.

How would apache be able to do this? Is there something else I can check? Its almost like its proxying certain files or something, but randomly so you never know when its going to inject it again...

Does anyone have any Idea? Has anyone ever heard of this type of attack and have any Idea how it is executed/exploited?

This weekend I will be moving my website to a different server, but I will continue to work on this server til I figure out how this happened and how I can prevent this in the future.

Thank you,
Curtis

Noway2 · 03-25-2011, 05:03 AM

Quote:

This weekend I will be moving my website to a different server, but I will continue to work on this server til I figure out how this happened and how I can prevent this in the future.

This is one of the smartest things that you can do. One thing I would also suggest is that you make an image of the drive and save it as a backup. Based upon some of your other comments, I do think you have a root level compromise. If at all possible I would even suggest that you make the image BEFORE you reboot the machine (thinking of the file in /dev). It is very important to determine how these happen so that it can be prevented in the future. LQ has some very knowledgeable people in this regard and we can, and will, help you with the process. Now, once you have secured the machine, lets begin.

First, lets confirm the permissions and ownership of the compromised files. What are the ownership and permissions of your Apache document root, and your /dev file. I would expect that /dev would be drwxr-xr-x root root and the files in the directory would be owned by root. Also, what are the ownership and permissions of the javascript files being modified and of the Apache directories? What I am driving at here is: was root privilege required to compromise the files or were they compromised and this used as the platform to gain root privilege?

Second, You already posted your distrobution level, etc in your initial post. Are using any content manager, like Joomla? What about CPanel?

Third, are you running any other server processes, e.g. SSH, FTP, DNS, mail, etc. If you are running SSH, what steps have you taken to secure it?

Fourth, I notice that your PHP, MySQL, and Apache are a little bit dated, but not terribly so. Debian tends to run a slower update process, which in many ways is one of its strengths. However, we should review the security and change logs to see what the differences and fixes between your version and the current version is. Specifically, have any known exploits that could explain this been addressed.

Fifth, do you have other users of the system or are you the only one?

Sixth, can you post, or attach the output of netstat -antupe, lsof - Pwln, /bin/ps axfwwwe, and who.

Seventh, were you running any sort of intrusion detection applications like Snort, Samhain, Tripwire, Logwatch, Ossec, etc?

You are also going to need / want to scrub your log files very carefully. There are some tools that can help do this, but we will need to find one that will do an after the fact analysis rather than a run time. I don't have a specific suggestion off hand, so I will wait on this one. Do you have a sense of when the trouble began? Do you have backup images that you can compare to that can help identify changes?

I am sure that there will be more questions, but the above will get things started.

curtisb · 03-26-2011, 05:25 PM

I ran through some of the things you suggested and nothing out of the ordinary showed up. I even installed unhide and then scp'd it and a few other binaries (netstat, ps, lsof etc..) from a different server and ran them from my home directory hoping that new binaries in a different directory would go unnoticed. Nothing showed up, but it continued to happen. Someone has definitely gotten me on this one. I have no choice but to move everything to a temporary server and format. I wish I had a way to keep an image of the drive but it would be quite large.

What security measures would you suggest that I take in order to be able to track something like this better next time?

You mentioned tripwire and a few other things. What would have the least impact on resources?

Is there a way for me to log my last logs and maybe syslog to a different server so they cannot erase their tracks so easily?

Any suggestions are welcome.

Thanks so much for all the help so far!

-Curtis

unSpawn · 03-26-2011, 08:15 PM

Quote:

Originally Posted by curtisb

it continued to happen. Someone has definitely gotten me on this one. I have no choice but to move everything to a temporary server and format.

Before you do please allow us to work with you on this. These days a common point of entry for the majority of attacks is through the web stack caused by for instance a lack of hardening, misconfiguration and running stale versions of (often: PHP-based) software. Since most attacks are messy and noisy if you have system and daemon logs and auth records, preferably dating back to before this all started, CYP run them through Logwatch? It's a quick and easy way to generate leads. Also please don't say "nothing out of the ordinary showed up" but post requested output (that is unless you know exactly what you're hunting for): being exact and detailed in your replies could make all of this more efficient. I'd also request a detailed listing of files you found (if any) in directories the web server user can write to.