I have been cracked so if you can help me out it would be greatly appreciated. I am running a debian sarge system with apache2 and mySQL. I am running some php apps so I am guessing this is where the problem lies. I have a cron job that tests my dynamic address every 5 minutes and updates a zone file on zoneedit.com. The web server uses port 8080 since verizon blocks port 80. This is a totally non critical web site that I set up just for fun and to learn. I had been cracked in the past due to bad PHP settings but figured all that out on my own. I guess I am greatfull for script kiddies since I am learning how to stop them

Anyway I went to my home page today and there was a time out error so I ssh into the server try to restart:

hereweare:/# /etc/init.d/apache2 restart
Forcing reload of web server: Apache2 ... no pidfile found! not running?(98)Address already in use: make_sock: could not bind to address 192.168.1.3:8080
no listening sockets available, shutting down
Unable to open logs
hereweare:/#

///So I ran ps aux to see if there are any odd processes

hereweare:/usr/sbin# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 1492 508 ? S Nov27 0:05 init [2]
root 2 0.0 0.0 0 0 ? S Nov27 0:00 [kflushd]
root 3 0.0 0.0 0 0 ? S Nov27 3:31 [kupdate]
root 4 0.0 0.0 0 0 ? S Nov27 0:00 [kswapd]
root 5 0.0 0.0 0 0 ? S Nov27 0:00 [keventd]
daemon 258 0.0 0.1 1608 452 ? Ss Nov27 0:00 /sbin/portmap
root 336 0.0 0.2 2244 828 ? Ss Nov27 1:21 /sbin/syslogd
root 339 0.0 0.2 1832 936 ? Ss Nov27 0:00 /sbin/klogd
bind 356 0.0 0.6 11144 2588 ? Ss Nov27 0:00 /usr/sbin/named -u bind
bind 357 0.0 0.6 11144 2588 ? S Nov27 0:00 /usr/sbin/named -u bind
bind 358 0.0 0.6 11144 2588 ? S Nov27 0:00 /usr/sbin/named -u bind
bind 359 0.0 0.6 11144 2588 ? S Nov27 0:00 /usr/sbin/named -u bind
bind 360 0.0 0.6 11144 2588 ? S Nov27 0:00 /usr/sbin/named -u bind
root 373 0.0 0.1 2220 764 ? Ss Nov27 0:01 /usr/sbin/inetd
root 392 0.0 0.3 2496 1240 ? S Nov27 0:00 sh /usr/bin/mysqld_safe
root 428 0.0 0.3 2496 1244 ? S Nov27 0:00 sh /usr/bin/mysqld_safe
mysql 429 0.0 8.3 77120 32472 ? S Nov27 0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
root 430 0.0 0.1 1476 488 ? S Nov27 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
mysql 433 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 434 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 435 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 436 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 437 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 438 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 439 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 440 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
mysql 441 0.0 8.3 77120 32472 ? S Nov27 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/va
root 567 0.0 0.3 3656 1332 ? Ss Nov27 0:01 /usr/lib/postfix/master
postfix 573 0.0 0.3 2996 1260 ? S Nov27 0:00 qmgr -l -t fifo -u -c
root 585 0.0 0.5 5764 1952 ? Ss Nov27 0:08 /usr/sbin/nmbd -D
root 587 0.0 0.6 8124 2572 ? Ss Nov27 0:02 /usr/sbin/smbd -D
root 590 0.0 0.6 8120 2516 ? S Nov27 0:00 /usr/sbin/smbd -D
root 599 0.0 0.4 3720 1548 ? Ss Nov27 0:03 /usr/sbin/sshd
root 610 0.0 0.8 4636 3108 ? Ss Nov27 0:00 /usr/bin/X11/xfs -daemon
root 708 0.0 0.3 2736 1488 ? S Nov27 0:00 bash /etc/rc2.d/S20xprint start
root 712 0.0 0.3 2736 1488 ? S Nov27 0:00 bash /etc/rc2.d/S20xprint start
root 713 0.0 0.3 2736 1492 ? S Nov27 0:00 bash /etc/rc2.d/S20xprint start
root 714 0.0 0.8 11724 3256 ? S Nov27 0:00 /usr/X11R6/bin/Xprt -ac -pn -nolisten tcp -audit 4 -fp /usr/X11R6/lib/X11/fonts/Type
root 721 0.0 0.2 2368 932 ? Ss Nov27 0:00 /sbin/rpc.statd
daemon 745 0.0 0.1 1676 644 ? Ss Nov27 0:00 /usr/sbin/atd
root 767 0.0 0.2 1756 820 ? Ss Nov27 0:00 /usr/sbin/cron
root 789 0.0 0.1 1484 472 tty2 Ss+ Nov27 0:00 /sbin/getty 38400 tty2
root 790 0.0 0.1 1484 472 tty3 Ss+ Nov27 0:00 /sbin/getty 38400 tty3
root 791 0.0 0.1 1484 472 tty4 Ss+ Nov27 0:00 /sbin/getty 38400 tty4
root 792 0.0 0.1 1484 472 tty5 Ss+ Nov27 0:00 /sbin/getty 38400 tty5
root 793 0.0 0.1 1484 472 tty6 Ss+ Nov27 0:00 /sbin/getty 38400 tty6
root 4756 0.0 0.1 1484 472 tty1 Ss+ Dec07 0:00 /sbin/getty 38400 tty1
www-data 12782 0.0 0.0 1344 328 ? Ss Dec10 0:00 bash
snort 13998 0.0 9.7 41796 37736 ? Ss Dec10 0:07 /usr/sbin/snort -m 027 -D -d -l /var/log/snort -u snort -g snort -c /etc/snort/snort
lp 14529 0.0 0.2 2452 868 ? Ss Dec10 0:00 /usr/sbin/lpd -s
root 20196 0.0 0.6 7176 2344 ? Ss 13:43 0:00 sshd: jmckenna [priv]
root 20198 0.0 0.6 7176 2344 ? S 13:43 0:00 sshd: jmckenna [priv]
jmckenna 20200 0.0 0.6 9232 2412 ? S 13:43 0:00 sshd: jmckenna@pts/1
jmckenna 20201 0.0 0.3 2564 1416 pts/1 Ss 13:43 0:00 -bash
root 20202 0.0 0.3 2528 1396 pts/1 S 13:43 0:00 bash
postfix 20213 0.0 0.3 2964 1164 ? S 13:57 0:00 pickup -l -t fifo -u -c
root 20361 0.0 0.2 2480 856 pts/1 R+ 15:05 0:00 ps aux
hereweare:/usr/sbin#

/// I found the bash process run by www-data to be odd (10th one from the bottom). Checking my http port...

hereweare:/usr/sbin# lsof -i :8080
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
a1 12782 www-data 3u IPv4 63318 TCP hereweare:webcache (LISTEN)
a1 12782 www-data 9u IPv4 228882 TCP hereweare:webcache->serv-1-3-194.lycos-vds.com:57726 (CLOSE)
hereweare:/usr/sbin#

///WTH is this lycos crap? Ok I got cracked for sure.

hereweare:/usr/sbin# netstat -nap --inet |grep 8080 |grep LISTEN
tcp 0 0 192.168.1.3:8080 0.0.0.0:* LISTEN 12782/bash
hereweare:/usr/sbin#

///Now is this really bash or did the cracker use the name bash to run some nasty things?

hereweare:/usr/sbin# lsof -p 12782
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
a1 12782 www-data cwd DIR 3,1 4096 2 /
a1 12782 www-data rtd DIR 3,1 4096 2 /
a1 12782 www-data txt REG 3,1 12336 724862 /tmp/del/a1
a1 12782 www-data mem REG 3,1 90248 160983 /lib/ld-2.3.2.so
a1 12782 www-data mem REG 3,1 1244752 160986 /lib/libc-2.3.2.so
a1 12782 www-data 0u CHR 1,3 177169 /dev/null
a1 12782 www-data 1u CHR 1,3 177169 /dev/null
a1 12782 www-data 2u CHR 1,3 177169 /dev/null
a1 12782 www-data 3u IPv4 63318 TCP hereweare:webcache (LISTEN)
a1 12782 www-data 4r FIFO 0,0 63359 pipe
a1 12782 www-data 5w FIFO 0,0 63359 pipe
a1 12782 www-data 6w REG 3,1 13354 210377 /var/log/apache2/error.log.1
a1 12782 www-data 7w REG 3,1 159545 210034 /var/log/apache2/access.log.1
a1 12782 www-data 8w REG 3,1 0 209416 /var/run/apache2/ssl_mutex.16073 (deleted)
a1 12782 www-data 9u IPv4 228882 TCP hereweare:webcache->serv-1-3-194.lycos-vds.com:57726 (CLOSE)
a1 12782 www-data 10u unix 0xcf14cc60 228906 socket
a1 12782 www-data 11u IPv4 228932 TCP *:2345 (LISTEN)
hereweare:/usr/sbin#

/// this is very disturbing

hereweare:/usr/sbin# cd /tmp
hereweare:/tmp# ls -la
total 1944
drwxr-xr-x 2 www-data www-data 4096 Dec 7 09:36 "
drwxrwxrwt 8 root root 8192 Dec 12 06:25 .
drwxr-xr-x 23 root root 4096 Jul 13 11:54 ..
drwxrwxrwt 2 root root 4096 Nov 27 09:34 .ICE-unix
drwxrwxrwt 2 root root 4096 Nov 27 09:35 .X11-unix
-r--r--r-- 1 root root 11 Nov 27 09:35 .X64-lock
drwxrwxrwt 2 root root 4096 Nov 27 09:35 .font-unix
-rwxr-xr-x 1 www-data www-data 6762 Sep 23 08:19 1.c
drwx------ 2 www-data www-data 4096 Dec 10 06:20 del
-rw-r--r-- 1 www-data www-data 1296898 Dec 10 03:51 del.tgz
-rwxr-xr-x 1 www-data www-data 428460 Jan 10 2004 do
-rw-r--r-- 1 www-data www-data 195236 Nov 24 11:42 do.tgz
-rwxr-xr-x 1 www-data www-data 3120 Sep 23 08:17 elfcd1.c
drwx------ 2 www-data www-data 4096 Dec 10 06:20 zzz
hereweare:/tmp#

So how in the world is some one able to write and execute to my /tmp directory as www-data?

drwxrwxrwt 8 root root 8192 Dec 12 06:25 tmp

///Here is the coments of 1.c

hereweare:/tmp# cat 1.c | pager

/*
* expand_stack SMP race local root exploit
*
* Copyright (C) 2005 Christophe Devine and Julien Tinnes
*
* This program is quite unreliable - you may have to run it
* several times before getting a rootshell. It was only tested
* so far on a bi-xeon running Debian testing / Linux 2.4.29-rc1.
*
* Vulnerability discovered by Paul Starzetz <ihaquer at isec.pl>
* http://www.isec.pl/vulnerabilities/i...-pagefault.txt
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/

///And elfcd1.c

hereweare:/tmp# cat elfcd1.c | pager


/*
* Linux binfmt_elf core dump buffer overflow
*
* Copyright (c) 2005 iSEC Security Research. All Rights Reserved.
*
* THIS PROGRAM IS FOR EDUCATIONAL PURPOSES *ONLY* IT IS PROVIDED "AS IS"
* AND WITHOUT ANY WARRANTY. COPYING, PRINTING, DISTRIBUTION, MODIFICATION
* WITHOUT PERMISSION OF THE AUTHOR IS STRICTLY PROHIBITED.
*
*/


I guess this is a DOS

/// Looks pretty nasty to me.

I could probably just shut down apache and delete all the files but I really want to nip this in the bud since it's bound to come back.

If any one has any ideas please help me out thanks in advance.

redir

I am running some php apps so I am guessing this is where the problem lies.
If you do then you probably also know you have been running obsolete versions of PHP apps or have them configured in less secure ways?.. Anyway. I guess most here know the drill by now:

1) read the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html. Since this looks like a breach with a known point of entry (as far as I am allowed to draw conclusions from the data you presented) I'd start by
2) mitigating the situation. After you've made full process (ps axfwwwe), open files (lsof -w -n) and network (netstat -nap) listings, proceed by shutting out users and shutting down and killing services that are unnecessary for admin tasks. Next raise the firewall so only your management IP (range) has access. Now you've made the box more managable, return to point 1 and act on it. Give priority to manually checking logs and auth data and verifying the integrity of the system (package manager, file integrity checker if installed) and proceed with a quick audit running Chkrootkit and/or Rootkit Hunter.

I. I could probably just shut down apache and delete all the files but I really want to nip this in the bud since it's bound to come back.
Start by checking services and apps you use for obsolete versions, configured using insecure settings and accessability.

II. If you have no independant and authoritative means to verify the integrity of your system you may skip presenting output of your audit and proceed with the next steps:

3) prepare backups (contaminated: store separate for perusal, not reuse),
4) repartition, reformat, re-install from scratch,
5) harden. Check out the LQ FAQ: Security references.


HTH

// BTW, temp dirs del, do, and zzz prolly contain flooders and bouncers. I'd like to request a tarball with del.tgz, do.tgz, and zzz anyway. If you can, please email me the D/L location. TIA

I was going to request the same thing Then again, i don't appear as trust-worthy since I no longer carry the mod tag... c'est la vie.

What's really interesting is that it appears the crackers simply copied proof-of-concept exploits released by researchers, verbatim. They didn't even bother to remove the copyright (who is going to enforce copyrights on exploit code? LOL?). It certainly lends credance to the claim by vendors that releasing PoC code simply hands tools to evil-doers who wouldn't have otherwise had access to them.

BTW 1.c and elfcd1.c are both local-root exploits. I guess the attacker downloaded multiple tools because they weren't certain which tool would work, or they tried one first and it didn't work, so they had to fetch another one.

Basically what happend was they found a vulnerable PHP app (most likely) that allowed them to run a reverse shell in place of your normal Apache process. After that they logged in and downloaded their tools... Actually, it's likely that the exploits failed for some reason. Either they could compile, or your system wasn't vulnerable. If they had worked, they would have root and you would probably be locked out.

I was going to request the same thing. Then again, I don't appear as trust-worthy since I no longer carry the mod tag... c'est la vie.
@chort: IMHO this has nothing to do with me still being or you no longer being a moderator but with a sincere interest in all things security. The only way to show that is by maintaining a regular presence in the LQ Linux Security forum and making consistent and constant quality posts in a way members recognise as respectful and helpful.
@OP: While I do not *explicitly* know chorts intentions with said material, I have taken good notice of chorts' (recent) posting history, and I do trust him to use it for research as would I (me being one of the LQ Linux Security moderators and Rootkit Hunter project lead amongst other things). Please note sharing this kind of material should be done outside of LQ fora so they don't get tainted.


It certainly lends credance to the claim by vendors that releasing PoC code simply hands tools to evil-doers who wouldn't have otherwise had access to them.
An interesting opinion wrt Full Disclosure but I would rather not see this thread expand on (or derail because of) a discussion of that. IMNSHO it is a highly flammable topic. If you want to pursue this best don your asbestos suit and open a new thread :-]

Thanks for your help. Check your PM's. I see no harm in anyone looking over these files, the more the merrier. I will play around a bit this morning and then I am off for the week end. Again thanks for your time.

redir: Sucks about your box. You can mount your /tmp with various options to help prevent this sort of thing from happening. I think the common recommendation is for nosuid,noexec,nodev. This should dissuade at least the skiddies. Granted your /tmp can still be remounted if the box is owned, but security is layers and this is just another layer. Defintely read the link unSpawn posted, though.