Rouge source using my ssh

thetawaverider · 07-25-2007, 06:09 PM

Hi,

I have a PHP-based site that appears to have been compromised. Running tcpdump results in entries like:

Code:

19:00:10.882396 IP dialup-4.231.2.175.Dial1.Houston1.Level3.net.49754 > 64.x.x.x.http: S 3394384288:3394384288(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK>
19:00:11.941656 IP dialup-4.231.2.175.Dial1.Houston1.Level3.net.49753 > 64.x.x.x.http: . ack 257740321 win 2048
19:00:11.971062 IP dialup-4.231.2.175.Dial1.Houston1.Level3.net.49753 > 64.x.x.x.http: P 0:495(495) ack 1 win 2048
19:00:12.246219 IP dialup-4.231.2.175.Dial1.Houston1.Level3.net.49754 > 64.x.x.x.http: . ack 257282436 win 2048
19:00:12.246236 IP dialup-4.231.2.175.Dial1.Houston1.Level3.net.49754 > 64.x.x.x.http: P 0:501(501) ack 1 win 2048

This site is not even in production (though it originated as a copy of a production site), so I know that the server is being used for nefarious means. The 64.x.x.x IP is not mine, but is that of a server on the same subnet as me, where this server is being attacked via my server. I've set up some pretty restrictive iptables rules:

Code:

#!/bin/bash

iptables -F -t filter
iptables -X
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
iptables -A INPUT -p tcp --dport 22 -s 63.x.x.x -d mymachine.mydomain.com -j ACCEPT
iptables -A OUTPUT -p tcp --sport 22 -s mymachine.mydomain.com -d 63.x.x.x -j ACCEPT

where only my IP (63.x.x.x) is allowed in and out over SSH. Even with these rules present, the tcpdump is still the same - that is, my subnet neighbor is still being attacked.

If anyone has any advice to give, I'm all ears (and all thanks too).

-TWR

thetawaverider · 07-25-2007, 06:59 PM

Just ran:

Code:

tcpdump port 22 | grep "64.x.x.x"

and

Code:

tcpdump port 80 | grep "64.x.x.x"

and both displayed the rouge output.

Strange, since all traffic except port 22 to/from my localhost is blocked.

-TWR

thetawaverider · 07-25-2007, 07:38 PM

Okay, added more rules, as port 80 is not dropped by the initial INPUT/OUTPUT DROP (thought it was). Definitely more than what is needed, but doing it just to demonstrate the point.

Code:

#!/bin/bash

iptables -F -t filter
iptables -X
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
iptables -A INPUT -p tcp --dport 22 -s 63.x.x.x -d mymachine.mydomain.com -j ACCEPT
iptables -A OUTPUT -p tcp --sport 22 -s mymachine.mydomain.com -d 63.x.x.x -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j DROP
iptables -A INPUT -p tcp --sport 80 -j DROP
iptables -A OUTPUT -p tcp --dport 80 -j DROP
iptables -A OUTPUT -p tcp --sport 80 -j DROP

Anything to do with port 80 both in INPUT and OUTPUT should be dropped. But it's not - the tcpdump still has entries with http as the destination port.

Could it be the case that my rules are being modified by a bot infecting my system? If so, it is not possible to tell with an "iptables -L", which shows:

Code:

Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  63.x.x.x             mymachine.mydomain.com tcp dpt:ssh 
DROP       tcp  --  anywhere             anywhere            tcp dpt:http 
DROP       tcp  --  anywhere             anywhere            tcp spt:http 

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  mymachine.mydomain.com  63.x.x.x      tcp spt:ssh 
DROP       tcp  --  anywhere             anywhere            tcp dpt:http 
DROP       tcp  --  anywhere             anywhere            tcp spt:http

-TWR

Matir · 07-25-2007, 10:22 PM

What makes you think those packets are from your machine? If there's only a hub between you and the other machine, then you could see traffic to it on tcpdump.

unixfool · 07-25-2007, 11:37 PM

Quote:

Originally Posted by Matir

What makes you think those packets are from your machine? If there's only a hub between you and the other machine, then you could see traffic to it on tcpdump.

I agree.

A few things...

tcpdump will do you no good if you don't know how to tell the packets are being dropped. Look at the TCP flags in the dump to see at what state the connections are.

Also, if you're sniffing before your firewall, you're going to see the traffic, regardless if you've blocked it or not.

Because you've a firewall and a packet trace, you have to remember to factor in how to read your sniffed logs, where you're sniffing at, and what type of device your interface is connected to.

thetawaverider · 07-26-2007, 12:11 AM

I think you may be right on the money - this is most likely traffic destined for a local machine. I've been a bit cautious recently (overly so), and here's why:

Recently, disabling port 443 caused a substantial decrease in the number of idle httpd processes. The site doesn't even use port 443, yet the number of idle processes was constantly 20-40% of the active ones (much greater than MaxSpareServers), leading me to believe that the targeting of port 443 was intentional. After this change, my fork rate (as per munin) jumped from near zero to a nice cycle between 15 and 25 forks/second.

Upon a recent reboot of the server, though, this large number of idle connections returned almost immediately, plus the fork rate dropped to near zero again. This machine has been the target of DDoS attacks in the past, plus has been infected via a hole in the PHP application (which is not current due to it's eminent replacement), so I wanted to make sure that the reboot did not trigger any sort of latent bot. It is possible that I may have been looking for a problem where none exists, but I'd rather be a bit over cautious in a situation like this.

The idle httpd connection problem does still exist, though, so if anyone has any suggestions, I'd like to hear them.

Thanks,
TWR