Importing blocklists into iptables

fruitwerks · 04-01-2009, 09:30 AM

I have a nice huge list in CIDR notation that I would like to import into my firewall script. I have tried a few different things but the code below appears to be the most promising.

Code:

cat /root/cidr_block.log | while read address; do
  /sbin/iptables -I INPUT -s "$address" -j DROP
done

but that does not do the trick...

Code:

' specified.4.2: invalid mask `22

here is the contents of the file (just for testing!)

Code:

4.2.153.32/22

now I can put a single ip or any other valid netblock and it still doesn't like it!

I have about 66,000 ranges I need to import so doing it by hand is not an option.

anyone seen this? My iptables works and I can manually add these 'non working' ranges to my script and it works. I get some interesting results when I try to echo...

Code:

# cat /root/cidr_block.log | while read address; do   echo /sbin/iptables -I INPUT -s "$address" -j DROP; done
 -j DROPtables -I INPUT -s 4.2.153.32/22

----

Dell PE2850 - Gentoo 2.6.27-gentoo-r8 #7 SMP Wed Mar 4 13:03:42 GMT 2009 x86_64 Intel(R) Xeon(TM) CPU 3.00GHz GenuineIntel GNU/Linux

theNbomr · 04-01-2009, 10:33 AM

You must have some extraneous data in your log file.

LQfruitwerks.sh:

Code:

#! /bin/bash
while read address; do
  /sbin/iptables -I INPUT -s "$address" -j DROP
done < LQfruitwerks.dat

LQfruitwerks.dat:

Code:

4.2.153.32/22

Code:

./LQfruitwerks.sh
/sbin/iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  4.2.152.0/22         anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

66,000 rules seems like a lot of processing on every received packet. I would question whether performance will suffer significantly. Perhaps you could report back on that once you get it running.

--- rod

win32sux · 04-01-2009, 11:06 AM

Quote:

Originally Posted by theNbomr

66,000 rules seems like a lot of processing on every received packet. I would question whether performance will suffer significantly.

I would be seriously concerned about that many blacklist rules too. To avoid performance issues, I recommend a separate chain with the IPs, where only packets in state NEW get sent. Example:

Code:

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -m state --state NEW -j CHECK_IP

fruitwerks · 04-01-2009, 05:17 PM

actually it is more like 240,000 rules and after dos2unix it appears there is a limitation because after testing a few lists, they all bork out on line 823.. some type of limitation I suppose.

Quote:

I would be seriously concerned about that many blacklist rules too. To avoid performance issues, I recommend a separate chain with the IPs, where only packets in state NEW get sent

That would be nice but some of these are advertising servers and that would allow the ads to trickle in. If I split up the lists, it makes more entries because some of them overlap. I simply need to block them totally.

Thanks!

win32sux · 04-01-2009, 06:22 PM

Quote:

Originally Posted by fruitwerks

actually it is more like 240,000 rules and after dos2unix it appears there is a limitation because after testing a few lists, they all bork out on line 823.. some type of limitation I suppose.

That seems weird. Can you post a link to this list so we can test it ourselves?

Quote:

That would be nice but some of these are advertising servers and that would allow the ads to trickle in.

Not sure I follow, as iptables won't care what kind of servers they are. When you stop connections from starting to/from an IP, the restriction applies to any IP-based traffic.

Quote:

If I split up the lists, it makes more entries because some of them overlap. I simply need to block them totally.

Once again, it would be great if you could post a link so we could try it and compare results.

fruitwerks · 04-01-2009, 06:28 PM

http://fruitwerks.us/cidr_block.log
or
http://fruitwerks.us/cidr_block.tar.gz

fruitwerks · 04-02-2009, 01:49 PM

Well I am currently importing the list within bash not from within my firewall script. I guess I start using iptables as service if this works. I guess it likes to bork in the script itself.

Just wondering what do most people use when they need to block this many addresses?

Thanks!

theNbomr · 04-02-2009, 01:57 PM

Hmm. I tried to download your list but my browser just times out. Perhaps you have me blacklisted... :-)
--- rod.

fruitwerks · 04-02-2009, 02:04 PM

hmm well then... here it is elsewhere http://www.megaupload.com/?d=DD6MAMAH

theNbomr · 04-03-2009, 12:56 AM

Okay, so I tried your data file, and stopped it after about 8000 entries (I added an echo line to the script so I could monitor the progress). It felt like the script was slowing down as it progressed. I didn't notice any slow-down of web browsing traffic, although admittedly, this is not a very scientific test. I had to leave my desk, and so I stopped the script. I may try letting it run much longer, and see what effect there is. At any rate, there was no evidence that it had crashed or hung up in any way. This was done on a Fedora-9 system, with a custom 2.6.27 kernel.
--- rod.

win32sux · 04-03-2009, 08:35 AM

I haven't looked at your file (couldn't get it to download yesterday), but I generated my own list of 240,000 random subnets to test with. The rules loaded fine up until 119,219 (at that point, iptables ran into a memory allocation problem).

fruitwerks · 04-03-2009, 01:42 PM

very interesting... I attempted to load the whole list as well, but I didn't have the time to sit and watch. I got it down to 2853 rules for now.

So how exactly would this work? I could use this for the larger set of rules. But again, I would like them to be dropped of course.

Quote:

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -m state --state NEW -j CHECK_IP

anyway here is my script.. should have posted this long ago!

Code:

#!/bin/bash

SYSCTL="/sbin/sysctl -w" 
IPT="/sbin/iptables"
INET_IFACE="eth1"
LOCAL_IFACE="eth0"
LOCAL_IP="172.24.0.8"
LOCAL_NET="172.24.0.0/24"
LOCAL_BCAST="172.24.0.255"
LO_IFACE="lo"
LO_IP="127.0.0.1"

/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe iptable_mangle
/sbin/modprobe iptable_nat
/sbin/modprobe xt_state

echo "1" > /proc/sys/net/ipv4/ip_forward
echo "1" > /proc/sys/net/ipv4/tcp_syncookies
echo "1" > /proc/sys/net/ipv4/conf/all/rp_filter
echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo "0" > /proc/sys/net/ipv4/conf/all/accept_source_route
echo "1" > /proc/sys/net/ipv4/conf/all/secure_redirects
echo "1" > /proc/sys/net/ipv4/conf/all/log_martians

$IPT -P INPUT ACCEPT
$IPT -P FORWARD ACCEPT
$IPT -P OUTPUT ACCEPT
$IPT -t nat -P PREROUTING ACCEPT
$IPT -t nat -P POSTROUTING ACCEPT
$IPT -t nat -P OUTPUT ACCEPT
$IPT -t mangle -P PREROUTING ACCEPT
$IPT -t mangle -P OUTPUT ACCEPT
$IPT -F
$IPT -t nat -F
$IPT -t mangle -F
$IPT -X
$IPT -t nat -X
$IPT -t mangle -X

$IPT -P INPUT DROP
$IPT -P OUTPUT DROP
$IPT -P FORWARD DROP

$IPT -N bad_packets
$IPT -N bad_tcp_packets
$IPT -N icmp_packets
$IPT -N udp_inbound
$IPT -N udp_outbound
$IPT -N tcp_inbound
$IPT -N tcp_outbound

$IPT -A bad_packets -p ALL -i $INET_IFACE -s $LOCAL_NET -j DROP
$IPT -A bad_packets -p ALL -m state --state INVALID -j DROP
$IPT -A bad_packets -p tcp -j bad_tcp_packets
$IPT -A bad_packets -p ALL -j RETURN

$IPT -A bad_tcp_packets -p tcp -i $LOCAL_IFACE -j RETURN
$IPT -A bad_tcp_packets -p tcp -i $LOCAL_IFACE ! --syn -m state --state NEW -j DROP
$IPT -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "fp=bad_tcp_packets:1 a=DROP "
$IPT -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL NONE -j LOG --log-prefix "fp=bad_tcp_packets:2 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL NONE -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL ALL -j LOG --log-prefix "fp=bad_tcp_packets:3 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL ALL -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL FIN,URG,PSH -j LOG --log-prefix "fp=bad_tcp_packets:4 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j LOG --log-prefix "fp=bad_tcp_packets:5 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,RST SYN,RST -j LOG --log-prefix "fp=bad_tcp_packets:6 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,FIN SYN,FIN -j LOG --log-prefix "fp=bad_tcp_packets:7 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
$IPT -A bad_tcp_packets -p tcp -j RETURN

$IPT -A icmp_packets --fragment -p ICMP -j LOG --log-prefix "fp=icmp_packets:1 a=DROP "
$IPT -A icmp_packets --fragment -p ICMP -j DROP

$IPT -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j LOG --log-prefix "fp=icmp_packets:2 a=ACCEPT "
$IPT -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j ACCEPT
$IPT -A icmp_packets -p ICMP -s 0/0 --icmp-type 11 -j ACCEPT
$IPT -A icmp_packets -p ICMP -j RETURN

$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 137 -j DROP
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 138 -j DROP
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 113 -j REJECT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 113 -j ACCEPT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 123 -j ACCEPT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 4779 -j ACCEPT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 4780 -j ACCEPT

$IPT -A udp_inbound -p UDP -s 0/0 --source-port 67 --destination-port 68 -j ACCEPT
$IPT -A udp_inbound -p UDP -j RETURN
$IPT -A udp_outbound -p UDP -s 0/0 -j ACCEPT

$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 80 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 113 -j REJECT
#$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 21 -j ACCEPT
#$IPT -A tcp_inbound -p TCP -s 0/0 --source-port 20 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 22 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 25 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 110 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 143 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 900:901 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 4776 -j ACCEPT
$IPT -A tcp_inbound -p TCP -j RETURN
$IPT -A tcp_outbound -p TCP -s 0/0 -j ACCEPT

$IPT -A INPUT -p ALL -i $LO_IFACE -j ACCEPT
$IPT -A INPUT -p ALL -j bad_packets
$IPT -A INPUT -p ALL -d 224.0.0.1 -j DROP
$IPT -I INPUT -s 81.157.0.0/16 -j DROP
$IPT -I INPUT -s 220.191.0.0/16 -j DROP
$IPT -I INPUT -s 60.12.0.0/16 -j DROP
$IPT -I INPUT -s 219.142.0.0/16 -j DROP
$IPT -I INPUT -s 202.205.0.0/16 -j DROP
$IPT -I INPUT -s 211.140.0.0/16 -j DROP
$IPT -I INPUT -s 84.38.0.0/16 -j DROP
$IPT -I INPUT -s 10.0.0.0/8 -j DROP
$IPT -I INPUT -s 192.0.0.0/8 -j DROP
$IPT -I INPUT -s 85.255.0.0/8 -j DROP

cat /root/cidr_block.log | while read address; do
  $IPT -I INPUT -s "$address" -j DROP
done

$IPT -A INPUT -p ALL -i $LOCAL_IFACE -s $LOCAL_NET -j ACCEPT
$IPT -A INPUT -p ALL -i $LOCAL_IFACE -d $LOCAL_BCAST -j ACCEPT
$IPT -A INPUT -p UDP -i $LOCAL_IFACE --source-port 68 --destination-port 67 -j ACCEPT
$IPT -A INPUT -p ALL -i $INET_IFACE -m state --state ESTABLISHED,RELATED -j ACCEPT

$IPT -A INPUT -p TCP -i $INET_IFACE -j tcp_inbound
$IPT -A INPUT -p UDP -i $INET_IFACE -j udp_inbound
$IPT -A INPUT -p ICMP -i $INET_IFACE -j icmp_packets
$IPT -A INPUT -m pkttype --pkt-type broadcast -j DROP

$IPT -A FORWARD -p ALL -j bad_packets

$IPT -A FORWARD -p tcp -i $LOCAL_IFACE -j tcp_outbound
$IPT -A FORWARD -p udp -i $LOCAL_IFACE -j udp_outbound
$IPT -A FORWARD -p ALL -i $LOCAL_IFACE -j ACCEPT
$IPT -A FORWARD -i $INET_IFACE -m state --state ESTABLISHED,RELATED -j ACCEPT

$IPT -A FORWARD -j LOG --log-prefix "fp=FORWARD:99 a=DROP "

$IPT -A OUTPUT -m state -p icmp --state INVALID -j DROP
$IPT -A OUTPUT -p ALL -s $LO_IP -j ACCEPT
$IPT -A OUTPUT -p ALL -o $LO_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -s $LOCAL_IP -j ACCEPT
$IPT -A OUTPUT -p ALL -o $LOCAL_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $INET_IFACE -j ACCEPT
$IPT -A OUTPUT -j LOG --log-prefix "fp=OUTPUT:99 a=DROP "

$IPT -t nat -A POSTROUTING -o $INET_IFACE -j MASQUERADE
$IPT -t mangle -A OUTPUT -o $INET_IFACE -j TTL --ttl-set 128

win32sux · 04-03-2009, 03:10 PM

Quote:

Originally Posted by fruitwerks

So how exactly would this work? I could use this for the larger set of rules. But again, I would like them to be dropped of course.

You just load the rules into the CHECK_IP chain, then send all packets in state NEW through it. If the connection is being initiated by a banned IP, it won't be allowed to proceed. If it isn't, the packet will continue downward, through whatever chain sent the packet to CHECK_IP originally. Packets which are part of a connection which has already been initiated won't get sent to CHECK_IP.

fruitwerks · 04-03-2009, 03:16 PM

So I would still need to load the rules? or can I tell it to reference the blocklist without loading the rules?

I wrote this firewall a while ago and I'm not good with scripting / coding, so I'm confused by some things in the script lol.

anomie · 04-03-2009, 03:16 PM

Quote:

Originally Posted by win32sux

I recommend a separate chain with the IPs, where only packets in state NEW get sent. Example:

Code:

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -m state --state NEW -j CHECK_IP

This is what I do. You want your INPUT chain to be clear and easy to read / maintain.

If you are only blacklisting for certain services, and can add extra criteria to the -j CHECK_IP rule, all the better. For example, if you are only blacklisting those addresses to tcp port 22 (ssh), then add the appropriate criteria to the rule so that you're not traversing CHECK_IP for every single packet that enters the INPUT chain.