LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-01-2009, 10:30 AM   #1
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Rep: Reputation: 15
Unhappy Importing blocklists into iptables


I have a nice huge list in CIDR notation that I would like to import into my firewall script. I have tried a few different things but the code below appears to be the most promising.

Code:
cat /root/cidr_block.log | while read address; do
  /sbin/iptables -I INPUT -s "$address" -j DROP
done
but that does not do the trick...

Code:
' specified.4.2: invalid mask `22
here is the contents of the file (just for testing!)

Code:
4.2.153.32/22
now I can put a single ip or any other valid netblock and it still doesn't like it!

I have about 66,000 ranges I need to import so doing it by hand is not an option.

anyone seen this? My iptables works and I can manually add these 'non working' ranges to my script and it works. I get some interesting results when I try to echo...

Code:
# cat /root/cidr_block.log | while read address; do   echo /sbin/iptables -I INPUT -s "$address" -j DROP; done
 -j DROPtables -I INPUT -s 4.2.153.32/22
----

Dell PE2850 - Gentoo 2.6.27-gentoo-r8 #7 SMP Wed Mar 4 13:03:42 GMT 2009 x86_64 Intel(R) Xeon(TM) CPU 3.00GHz GenuineIntel GNU/Linux
 
Old 04-01-2009, 11:33 AM   #2
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,398
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
You must have some extraneous data in your log file.

LQfruitwerks.sh:
Code:
#! /bin/bash
while read address; do
  /sbin/iptables -I INPUT -s "$address" -j DROP
done < LQfruitwerks.dat
LQfruitwerks.dat:
Code:
4.2.153.32/22
Code:
./LQfruitwerks.sh
/sbin/iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  4.2.152.0/22         anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
66,000 rules seems like a lot of processing on every received packet. I would question whether performance will suffer significantly. Perhaps you could report back on that once you get it running.


--- rod

Last edited by theNbomr; 04-01-2009 at 11:35 AM.
 
Old 04-01-2009, 12:06 PM   #3
win32sux
LQ Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 380Reputation: 380Reputation: 380Reputation: 380
Quote:
Originally Posted by theNbomr View Post
66,000 rules seems like a lot of processing on every received packet. I would question whether performance will suffer significantly.
I would be seriously concerned about that many blacklist rules too. To avoid performance issues, I recommend a separate chain with the IPs, where only packets in state NEW get sent. Example:
Code:
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -m state --state NEW -j CHECK_IP
 
Old 04-01-2009, 06:17 PM   #4
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Original Poster
Rep: Reputation: 15
actually it is more like 240,000 rules and after dos2unix it appears there is a limitation because after testing a few lists, they all bork out on line 823.. some type of limitation I suppose.

Quote:
I would be seriously concerned about that many blacklist rules too. To avoid performance issues, I recommend a separate chain with the IPs, where only packets in state NEW get sent
That would be nice but some of these are advertising servers and that would allow the ads to trickle in. If I split up the lists, it makes more entries because some of them overlap. I simply need to block them totally.

Thanks!
 
Old 04-01-2009, 07:22 PM   #5
win32sux
LQ Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 380Reputation: 380Reputation: 380Reputation: 380
Quote:
Originally Posted by fruitwerks View Post
actually it is more like 240,000 rules and after dos2unix it appears there is a limitation because after testing a few lists, they all bork out on line 823.. some type of limitation I suppose.
That seems weird. Can you post a link to this list so we can test it ourselves?

Quote:
That would be nice but some of these are advertising servers and that would allow the ads to trickle in.
Not sure I follow, as iptables won't care what kind of servers they are. When you stop connections from starting to/from an IP, the restriction applies to any IP-based traffic.

Quote:
If I split up the lists, it makes more entries because some of them overlap. I simply need to block them totally.
Once again, it would be great if you could post a link so we could try it and compare results.
 
Old 04-01-2009, 07:28 PM   #6
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Original Poster
Rep: Reputation: 15
http://fruitwerks.us/cidr_block.log
or
http://fruitwerks.us/cidr_block.tar.gz

 
Old 04-02-2009, 02:49 PM   #7
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Original Poster
Rep: Reputation: 15
Well I am currently importing the list within bash not from within my firewall script. I guess I start using iptables as service if this works. I guess it likes to bork in the script itself.

Just wondering what do most people use when they need to block this many addresses?

Thanks!
 
Old 04-02-2009, 02:57 PM   #8
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,398
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Hmm. I tried to download your list but my browser just times out. Perhaps you have me blacklisted... :-)
--- rod.
 
Old 04-02-2009, 03:04 PM   #9
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Original Poster
Rep: Reputation: 15
hmm well then... here it is elsewhere http://www.megaupload.com/?d=DD6MAMAH
 
Old 04-03-2009, 01:56 AM   #10
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,398
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Okay, so I tried your data file, and stopped it after about 8000 entries (I added an echo line to the script so I could monitor the progress). It felt like the script was slowing down as it progressed. I didn't notice any slow-down of web browsing traffic, although admittedly, this is not a very scientific test. I had to leave my desk, and so I stopped the script. I may try letting it run much longer, and see what effect there is. At any rate, there was no evidence that it had crashed or hung up in any way. This was done on a Fedora-9 system, with a custom 2.6.27 kernel.
--- rod.
 
Old 04-03-2009, 09:35 AM   #11
win32sux
LQ Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 380Reputation: 380Reputation: 380Reputation: 380
I haven't looked at your file (couldn't get it to download yesterday), but I generated my own list of 240,000 random subnets to test with. The rules loaded fine up until 119,219 (at that point, iptables ran into a memory allocation problem).

Last edited by win32sux; 04-03-2009 at 11:03 AM.
 
Old 04-03-2009, 02:42 PM   #12
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Original Poster
Rep: Reputation: 15
very interesting... I attempted to load the whole list as well, but I didn't have the time to sit and watch. I got it down to 2853 rules for now.

So how exactly would this work? I could use this for the larger set of rules. But again, I would like them to be dropped of course.

Quote:
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -m state --state NEW -j CHECK_IP
anyway here is my script.. should have posted this long ago!

Code:
#!/bin/bash

SYSCTL="/sbin/sysctl -w" 
IPT="/sbin/iptables"
INET_IFACE="eth1"
LOCAL_IFACE="eth0"
LOCAL_IP="172.24.0.8"
LOCAL_NET="172.24.0.0/24"
LOCAL_BCAST="172.24.0.255"
LO_IFACE="lo"
LO_IP="127.0.0.1"

/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe iptable_mangle
/sbin/modprobe iptable_nat
/sbin/modprobe xt_state

echo "1" > /proc/sys/net/ipv4/ip_forward
echo "1" > /proc/sys/net/ipv4/tcp_syncookies
echo "1" > /proc/sys/net/ipv4/conf/all/rp_filter
echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo "0" > /proc/sys/net/ipv4/conf/all/accept_source_route
echo "1" > /proc/sys/net/ipv4/conf/all/secure_redirects
echo "1" > /proc/sys/net/ipv4/conf/all/log_martians

$IPT -P INPUT ACCEPT
$IPT -P FORWARD ACCEPT
$IPT -P OUTPUT ACCEPT
$IPT -t nat -P PREROUTING ACCEPT
$IPT -t nat -P POSTROUTING ACCEPT
$IPT -t nat -P OUTPUT ACCEPT
$IPT -t mangle -P PREROUTING ACCEPT
$IPT -t mangle -P OUTPUT ACCEPT
$IPT -F
$IPT -t nat -F
$IPT -t mangle -F
$IPT -X
$IPT -t nat -X
$IPT -t mangle -X

$IPT -P INPUT DROP
$IPT -P OUTPUT DROP
$IPT -P FORWARD DROP

$IPT -N bad_packets
$IPT -N bad_tcp_packets
$IPT -N icmp_packets
$IPT -N udp_inbound
$IPT -N udp_outbound
$IPT -N tcp_inbound
$IPT -N tcp_outbound

$IPT -A bad_packets -p ALL -i $INET_IFACE -s $LOCAL_NET -j DROP
$IPT -A bad_packets -p ALL -m state --state INVALID -j DROP
$IPT -A bad_packets -p tcp -j bad_tcp_packets
$IPT -A bad_packets -p ALL -j RETURN

$IPT -A bad_tcp_packets -p tcp -i $LOCAL_IFACE -j RETURN
$IPT -A bad_tcp_packets -p tcp -i $LOCAL_IFACE ! --syn -m state --state NEW -j DROP
$IPT -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "fp=bad_tcp_packets:1 a=DROP "
$IPT -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL NONE -j LOG --log-prefix "fp=bad_tcp_packets:2 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL NONE -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL ALL -j LOG --log-prefix "fp=bad_tcp_packets:3 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL ALL -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL FIN,URG,PSH -j LOG --log-prefix "fp=bad_tcp_packets:4 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j LOG --log-prefix "fp=bad_tcp_packets:5 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,RST SYN,RST -j LOG --log-prefix "fp=bad_tcp_packets:6 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,FIN SYN,FIN -j LOG --log-prefix "fp=bad_tcp_packets:7 a=DROP "
$IPT -A bad_tcp_packets -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
$IPT -A bad_tcp_packets -p tcp -j RETURN

$IPT -A icmp_packets --fragment -p ICMP -j LOG --log-prefix "fp=icmp_packets:1 a=DROP "
$IPT -A icmp_packets --fragment -p ICMP -j DROP

$IPT -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j LOG --log-prefix "fp=icmp_packets:2 a=ACCEPT "
$IPT -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j ACCEPT
$IPT -A icmp_packets -p ICMP -s 0/0 --icmp-type 11 -j ACCEPT
$IPT -A icmp_packets -p ICMP -j RETURN

$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 137 -j DROP
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 138 -j DROP
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 113 -j REJECT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 113 -j ACCEPT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 123 -j ACCEPT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 4779 -j ACCEPT
$IPT -A udp_inbound -p UDP -s 0/0 --destination-port 4780 -j ACCEPT

$IPT -A udp_inbound -p UDP -s 0/0 --source-port 67 --destination-port 68 -j ACCEPT
$IPT -A udp_inbound -p UDP -j RETURN
$IPT -A udp_outbound -p UDP -s 0/0 -j ACCEPT

$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 80 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 113 -j REJECT
#$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 21 -j ACCEPT
#$IPT -A tcp_inbound -p TCP -s 0/0 --source-port 20 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 22 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 25 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 110 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 143 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 900:901 -j ACCEPT
$IPT -A tcp_inbound -p TCP -s 0/0 --destination-port 4776 -j ACCEPT
$IPT -A tcp_inbound -p TCP -j RETURN
$IPT -A tcp_outbound -p TCP -s 0/0 -j ACCEPT

$IPT -A INPUT -p ALL -i $LO_IFACE -j ACCEPT
$IPT -A INPUT -p ALL -j bad_packets
$IPT -A INPUT -p ALL -d 224.0.0.1 -j DROP
$IPT -I INPUT -s 81.157.0.0/16 -j DROP
$IPT -I INPUT -s 220.191.0.0/16 -j DROP
$IPT -I INPUT -s 60.12.0.0/16 -j DROP
$IPT -I INPUT -s 219.142.0.0/16 -j DROP
$IPT -I INPUT -s 202.205.0.0/16 -j DROP
$IPT -I INPUT -s 211.140.0.0/16 -j DROP
$IPT -I INPUT -s 84.38.0.0/16 -j DROP
$IPT -I INPUT -s 10.0.0.0/8 -j DROP
$IPT -I INPUT -s 192.0.0.0/8 -j DROP
$IPT -I INPUT -s 85.255.0.0/8 -j DROP

cat /root/cidr_block.log | while read address; do
  $IPT -I INPUT -s "$address" -j DROP
done

$IPT -A INPUT -p ALL -i $LOCAL_IFACE -s $LOCAL_NET -j ACCEPT
$IPT -A INPUT -p ALL -i $LOCAL_IFACE -d $LOCAL_BCAST -j ACCEPT
$IPT -A INPUT -p UDP -i $LOCAL_IFACE --source-port 68 --destination-port 67 -j ACCEPT
$IPT -A INPUT -p ALL -i $INET_IFACE -m state --state ESTABLISHED,RELATED -j ACCEPT

$IPT -A INPUT -p TCP -i $INET_IFACE -j tcp_inbound
$IPT -A INPUT -p UDP -i $INET_IFACE -j udp_inbound
$IPT -A INPUT -p ICMP -i $INET_IFACE -j icmp_packets
$IPT -A INPUT -m pkttype --pkt-type broadcast -j DROP

$IPT -A FORWARD -p ALL -j bad_packets

$IPT -A FORWARD -p tcp -i $LOCAL_IFACE -j tcp_outbound
$IPT -A FORWARD -p udp -i $LOCAL_IFACE -j udp_outbound
$IPT -A FORWARD -p ALL -i $LOCAL_IFACE -j ACCEPT
$IPT -A FORWARD -i $INET_IFACE -m state --state ESTABLISHED,RELATED -j ACCEPT

$IPT -A FORWARD -j LOG --log-prefix "fp=FORWARD:99 a=DROP "

$IPT -A OUTPUT -m state -p icmp --state INVALID -j DROP
$IPT -A OUTPUT -p ALL -s $LO_IP -j ACCEPT
$IPT -A OUTPUT -p ALL -o $LO_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -s $LOCAL_IP -j ACCEPT
$IPT -A OUTPUT -p ALL -o $LOCAL_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $INET_IFACE -j ACCEPT
$IPT -A OUTPUT -j LOG --log-prefix "fp=OUTPUT:99 a=DROP "

$IPT -t nat -A POSTROUTING -o $INET_IFACE -j MASQUERADE
$IPT -t mangle -A OUTPUT -o $INET_IFACE -j TTL --ttl-set 128
 
Old 04-03-2009, 04:10 PM   #13
win32sux
LQ Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 380Reputation: 380Reputation: 380Reputation: 380
Quote:
Originally Posted by fruitwerks View Post
So how exactly would this work? I could use this for the larger set of rules. But again, I would like them to be dropped of course.
You just load the rules into the CHECK_IP chain, then send all packets in state NEW through it. If the connection is being initiated by a banned IP, it won't be allowed to proceed. If it isn't, the packet will continue downward, through whatever chain sent the packet to CHECK_IP originally. Packets which are part of a connection which has already been initiated won't get sent to CHECK_IP.
 
Old 04-03-2009, 04:16 PM   #14
fruitwerks
Member
 
Registered: Apr 2009
Posts: 77

Original Poster
Rep: Reputation: 15
So I would still need to load the rules? or can I tell it to reference the blocklist without loading the rules?

I wrote this firewall a while ago and I'm not good with scripting / coding, so I'm confused by some things in the script lol.
 
Old 04-03-2009, 04:16 PM   #15
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora
Posts: 3,935
Blog Entries: 5

Rep: Reputation: Disabled
Quote:
Originally Posted by win32sux
I recommend a separate chain with the IPs, where only packets in state NEW get sent. Example:
Code:
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -m state --state NEW -j CHECK_IP
This is what I do. You want your INPUT chain to be clear and easy to read / maintain.

If you are only blacklisting for certain services, and can add extra criteria to the -j CHECK_IP rule, all the better. For example, if you are only blacklisting those addresses to tcp port 22 (ssh), then add the appropriate criteria to the rule so that you're not traversing CHECK_IP for every single packet that enters the INPUT chain.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
importing audio CD jani1982 Linux - Software 2 03-29-2008 09:11 AM
Importing favourites from IE sipickles Linux - Newbie 3 10-29-2007 12:56 PM
Mailbox Importing skulbite Linux - Server 3 01-29-2007 11:12 AM
iptables v1.2.9: Unknown arg `/sbin/iptables' Try `iptables -h' or 'iptables --help' Niceman2005 Linux - Security 4 12-29-2005 09:20 PM
importing??? mojozoox Linux - Software 3 08-25-2003 08:43 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration