LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 02-04-2013, 05:50 PM   #1
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Rep: Reputation: 42
script to update blocked hosts


Ladies & Gents,

Hopefully this is in the right forum.

I have written this script to download the drop.txt file from http://www.spamhaus.org/drop/ and update and reload my firewall. I think I have every thing right but I am not much of a programmer. Would you please take a look at it to see if it will do what I want it to, that is;

1. download 2 text files
2. merge them
3. replace the data file for the firewall
4. reload the firewall

Code:
#! /bin/bash

wget http://www.spamhaus.org/drop/drop.txt
wget http://www.spamhaus.org/drop/edrop.txt

cat edrop.txt >> drop.txt

mv drop.txt /ect/arno-iptables-firewall/blocked-hosts

arno-iptables-firewall force-reload

I plan to set this up a cron job to run once a day.

Thanks.
 
Old 02-04-2013, 07:59 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,743
Blog Entries: 54

Rep: Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972
Moved: This thread is more suitable in the Programming forum and has been moved accordingly to help your thread/question get the exposure it deserves.

Code:
#!/bin/sh --
# set debug and error mode when testing:
set -vx
# Set default behaviour:
LANG=C; LC_ALL=C; export LANG LC_ALL
_TMPFILE=`mktemp -p /tmp drop_upd.XXXXXXXXXX` && { ( wget -q http://www.spamhaus.org/drop/drop.txt \
-O /dev/stdout; wget -q http://www.spamhaus.org/drop/edrop.txt -O /dev/stdout ) \
| grep -e "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\." > "${_TMPFILE}"; [ -s "${_TMPFILE}" ] || \
{ rm -f "${_TMPFILE}"; exit 0; }; [ -f /etc/arno-iptables-firewall/blocked-hosts ] && cp { \
/etc/arno-iptables-firewall/blocked-hosts /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts && \
cat "${_TMPFILE}" > /etc/arno-iptables-firewall/blocked-hosts && arno-iptables-firewall force-reload; }
}; rm -f "${_TMPFILE}"; exit 0
 
Old 02-04-2013, 08:07 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,743
Blog Entries: 54

Rep: Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972
*Note the script only filters comments that ^;, does not 'egrep -v' or 'sort -u' any IPv4 addresses but more importantly the data DROP and EDROP provide doesn't replace the RBLs like the SBL, XBL and PBL. So if you haven't already then implement a milter first.
 
Old 02-05-2013, 07:29 PM   #4
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
Thanks unSpawn

I am having issues understanding just what the script is doing. This line in particular
Code:
; [ -s "${_TMPFILE}" ] || \
My guess is that it is saving the tmpfile but I am not a programmer. If the -s is tied to grep it suppress error/warnings and I can't locate it as an option for wget, so what command it is tied to I don't understand.

It also looks like the script is adding the date to the blocked-host file name and the config for arno's scripts would need to be changed also at the time somehow to recognize the new name every time the script is run.

I do believe that arno's understands that ; is a commenting mark. I know that it understands # as a commenting mark. I don't have ready access to a box with arno's installed to check for sure. But I will.
 
Old 02-05-2013, 08:20 PM   #5
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
OK so I was wrong.

From Arno's FAQ

Q: What's the proper way to use the blocked hosts file?
A: Just put the hostname or IP of the host(s) you want to block in "/etc/iptables-blocked-hosts" (default location). You can use comments (starting with the #-character) but it can only be used when the whole line is a comment!

So I guess I need to get rid of the ;comments
 
Old 02-06-2013, 09:35 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,743
Blog Entries: 54

Rep: Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972
Code:
#!/bin/sh --
# set debug and error mode when testing:
set -vx
# Set default behaviour:
LANG=C; LC_ALL=C; export LANG LC_ALL

_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
if [ -f "${_TMPFILE}" ]; then
 # If the temporary file exists make wget download data quiet and output to stdout
 # and redirect stdout to the temporary file.
 wget -q http://www.spamhaus.org/drop/drop.txt -O /dev/stdout > "${_TMPFILE}" 
 wget -q http://www.spamhaus.org/drop/edrop.txt -O /dev/stdout >> "${_TMPFILE}"
 if [ ! -s "${_TMPFILE}" ]; then
  # If the file is empty then exit.
  rm -f "${_TMPFILE}"; exit 1
 fi
 if [ -f /etc/arno-iptables-firewall/blocked-hosts ]; then
  # If this filee xists then make a backup. 
  cp /etc/arno-iptables-firewall/blocked-hosts /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
  if [ $? -eq 0 ]; then
   # If making a backup succeeded then filter the temporary file for only IPv4-like octets and 
   # fill the aforementioned file with it. *It would have been better to compare both files and only add
   # the differences or use 'sort -u file0 file1'.
   egrep -e "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\." "${_TMPFILE}" > /etc/arno-iptables-firewall/blocked-hosts
   if [ $? -eq 0 ]; then
    # If populating the file ended OK then restart the service
    arno-iptables-firewall force-reload
   fi
  else
   # If making a backup did not succeeded then clean up the mess and exit.
   rm -f "${_TMPFILE}"; exit 1
  fi
else
 # If the temporary file does not exist then exit.
 exit 1
fi
# Reached the end of the script. Remove temp file and exit cleanly.
rm -f "${_TMPFILE}"
exit 0
 
Old 02-08-2013, 04:48 PM   #7
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
Thanks again unSpawn

I was having an issue with the egrep command in the script but a guy from our local lug advised me to change it to

Code:
cat "${_TMPFILE}" | sed -e 's/\;.*$//g' > /etc/arno-iptables-firewall/blocked-hosts
and that works fine. But for some reason the wget edrop.txt is overwriting drop.txt when the download takes place. I have looked at the script but don't understand why it would do that, as the edrop line has >> which as I understand it means to append to the file.

I also had to add a fi to the end of the script. fyi

Other than that the script works fine but blocked-hosts only contains the data from edorp.txt
 
Old 02-09-2013, 07:43 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,743
Blog Entries: 54

Rep: Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972
Quote:
Originally Posted by rbees View Post
for some reason the wget edrop.txt is overwriting drop.txt when the download takes place.
The second one was actually more of an explanation. I don't script that way anymore. Did you try running the script I posted originally? Else post your unabbreviated script.
 
Old 02-09-2013, 07:20 PM   #9
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
Code:
#!/bin/sh --
set -vx
LANG=C; LC_ALL=C; export LANG LC_ALL
_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
if [ -f "${_TMPFILE}" ]; then
 wget -q http://www.spamhaus.org/drop/drop.txt -O /dev/stdout > "${_TMPFILE}" 
 wget -q http://www.spamhaus.org/drop/edrop.txt -O /dev/stdout >> "${_TMPFILE}"
 if [ ! -s "${_TMPFILE}" ]; then
  rm -f "${_TMPFILE}"; exit 1
 fi
 if [ -f /etc/arno-iptables-firewall/blocked-hosts ]; then
  cp /etc/arno-iptables-firewall/blocked-hosts /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
  if [ $? -eq 0 ]; then
   cat "${_TMPFILE}" | sed -e 's/\;.*$//g' > /etc/arno-iptables-firewall/blocked-hosts
   # why not just sed the ; to # and be done as arno's understands the #?
   if [ $? -eq 0 ]; then
    arno-iptables-firewall force-reload
   fi
  else
   rm -f "${_TMPFILE}"; exit 1
  fi
else
 exit 1
fi
rm -f "${_TMPFILE}"
exit 0
fi
 
Old 02-09-2013, 07:51 PM   #10
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
I tried adding
Code:
read -p "press [Enter] to continue..."
between after each wget line to see if the data was actually getting written to the tmpfile but the script does not pause for some reason. I am using sudo.
 
Old 02-09-2013, 08:31 PM   #11
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,743
Blog Entries: 54

Rep: Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972
Except for the misplaced "fi" I can't see from your script why it failed.
Code:
#!/bin/sh --
LANG=C; LC_ALL=C; export LANG LC_ALL; 
URI="http://www.spamhaus.org/drop"; CONF="/etc/arno-iptables-firewall/blocked-hosts"
_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
[ -f "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; exit 1; }
( wget -q ${URI}/drop.txt -O /dev/stdout; wget -q ${URI}/edrop.txt -O /dev/stdout ) > "${_TMPFILE}"
[ -s "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; rm -f "${_TMPFILE}"; exit 1; }
OLDLINES($(wc -l "${CONF}")); NEWLINES($(wc -l "${_TMPFILE}"))
cp -f "${CONF}" /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
sed -e 's/\;.*$//g' "${_TMPFILE}" > "${CONF}" && arno-iptables-firewall force-reload
echo "Blocked-hosts had ${OLDLINES[0]} lines, now ${NEWLINES[0]}. Finished, exiting."
rm -f "${_TMPFILE}"; exit 0
 
Old 02-09-2013, 09:05 PM   #12
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
looks like the script is trying to use a variable before it is declared

Code:
sudo /home/user/3drop.list
/home/user/3drop.list: 8: /home/user/3drop.list: Syntax error: word unexpected (expecting ")")
shouldn't the line be

Code:
OLDLINES=$($(wc -l "${CONF}")); NEWLINES=$($(wc -l "${_TMPFILE}"))
instead of

Code:
OLDLINES($(wc -l "${CONF}")); NEWLINES($(wc -l "${_TMPFILE}"))
 
Old 02-09-2013, 09:13 PM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,743
Blog Entries: 54

Rep: Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972Reputation: 2972
Sorry, it's an array, should be
Code:
OLDLINES=($(wc -l "${CONF}")); NEWLINES=($(wc -l "${_TMPFILE}"))
 
Old 02-09-2013, 09:26 PM   #14
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
Thanks,

It seams that my bash does not like the extra ()'s in the script.

Code:
$ sudo /home/user/3drop.list
[sudo] password for user: 
/home/user/3drop.list: 8: /home/kingbee/sys-config/external/3drop.list: Syntax error: "(" unexpected
So I removed them and now

Code:
:~$ sudo /home/user/3drop.list
Arno's Iptables Firewall Script v2.0.1c
-------------------------------------------------------------------------------

Feb 09 21:19:37 All firewall rules applied.
/home/user/3drop.list: 11: /home/user/3drop.list: Bad substitution
 
Old 02-09-2013, 09:27 PM   #15
rbees
Member
 
Registered: Mar 2004
Location: northern michigan usa
Distribution: Debian Squeeze, Whezzy, Jessie
Posts: 681

Original Poster
Rep: Reputation: 42
the current script
Code:
#!/bin/sh --
LANG=C; LC_ALL=C; export LANG LC_ALL; 
URI="http://www.spamhaus.org/drop"; CONF="/etc/arno-iptables-firewall/blocked-hosts"
_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
[ -f "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; exit 1; }
( wget -q ${URI}/drop.txt -O /dev/stdout; wget -q ${URI}/edrop.txt -O /dev/stdout ) > "${_TMPFILE}"
[ -s "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; rm -f "${_TMPFILE}"; exit 1; }
OLDLINES=$(wc -l "${CONF}"); NEWLINES=$(wc -l "${_TMPFILE}")
cp -f "${CONF}" /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
sed -e 's/\;.*$//g' "${_TMPFILE}" > "${CONF}" && arno-iptables-firewall force-reload
echo "Blocked-hosts had ${OLDLINES[0]} lines, now ${NEWLINES[0]}. Finished, exiting."
rm -f "${_TMPFILE}"; exit 0
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Host is blocked because of many connection error;unblock with mysqladmin-flush-hosts arun5002 Linux - Server 1 11-28-2011 10:11 AM
blocked update lucid lynx tiomoco Linux - Newbie 2 03-08-2011 01:54 PM
Script for hosts, numbers of hosts and users connected to squid server arunabh_biswas Programming 5 08-28-2010 05:11 AM
script to update /etc/hosts on dhcp obtained ip change. juanctes Linux - General 1 02-02-2008 10:17 AM
script to update /etc/hosts bradut Linux - Newbie 8 07-16-2002 11:52 AM


All times are GMT -5. The time now is 04:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration