script to update blocked hosts

rbees · 02-04-2013, 04:50 PM

Ladies & Gents,

Hopefully this is in the right forum.

I have written this script to download the drop.txt file from http://www.spamhaus.org/drop/ and update and reload my firewall. I think I have every thing right but I am not much of a programmer. Would you please take a look at it to see if it will do what I want it to, that is;

1. download 2 text files
2. merge them
3. replace the data file for the firewall
4. reload the firewall

Code:

#! /bin/bash

wget http://www.spamhaus.org/drop/drop.txt
wget http://www.spamhaus.org/drop/edrop.txt

cat edrop.txt >> drop.txt

mv drop.txt /ect/arno-iptables-firewall/blocked-hosts

arno-iptables-firewall force-reload

I plan to set this up a cron job to run once a day.

Thanks.

unSpawn · 02-04-2013, 06:59 PM

Moved: This thread is more suitable in the Programming forum and has been moved accordingly to help your thread/question get the exposure it deserves.

Code:

#!/bin/sh --
# set debug and error mode when testing:
set -vx
# Set default behaviour:
LANG=C; LC_ALL=C; export LANG LC_ALL
_TMPFILE=`mktemp -p /tmp drop_upd.XXXXXXXXXX` && { ( wget -q http://www.spamhaus.org/drop/drop.txt \
-O /dev/stdout; wget -q http://www.spamhaus.org/drop/edrop.txt -O /dev/stdout ) \
| grep -e "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\." > "${_TMPFILE}"; [ -s "${_TMPFILE}" ] || \
{ rm -f "${_TMPFILE}"; exit 0; }; [ -f /etc/arno-iptables-firewall/blocked-hosts ] && cp { \
/etc/arno-iptables-firewall/blocked-hosts /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts && \
cat "${_TMPFILE}" > /etc/arno-iptables-firewall/blocked-hosts && arno-iptables-firewall force-reload; }
}; rm -f "${_TMPFILE}"; exit 0

unSpawn · 02-04-2013, 07:07 PM

*Note the script only filters comments that ^;, does not 'egrep -v' or 'sort -u' any IPv4 addresses but more importantly the data DROP and EDROP provide doesn't replace the RBLs like the SBL, XBL and PBL. So if you haven't already then implement a milter first.

rbees · 02-05-2013, 06:29 PM

Thanks unSpawn

I am having issues understanding just what the script is doing. This line in particular

Code:

; [ -s "${_TMPFILE}" ] || \

My guess is that it is saving the tmpfile but I am not a programmer. If the -s is tied to grep it suppress error/warnings and I can't locate it as an option for wget, so what command it is tied to I don't understand.

It also looks like the script is adding the date to the blocked-host file name and the config for arno's scripts would need to be changed also at the time somehow to recognize the new name every time the script is run.

I do believe that arno's understands that ; is a commenting mark. I know that it understands # as a commenting mark. I don't have ready access to a box with arno's installed to check for sure. But I will.

rbees · 02-05-2013, 07:20 PM

OK so I was wrong.

From Arno's FAQ

Q: What's the proper way to use the blocked hosts file?
A: Just put the hostname or IP of the host(s) you want to block in "/etc/iptables-blocked-hosts" (default location). You can use comments (starting with the #-character) but it can only be used when the whole line is a comment!

So I guess I need to get rid of the ;comments

unSpawn · 02-06-2013, 08:35 AM

Code:

#!/bin/sh --
# set debug and error mode when testing:
set -vx
# Set default behaviour:
LANG=C; LC_ALL=C; export LANG LC_ALL

_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
if [ -f "${_TMPFILE}" ]; then
 # If the temporary file exists make wget download data quiet and output to stdout
 # and redirect stdout to the temporary file.
 wget -q http://www.spamhaus.org/drop/drop.txt -O /dev/stdout > "${_TMPFILE}" 
 wget -q http://www.spamhaus.org/drop/edrop.txt -O /dev/stdout >> "${_TMPFILE}"
 if [ ! -s "${_TMPFILE}" ]; then
  # If the file is empty then exit.
  rm -f "${_TMPFILE}"; exit 1
 fi
 if [ -f /etc/arno-iptables-firewall/blocked-hosts ]; then
  # If this filee xists then make a backup. 
  cp /etc/arno-iptables-firewall/blocked-hosts /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
  if [ $? -eq 0 ]; then
   # If making a backup succeeded then filter the temporary file for only IPv4-like octets and 
   # fill the aforementioned file with it. *It would have been better to compare both files and only add
   # the differences or use 'sort -u file0 file1'.
   egrep -e "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\." "${_TMPFILE}" > /etc/arno-iptables-firewall/blocked-hosts
   if [ $? -eq 0 ]; then
    # If populating the file ended OK then restart the service
    arno-iptables-firewall force-reload
   fi
  else
   # If making a backup did not succeeded then clean up the mess and exit.
   rm -f "${_TMPFILE}"; exit 1
  fi
else
 # If the temporary file does not exist then exit.
 exit 1
fi
# Reached the end of the script. Remove temp file and exit cleanly.
rm -f "${_TMPFILE}"
exit 0

rbees · 02-08-2013, 03:48 PM

Thanks again unSpawn

I was having an issue with the egrep command in the script but a guy from our local lug advised me to change it to

Code:

cat "${_TMPFILE}" | sed -e 's/\;.*$//g' > /etc/arno-iptables-firewall/blocked-hosts

and that works fine. But for some reason the wget edrop.txt is overwriting drop.txt when the download takes place. I have looked at the script but don't understand why it would do that, as the edrop line has >> which as I understand it means to append to the file.

I also had to add a fi to the end of the script. fyi

Other than that the script works fine but blocked-hosts only contains the data from edorp.txt

unSpawn · 02-09-2013, 06:43 AM

Quote:

Originally Posted by rbees

for some reason the wget edrop.txt is overwriting drop.txt when the download takes place.

The second one was actually more of an explanation. I don't script that way anymore. Did you try running the script I posted originally? Else post your unabbreviated script.

rbees · 02-09-2013, 06:20 PM

Code:

#!/bin/sh --
set -vx
LANG=C; LC_ALL=C; export LANG LC_ALL
_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
if [ -f "${_TMPFILE}" ]; then
 wget -q http://www.spamhaus.org/drop/drop.txt -O /dev/stdout > "${_TMPFILE}" 
 wget -q http://www.spamhaus.org/drop/edrop.txt -O /dev/stdout >> "${_TMPFILE}"
 if [ ! -s "${_TMPFILE}" ]; then
  rm -f "${_TMPFILE}"; exit 1
 fi
 if [ -f /etc/arno-iptables-firewall/blocked-hosts ]; then
  cp /etc/arno-iptables-firewall/blocked-hosts /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
  if [ $? -eq 0 ]; then
   cat "${_TMPFILE}" | sed -e 's/\;.*$//g' > /etc/arno-iptables-firewall/blocked-hosts
   # why not just sed the ; to # and be done as arno's understands the #?
   if [ $? -eq 0 ]; then
    arno-iptables-firewall force-reload
   fi
  else
   rm -f "${_TMPFILE}"; exit 1
  fi
else
 exit 1
fi
rm -f "${_TMPFILE}"
exit 0
fi

rbees · 02-09-2013, 06:51 PM

I tried adding

Code:

read -p "press [Enter] to continue..."

between after each wget line to see if the data was actually getting written to the tmpfile but the script does not pause for some reason. I am using sudo.

unSpawn · 02-09-2013, 07:31 PM

Except for the misplaced "fi" I can't see from your script why it failed.

Code:

#!/bin/sh --
LANG=C; LC_ALL=C; export LANG LC_ALL; 
URI="http://www.spamhaus.org/drop"; CONF="/etc/arno-iptables-firewall/blocked-hosts"
_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
[ -f "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; exit 1; }
( wget -q ${URI}/drop.txt -O /dev/stdout; wget -q ${URI}/edrop.txt -O /dev/stdout ) > "${_TMPFILE}"
[ -s "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; rm -f "${_TMPFILE}"; exit 1; }
OLDLINES($(wc -l "${CONF}")); NEWLINES($(wc -l "${_TMPFILE}"))
cp -f "${CONF}" /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
sed -e 's/\;.*$//g' "${_TMPFILE}" > "${CONF}" && arno-iptables-firewall force-reload
echo "Blocked-hosts had ${OLDLINES[0]} lines, now ${NEWLINES[0]}. Finished, exiting."
rm -f "${_TMPFILE}"; exit 0

rbees · 02-09-2013, 08:05 PM

looks like the script is trying to use a variable before it is declared

Code:

sudo /home/user/3drop.list
/home/user/3drop.list: 8: /home/user/3drop.list: Syntax error: word unexpected (expecting ")")

shouldn't the line be

Code:

OLDLINES=$($(wc -l "${CONF}")); NEWLINES=$($(wc -l "${_TMPFILE}"))

instead of

Code:

OLDLINES($(wc -l "${CONF}")); NEWLINES($(wc -l "${_TMPFILE}"))

unSpawn · 02-09-2013, 08:13 PM

Sorry, it's an array, should be

Code:

OLDLINES=($(wc -l "${CONF}")); NEWLINES=($(wc -l "${_TMPFILE}"))

rbees · 02-09-2013, 08:26 PM

Thanks,

It seams that my bash does not like the extra ()'s in the script.

Code:

$ sudo /home/user/3drop.list
[sudo] password for user: 
/home/user/3drop.list: 8: /home/kingbee/sys-config/external/3drop.list: Syntax error: "(" unexpected

So I removed them and now

Code:

:~$ sudo /home/user/3drop.list
Arno's Iptables Firewall Script v2.0.1c
-------------------------------------------------------------------------------

Feb 09 21:19:37 All firewall rules applied.
/home/user/3drop.list: 11: /home/user/3drop.list: Bad substitution

rbees · 02-09-2013, 08:27 PM

the current script

Code:

#!/bin/sh --
LANG=C; LC_ALL=C; export LANG LC_ALL; 
URI="http://www.spamhaus.org/drop"; CONF="/etc/arno-iptables-firewall/blocked-hosts"
_TMPFILE=$(mktemp -p /tmp drop_upd.XXXXXXXXXX)
[ -f "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; exit 1; }
( wget -q ${URI}/drop.txt -O /dev/stdout; wget -q ${URI}/edrop.txt -O /dev/stdout ) > "${_TMPFILE}"
[ -s "${_TMPFILE}" ] || { echo "Error, exiting." >/dev/stderr; rm -f "${_TMPFILE}"; exit 1; }
OLDLINES=$(wc -l "${CONF}"); NEWLINES=$(wc -l "${_TMPFILE}")
cp -f "${CONF}" /var/cache/$(/bin/date +'%Y%m%d')_blocked-hosts
sed -e 's/\;.*$//g' "${_TMPFILE}" > "${CONF}" && arno-iptables-firewall force-reload
echo "Blocked-hosts had ${OLDLINES[0]} lines, now ${NEWLINES[0]}. Finished, exiting."
rm -f "${_TMPFILE}"; exit 0