LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 07-30-2017, 03:55 AM   #1
frater
Member
 
Registered: Jul 2008
Posts: 121

Rep: Reputation: 23
Convert CIDR to a reverse IP to be used in a DNSBL


A DNSBL or DNS blacklist is a DNS server that responds to a query of a reverse IP with an answer or not.
If it responds it means the IP is in the blacklist or the reverse.

An IP 10.20.30.40 would be queried like this "40.30.20.10.blacklist.org"

Not each IP needs to be individually listed in that DNS-server.
If all addresses in the "10.20.30.0/24" should be blacklisted, the DNS-server only needs to have 1 DNS-entry and that's "*.30.20.10.blacklist.org"

I have created a bash-script that will fill my DNS-server with all these DNS-entries by taking a file containing all the IP-addresses.

The script works, but it is not able to handle CIDR's that are NOT /8, /16, /24 or /32.
This because the wildcard-DNS system is based on a decimal system and not binary.

If I could convert the list before processing so that it would only contain /8, /16, /24 and /32 blocks it would work for all addresses.

10.20.30.0/23 would become 10.20.30.0/24 and 10.20.31.0/24
So it's somehow the reverse of the utility I often use: "aggregate"

I created this script after reading this question:
https://talk.plesk.com/threads/email-blacklists.344453/

Here's the script as it is now:

Can someone help me to improve the script so it can also handle the remaining IP-blocks

Code:
cat /usr/local/sbin/dnsbl
#!/bin/bash

BLOCKLIST=/opt/ASSP/files/blockip.txt
DNSBL=dnsbl.blocklist.org
PLESKDNS=/usr/local/psa/bin/dns
LOCALHOST=127.0.0.2

THISSCRIPT="`readlink -f $0`"
SCRIPTNAME=${THISSCRIPT##*/}
BASE=${THISSCRIPT##*/}
[ -z "${BASE}" ] && BASE=${0##*/}
LOG=/var/log/${BASE}.log

if [ ! -e "${BLOCKLIST}" ] ; then
  echo "The blocklist ${BLOCKLIST} does not exist, quitting"
  exit 1
fi

TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
TMPLOG=${TMPDIR}/log
date >${TMPLOG}

# Filter IPv4 addresses, 1 per row, only spaces or tabs allowed in front
egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t'| sort -uo ${TMPDIR}/all_blocks
echo "Found `grep -c '.' ${TMPDIR}/all_blocks` IP-blocks " | tee -a ${TMPLOG}

# Filter the plain IP's and CIDR with /32,/24,/16 and /8
egrep -o '^[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/32|[0-9]+/24|[0-9]+/16|[0-9]+/8)$' ${TMPDIR}/all_blocks >${TMPDIR}/blocks

echo "I was able to use `grep -c '.' ${TMPDIR}/blocks` IP-blocks " | tee -a ${TMPLOG}
echo -e "These CIDR's are ignored:\n`grep -vf ${TMPDIR}/blocks ${TMPDIR}/all_blocks`\n\n"  | tee -a ${TMPLOG}

while read BLOCK ; do
  # remove the CIDR, Split the IPv4 in 4 segments keeping only the digits and reverse the order (tac)
  echo "${BLOCK}" | sed 's/\/.*//g' | egrep -o '[0-9]+($|\.)' | tr -d '.' | tac >${TMPDIR}/ip4

  ASTERISKS=0                                         # /32 or plain
  echo "${BLOCK}" | grep -q "/24" && ASTERISKS=1      # /24
  echo "${BLOCK}" | grep -q "/16" && ASTERISKS=2      # /16
  echo "${BLOCK}" | grep -q "/8"  && ASTERISKS=3      # /8

  N=1
  REV_IP=""
  while read IP4 ; do
    if [ $N -ge ${ASTERISKS} ] ; then
      if [ $N -eq ${ASTERISKS} ] ; then
        REV_IP="${REV_IP}*."            # Replace whatever is there with an asterisk
      else
        REV_IP="${REV_IP}${IP4}."       # Write the IPv4 segment
      fi
    fi
    let N+=1
  done<${TMPDIR}/ip4
  # Write the reversed IP to the file after stripping the superfluous last dot
  echo "${REV_IP}" | sed 's/\.$//g' >>${TMPDIR}/rev_ip4

done<${TMPDIR}/blocks

if [ -s ${TMPDIR}/rev_ip4 ] ; then
  # Add the IPBLOCKS to DNS
  N=1
  while read IPBLOCK ; do
    echo "${N}. Add ${IPBLOCK}.${DNSBL} IN A ${LOCALHOST}" | tee -a ${TMPLOG}
    ${PLESKDNS} --add ${DNSBL} -a "${IPBLOCK}" -ip ${LOCALHOST} | tee -a ${TMPLOG}
    let N+=1
  done<${TMPDIR}/rev_ip4

  cat ${TMPLOG} >>${LOG}        # Write temporary log to log
fi

rm -r ${TMPDIR}

Last edited by frater; 07-30-2017 at 05:10 AM.
 
Old 07-30-2017, 12:40 PM   #2
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
You mean that wildcard DNS has two choices, either an asterisk or an octet, no masking. Unfortunately, this seems to be an obscure scenario with no solution readily available. There is a PERL library for converting IP addresses. That might be a place to start.

I would separate all the IPs that require conversion into a file, and then work on that file. After the conversion is done, append to the master file. I'm sorry I could not be of more assistance. I just can't remember how I did this before.

I know I used a PERL script. But it's been years now.
 
Old 07-30-2017, 12:45 PM   #3
frater
Member
 
Registered: Jul 2008
Posts: 121

Original Poster
Rep: Reputation: 23
I thought it over some more and was able to do it myself.
Here's the adapted code.
I'm sure it can be improved, but the base code is there now and exactly doing what I want it to do.
Just finished writing it.

cat /usr/local/sbin/dnsbl
Code:
#!/bin/bash

BLOCKLIST=/opt/ASSP/files/blockip.txt
DNSBL=dnsbl.blacklist.org
PLESKDNS=/usr/local/psa/bin/dns
LOCALHOST=127.0.0.2

THISSCRIPT="`readlink -f $0`"
SCRIPTNAME=${THISSCRIPT##*/}
BASE=${THISSCRIPT##*/}
[ -z "${BASE}" ] && BASE=${0##*/}
LOG=/var/log/${BASE}.log

if [ ! -e "${BLOCKLIST}" ] ; then
  echo "The blocklist ${BLOCKLIST} does not exist, quitting"
  exit 1
fi

TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
TMPLOG=${TMPDIR}/log
date >${TMPLOG}

# Filter IPv4 addresses, 1 per row, only spaces or tabs allowed in front
egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t'| sort -uo ${TMPDIR}/all_blocks
echo "Found `grep -c '.' ${TMPDIR}/all_blocks` IP-blocks " | tee -a ${TMPLOG}

# Filter the plain IP's and CIDR with /32,/24,/16 and /8
egrep -o '^[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/32|[0-9]+/24|[0-9]+/16|[0-9]+/8)$' ${TMPDIR}/all_blocks >${TMPDIR}/blocks

grep -vf ${TMPDIR}/blocks ${TMPDIR}/all_blocks >${TMPDIR}/rest_blocks

if ! which ipcalc >/dev/null ; then
  echo "I was able to use `grep -c '.' ${TMPDIR}/blocks` IP-blocks " | tee -a ${TMPLOG}
  echo -e "These CIDR's are ignored:\n`grep -vf ${TMPDIR}/blocks ${TMPDIR}/all_blocks`\n\n"  | tee -a ${TMPLOG}
else
  echo "I have `grep -c . ${TMPDIR}/rest_blocks` remaining blocks that I need to break down"
  while read BLOCK ; do
    NETWORK=`ipcalc -n "${BLOCK}" | awk -F= '{print $2}'`
    BROADCAST=`ipcalc -b "${BLOCK}" | awk -F= '{print $2}'`
    SUFFIX=`echo "${BLOCK}" | sed 's/.*\///g'`

    BLOCKS=`ipcalc -m "${NETWORK}/${SUFFIX}" | egrep -o '[0-9]+' | grep -c '255'`
    echo "${NETWORK}" | egrep -o '[0-9]+' >${TMPDIR}/segments

    N=1
    BASE=""
    echo -n '' >${TMPDIR}/brokendownblocks
    while read SEGMENT ; do
      [ ${BLOCKS} -lt ${N} ] && break
      BASE="${BASE}${SEGMENT}."
      let N+=1
    done<${TMPDIR}/segments

    START="`echo ${NETWORK} | sed "s/${BASE}//g" | sed 's/\..*//g'`"
    END="`echo ${BROADCAST} | sed "s/${BASE}//g" | sed 's/\..*//g'`"

    NEW_SUFFIX="/32"
    if [ ${SUFFIX} -lt 16 ] ; then
      if [ ${SUFFIX} -lt 8 ] ; then
        NEW_SUFFIX=".0.0.0/8"
      else
        NEW_SUFFIX=".0.0/16"
      fi
    elif [ ${SUFFIX} -lt 24 ] ; then
      NEW_SUFFIX=".0/24"
    fi

    N=${START}
    while [ ${N} -le ${END} ]  ; do
      echo "${BASE}${N}${NEW_SUFFIX}" >>${TMPDIR}/brokendownblocks
      let N+=1
    done
    echo "I broke down ${NETWORK}/${SUFFIX} into `grep -c . ${TMPDIR}/brokendownblocks` ${NEW_SUFFIX} blocks"
    cat ${TMPDIR}/brokendownblocks >>${TMPDIR}/blocks
  done<${TMPDIR}/rest_blocks
  echo "I now have in total `grep -c . ${TMPDIR}/blocks` blocks"
fi

I=1
while read BLOCK ; do
  echo -en "${I}     \r"

  # remove the CIDR, Split the IPv4 in 4 segments keeping only the digits and reverse the order (tac)
  echo "${BLOCK}" | sed 's/\/.*//g' | egrep -o '[0-9]+($|\.)' | tr -d '.' | tac >${TMPDIR}/ip4

  ASTERISKS=0                                         # /32 or plain
  echo "${BLOCK}" | grep -q "/24" && ASTERISKS=1      # /24
  echo "${BLOCK}" | grep -q "/16" && ASTERISKS=2      # /16
  echo "${BLOCK}" | grep -q "/8"  && ASTERISKS=3      # /8

  N=1
  REV_IP=""
  while read IP4 ; do
    if [ $N -ge ${ASTERISKS} ] ; then
      if [ $N -eq ${ASTERISKS} ] ; then
        REV_IP="${REV_IP}*."            # Replace whatever is there with an asterisk
      else
        REV_IP="${REV_IP}${IP4}."       # Write the IPv4 segment
      fi
    fi
    let N+=1
  done<${TMPDIR}/ip4
  # Write the reversed IP to the file after stripping the superfluous last dot
  echo "${REV_IP}" | sed 's/\.$//g' >>${TMPDIR}/rev_ip4
  let I+=1
done<${TMPDIR}/blocks

if [ -s ${TMPDIR}/rev_ip4 ] ; then
  # Add the IPBLOCKS to DNS
  N=1
  while read IPBLOCK ; do
    echo "${N}. Add ${IPBLOCK}.${DNSBL} IN A ${LOCALHOST}" | tee -a ${TMPLOG}
    ${PLESKDNS} --add ${DNSBL} -a "${IPBLOCK}" -ip ${LOCALHOST} | tee -a ${TMPLOG}
    let N+=1
  done<${TMPDIR}/rev_ip4

  cat ${TMPLOG} >>${LOG}        # Write temporary log to log
fi

rm -r ${TMPDIR}
 
Old 07-30-2017, 01:20 PM   #4
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
You're a good programmer. I was very surprised to see ipcalc, because in trying to answer your question I found it was a possibility. But I didn't think it was elegant enough.
 
Old 07-30-2017, 01:27 PM   #5
frater
Member
 
Registered: Jul 2008
Posts: 121

Original Poster
Rep: Reputation: 23
Thanks, but I know for a fact there are many better programmers here.
Would still like to see some improvements if anyone's sees one.
 
Old 07-30-2017, 01:28 PM   #6
frater
Member
 
Registered: Jul 2008
Posts: 121

Original Poster
Rep: Reputation: 23
I decided to create a script which would do the reverse of what I just did.
It gives a list of subnets that it gathered from the DNS-file of your BIND folder.

cat /usr/local/sbin/dns2bl
Code:
#!/bin/bash

BLOCKLIST=/opt/ASSP/files/blockip.txt
DNSBL=dnsbl.blacklist.org
PLESKDNS=/usr/local/psa/bin/dns
LOCALHOST=127.0.0.2
BINDDIR=/var/named/run-root/var

DNSFILE=${BINDDIR}/${DNSBL}

THISSCRIPT="`readlink -f $0`"
SCRIPTNAME=${THISSCRIPT##*/}
BASE=${THISSCRIPT##*/}
[ -z "${BASE}" ] && BASE=${0##*/}
LOG=/var/log/${BASE}.log

if [ ! -e "${BLOCKLIST}" ] ; then
  echo "The blocklist ${BLOCKLIST} does not exist, quitting"
  exit 1
fi

TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
TMPLOG=${TMPDIR}/log
date >${TMPLOG}

egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t' | sed -r 's/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/&\/32/g' | aggregate -t 2>/dev/null > ${TMPDIR}/blocklist


# egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t' | aggregate -t 2>/dev/null > ${TMPDIR}/blocklist
egrep "^(\*|[0-9]+)\.[0-9.]+${DNSBL}.*IN.*A.*127" ${DNSFILE} | sed "s/\.${DNSBL}.*//g;s/^\*\.//g" | grep -v "\.0\.0\.127" >${TMPDIR}/allips

if [ ! -s ${TMPDIR}/allips ] ; then
  echo "No DNS-entries found in Zone ${DNSBL}"
else
  touch ${TMPDIR}/reversed_raw
  while read IP ; do
    # Create a reversed list of the IP
    echo "${IP}" | egrep -o '[0-9]+' | tac >${TMPDIR}/segments

    N=1
    REVERSED_IP=""
    while read SEGMENT ; do
      if [ $N -eq 1 ] ; then
        REVERSED_IP="${SEGMENT}"
      else
        REVERSED_IP="${REVERSED_IP}.${SEGMENT}"
      fi
      let N+=1
    done<${TMPDIR}/segments

    if   [ $N -eq 5 ] ; then
      REVERSED_IP="${REVERSED_IP}/32"
    elif [ $N -eq 4 ] ; then
      REVERSED_IP="${REVERSED_IP}.0/24"
    elif [ $N -eq 3 ] ; then
       REVERSED_IP="${REVERSED_IP}.0.0/16"
    else
      REVERSED_IP="${REVERSED_IP}.0.0.0/8"
    fi
    echo "${REVERSED_IP}" >>${TMPDIR}/reversed_raw
  done<${TMPDIR}/allips
  cat ${TMPDIR}/reversed_raw  | aggregate -t 2>/dev/null >${TMPDIR}/reversed

  echo -e "Subnets gathered from DNS:\n"  >&2
  cat ${TMPDIR}/reversed
  echo -e "\n\nSubnets in ${BLOCKLIST}:\n`cat ${TMPDIR}/blocklist`\n\n" >&2
  echo -e "Difference:\n`diff ${TMPDIR}/blocklist ${TMPDIR}/reversed`\n\n" >&2
fi

rm -r ${TMPDIR}

Last edited by frater; 07-31-2017 at 04:53 AM.
 
Old 08-02-2017, 02:34 AM   #7
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,524

Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
Quote:
Originally Posted by frater View Post
I know for a fact there are many better programmers here.
There is always someone better.

There are two kinds of people: those who pursue excellence and everyone else.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
if I convert an OO document to PDF, can I reverse it? newbiesforever Linux - Software 4 10-01-2010 10:46 PM
Sendmail - FEATURE(`dnsbl') - Creating exceptions while using FEATURE(`dnsbl') amonamarth Linux - Server 0 01-25-2010 01:21 PM
[SOLVED] CIDR explanantion? qwertyjjj Linux - Newbie 2 08-18-2009 07:02 AM
C cidr argv quantt Programming 11 01-26-2009 07:03 AM
convert a logical expression to reverse Polish notation in Java manolakis Programming 9 08-25-2008 07:45 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 08:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration