[SOLVED] Parsing text files in BASH!

UrbanPykey · 05-29-2011, 05:58 PM

Please help!

I'm trying to write a script to list all open ports in the MINIUNPND chain in iptables and use the procotol, port and destination ip to open ports on another router using upnpc.
Here is the output of iptables -L MINIUPNPD

Code:

>iptables -L MINIUPNPD
Chain MINIUPNPD (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             192.168.3.124       tcp dpt:19955 
ACCEPT     tcp  --  anywhere             192.168.3.124       tcp dpt:20054
ACCEPT     tcp  --  anywhere             192.168.3.130       udp dpt:10654
ACCEPT     tcp  --  anywhere             192.168.3.121       tcp dpt:29955
ACCEPT     tcp  --  anywhere             192.168.3.165       udp dpt:55555

I have managed to pipe the above command into a text file with this script

Code:

#!/bin/bash

iptables -L MINIUPNPD > tmp.txt

# REMOVE TOP 2 LINES OF OUTPUT
sed -e '1,2d' tmp.txt > tmpa.txt

#OUTPUT ONLY FIELDS NEEDED
awk '{print $5 " " $6 " " $7}' tmpa.txt >tmp.txt

Which gives the output

Code:

>cat tmp.txt
192.168.3.124 tcp dpt:19955
192.168.3.124 tcp dpt:20054
192.168.3.130 udp dpt:10654
192.168.3.121 tcp dpt:29955
192.168.3.165 udp dpt:55555

Now this is where i get stuck! I need to read each line and set the fields as variables so i can use them thus

Code:

upnpc -r -a $destiation $portnumber $protocol

and repeat thius for every line in the file!

i have tryied using various scripts based around this

Code:

index=0
ips="tmp.txt"
while read line ; do
        MYDEST[$index]=$(awk -v t="$line" '{print $5}')
        MYPORT[$index]=$(awk -v t="$line" '{print $7}' | cut -d : -f 2 | cut -d$
        MYPROT[$index]=$(awk -v t="$line" '{print $6}')

        index=$(($index+1))
done < $ips

This didn't work so i had to run the loop 3 times, once for each array i wanted to fill

Code:

index=0
ips="tmp.txt"
while read line ; do
        MYDEST[$index]=$(awk -v t="$line" '{print $5}')
        index=$(($index+1))
done < $ips

index=0
ips="tmp.txt"
while read line ; do
        s=$(awk -v t="$line" '{print $7}')
        MYPORT[$index]=$s
        index=$(($index+1))
done < $ips

index=0
ips="tmp.txt"
while read line ; do
        MYPROT[$index]=$(awk -v t="$line" '{print $6}')
        index=$(($index+1))
done < $ips

echo "Destinations are"
echo "${MYDEST[*]}"
echo "Protols are"
echo "${MYPROT[*]}"
echo "Ports are"
echo "${MYPORT[*]}"

No matter what i do i cant seem to remove the first 4 characters from the MYPROT array to leave only the digits. Also i cant seem to read the array back???

I thought it would simply be a loop reading each line and passing the fields in variables, executing upnpc commands i need then moving to the next line of the file until it reached the EOF

Someone please help as i have spent 2 days trying to get this to work.

Thanks in advance guys

David the H. · 05-29-2011, 06:24 PM

Code:

MYDEST[$index]=$(awk -v t="$line" '{print $5}')

You set the awk variable "t", but then never use it in the command. I think you want to do something like echo "$line" | awk ... instead.

I don't have time to test it out now, but I'd probably get awk to print all three fields at once, and use that to set an intermediate array. Then use that array to set your final array variables. Something like this:

Code:

while read line; do

	temparray=(  $( echo $line | awk '{ print $5,$6,$7 } ) )
        MYDEST[$index]=${temparray[0]}
        MYPORT[$index]=${temparray[1]}
        MYPROT[$index]=${temparray[2]}

done < $ips

theNbomr · 05-29-2011, 06:27 PM

Assuming you want to keep all of the work up to the point of extracting the data to a file:

Code:

while read a b c; do echo upnpc -r -a $a $b $c; done < <( cat tmp.txt )

There are some optimizations you can use to get you to this point. When you're satisfied with the output, remove the echo command.

--- rod.

EDIT: missed some of the requirements, see below

theNbomr · 05-29-2011, 06:35 PM

Okay, couldn't let it go:

Code:

while read a b c; do
    c=$(echo $c | cut -d: -f2) 
    echo upnpc -r -a $a $c $b
done < <(awk 'NF == 7 { print $5, $6, $7 }' /tmp/LQUrbanPykey.iptables)

The data file /tmp/LQUrbanPykey.iptables is the raw output from your original iptables command.

Output sample:

Code:

upnpc -r -a 192.168.3.124 19955 tcp
upnpc -r -a 192.168.3.124 20054 tcp
upnpc -r -a 192.168.3.130 10654 udp
upnpc -r -a 192.168.3.121 29955 tcp
upnpc -r -a 192.168.3.165 55555 udp

--- rod.

grail · 05-29-2011, 07:08 PM

How about:

Code:

iptables -L MINIUPNPD | awk 'NR > 2{sub(/.*:/,"",$7),cmd = "upnpc -r -a "$5" "$7" "$6;print | cmd}'

UrbanPykey · 05-29-2011, 07:19 PM

Wow Thank you theNbomr (rod) You are a star!
That script works lovely! Any chance you can please explain HOW it works? I understand most if it except the syntax of "(awk 'NF == 7 { print $5, $6, $7 }' /tmp/LQUrbanPykey.iptables)".

Its the syntax of operators i get confused with in bash!

Now all i have to do is get this script run by cron at regular intervals or whenever the MINIPNPD chain of iptables changes!

Oh and grail your script cause me to get a syntax error
[CODE]
iptables -L | awk 'NR > 2{sub(/.*:/,"",$7),cmd = "upnpc -r -a "$5" "$7" "$6;print | cmd}'
awk: NR > 2{sub(/.*:/,"",$7),cmd = "upnpc -r -a "$5" "$7" "$6;print | cmd}
awk: ^ syntax error
[CODE]

UrbanPykey · 05-29-2011, 07:44 PM

I think i understand now! NF==7 states it only looks at lines with EXACTLY 7 fields yes?
Just the syntax of the
[code]c=$(echo $c | cut -d: -f2)[code]

I have to research! Oh and regular checking the MINIUPNP chain of iptables

Once again THANKS!

theNbomr · 05-29-2011, 08:01 PM

Quote:

Originally Posted by UrbanPykey

I think i understand now! NF==7 states it only looks at lines with EXACTLY 7 fields yes?
Just the syntax of the

Code:

c=$(echo $c | cut -d: -f2)

That part chops off the leading 'dpt:' part of the last field. It uses cut to break the string into fields delimited by ':' ( -d: ), and takes the second field ( -f2 ) from the result.

grails effort trumps mine, once you get it working; doesn't use a temp file.

The awk part does only look at records with exactly 7 fields.

--- rod.

PS [CODE]how to use code tags[/CODE]

UrbanPykey · 05-29-2011, 08:49 PM

It would take me a degree in BASH to work out HOW Grails script works! Your script is more at my level, ONCE i get into my head how the syntax of bash works a bit more. It all seemed so easy laid out on paper in pseudo code!
Anyways here is my completed script, i have added some extra variables to avoid upnpc failing to discover the remote router and to set the destination to the local IP of the local ClearOS gateway

Code:

#!/bin/bash
#
#  ****************************************************
#  * Remote router upnp Port opener          ver 1.01 *
#  *                                                  *
#  * Written by K. Winstanley                29/03/11 *
#  *                                                  *
#  * credit to LinuxQuestions Forum users             *
#  * http://www.linuxquestions.org                    *
#  *                                                  *
#  * This script reads the MINIUPNPD chain of iptables*
#  * containing all upnp opened ports on the local    *
#  * ClearOS gateway and repeats these port mappings  *
#  * on the upstream router using the the local       *
#  * gateway as the destination                       *
#  ****************************************************

# Read local iptables upnp redirects

iptables -L MINIUPNPD > tmp.txt

# Read local Gateway IP

localip=$(ifconfig wlan0 | grep "inet addr" | cut -d : -f 2 | cut -d " " -f 1)

# Set Remote router IGD rootDesc in case of failed discovery

rootdesc="http://192.168.1.254:80/upnp/IGD.xml"

# Parse temp file

while read a b; do                                # Read parameters from tempfile
        b=$(echo $b | cut -d: -f2)                # Clean port field
        upnpc -u $rootdesc -a $localip $b $b $a   # Set ports on remote router to local gateway

done < <(awk 'NF == 7 { print $6, $7 }' tmp.txt)  # Filter tempfile with AWK

rm tmp.txt                                        # Remove tempfile
# end

Many thanks guys

grail · 05-29-2011, 09:47 PM

Well mine was a simple typo

The comma should be a semi colon, try:

Code:

iptables -L MINIUPNPD | awk 'NR > 2{sub(/.*:/,"",$7);cmd = "upnpc -r -a "$5" "$7" "$6;print | cmd}'

Mine is simple when you look at it

1. iptables -L MINIUPNPD | awk - redirect the output of iptables command into awk

2. NR > 2 - this is equivalent to your sed of removing the first 2 lines, mine just ignores them. (PS should you still only be wanting to look at lines with 7 arguments,
you can simply add - && NF == 7 before curly brace)

3. sub(/.*:/,"",$7) - remove everything up until the full colon from the last field (eg. dpt:19955 becomes 19955)

4. cmd = "upnpc -r -a "$5" "$7" "$6 - this is setting the variable cmd to a string representation of what you wish to execute (using your first line of input from
iptables in post #1 it would be - cmd = "upnpc -r -a 192.168.3.124 19955 tcp" )

5. print | cmd - use the command in the shell

Let me know if you need anymore information

UrbanPykey · 05-30-2011, 06:27 AM

AH! i understand! AWK is performing all the operations within the ' ' delimiters. so your script is a lot less processor use since it only calls 3 "programs" AWK, IPTABLES and UPNPC whereas mine performs alot more operations!

Only problem now is working out if i can use incron or inotify to monitor the changes to the MINIUPNPD chain and if it notices a change, run the upnpforwarder script! only thing i can think so far, and i have only just started thinkling about it, would be altering MINIUPNPD slightly to "touch" a file when it adds rules to the MNINIUPNPD chain. incron or inotify can then look for this "touch"ed file.

Any ideas would be much appreciated!

grail · 05-30-2011, 08:05 AM

Sorry ... but I do not know much about iptables and nothing about MINIUPNPD. It may pay to raise another question as the gist of this one has been answered.

David the H. · 05-30-2011, 10:58 AM

Code:

b=$(echo $b | cut -d: -f2)

This always bugs me. There's no need to use cut when built-in parameter substitution works just as well.

Code:

b=${b#*:}

You can also use it to replace the nested cuts in the localip setting.

Code:

localip=$( ifconfig wlan0 | grep "inet addr" )
localip=${localip#*:}
localip=${localip%% *}

An alternate technique would be to alter awk's field separator to account for both spaces and colons, then you can output the field you want directly and not have to worry about splitting it later.

Code:

done < <(awk -F "[[:blank:]:]+" 'NR > 2 { print $6, $8 }' tmp.txt)

In case you need it broken down, -F "[[:blank:]:]+" sets the field separator to a regex that matches to "a string of one or more blank spaces and/or colons". [:blank:] is a character class that matches both space and tab, but not newline.

This should break the colon-ated part into two separate fields. Also, my personal preference is to use NR > 2 to simply skip the first two lines, instead of testing the number of fields on every line. But if you do decide to continue using NF instead, don't forget that NF == 8 now.

How about using the awk command to directly filter the output of iptables instead, so that the temp file only contains the fields you want?

Code:

iptables -L MINIUPNPD | awk -F "[[:blank:]:]+" 'NR > 2 { print $6, $8 }' >tmp.txt

You could even use the command to feed the while loop directly and dispense with a temp file completely.

Speaking of which, this also gives you yet another option for setting localip.

Code:

localip=$( ifconfig wlan0 | awk -F "[[:blank:]:]+" '/inet addr/ {print $4}' )