[SOLVED] Scripting Help: Using "File1" as list of objects to search for in "File2"

Asharru · 08-30-2010, 05:03 PM

Hello all,

I need help with a project I am working on. The goal is to auto-update a webpage that contains links to network devices. These devices are all set to DHCP, and their hostnames are not guaranteed to stay the same, so static IPs or accessing via hostname is NOT an option.

Process:

Run a script every hour or so to query devices on the local network to generate a list of responding devices by MAC address and what their current IP is (results.lst). Using a "static" MAC address list file "printers.db" as the database search "results.lst" to find specific MAC addresses. When a match is found, grab the IP from "results.lst" and update "index.html" with the new IP address. No comparison needs to be done between "results.lst" IP and "index.html" IP as I just want to overwrite whatever is there.

Here is what I have so far (using dummy data):
PRINTERS.DB

Product Mac Address
Printer1 00:00:00:00:00:11
Printer2 00:00:00:00:11:11
Printer3 00:00:00:11:11:11
Printer4 00:00:11:11:11:11
Printer5 00:11:11:11:11:11
Printer6 11:11:11:11:11:11

TEST.SH

Code:

#!/bin/sh

#------------------------------
# Quick Links Updater v2.0    |
# By Rob M.                   |
#  with help from Tim F.      |
#------------------------------

# Clear previous runs
if [ -f results.lst ]
then
rm results.lst
fi

#Get network data and format to remove extraneous data
nmap -sP 192.168.1.0/24 | grep -iv -e "Raw" -e "is down" -e "Nmap" -e "Completed" -e "initiating" -e "Scanning" -e "NSE" | sed -e 's/) is.*//' -e 's/ is.*//' -e 's/[ ^I]*$//' -e '/^$/ d' > results.lst

# concatonate lines and remove more extraneous data
sed '
/([A-Z]/ {
# append a line
        N
# search for 192.168 on the second line
        /\n.*192.168./ {
# found it - now edit making one line
               s/([A-Z].*\n.*192.168./''192.168./
        }
}' results.lst > results.tmp

# remove final bit of extraneous data and delete temp file
cat results.tmp | grep -iv -e "Host " -e "(Company)" | sed -e 's/MAC Address: //' > results.lst
rm results.tmp

# add end catch to file for loop
echo "end" >> results.lst

RESULTS.LST

00:00:00:00:00:00 192.168.1.2
00:00:00:00:00:11 192.168.1.3
00:00:00:00:11:11 192.168.1.4
00:00:00:11:11:11 192.168.1.5
00:00:11:11:11:11 192.168.1.6
00:11:11:11:11:11 192.168.1.7
11:11:11:11:11:11 192.168.1.8
11:00:00:00:00:00 192.168.1.9
11:11:00:00:00:00 192.168.1.10
11:11:11:00:00:00 192.168.1.11
11:11:11:11:00:00 192.168.1.12
11:11:11:11:11:00 192.168.1.13
11:00:11:00:11:00 192.168.1.14
00:11:00:11:00:11 192.168.1.15
11:00:00:00:00:11 192.168.1.16
00:00:11:11:00:00 192.168.1.17

I've tried using awk, grep, sed, cut and while read, until, for, if... However I cannot seem to get it to:

1) Read the first line from the "printers.db" to pull the MAC address from the list
2) Search each line of "results.lst" until it finds a match and pull the IP address for that match
3) Pull the product name from "printers.db"
4) Search "index.html" for the product name
5) Replace the IP address in "index.html" with the IP address pulled from "results.lst"

Any help would be greatly appreciated.

Sergei Steshenko · 08-30-2010, 05:15 PM

Quote:

Originally Posted by Asharru

...I cannot seem to get it to:

1) Read the first line from the "printers.db"
...
Any help would be greatly appreciated.

How about 'man 1 head' ?

ghostdog74 · 08-30-2010, 07:45 PM

Code:

nmap -sP 192.168.0.1/24|awk '/report/{gsub("[()]","",$NF);ip=$NF}/MAC/{map[ip]=$3}END{for(i in map)print i,map[i]}' >result.lst
pname=$(awk 'FNR==2&&FNR==NR{pr=$1;pmac=$2;next} $1~pmac{print $2} ' printers.db result.lst)
# search for product name 
sed "s/<your pattern>/$pname/" html

Asharru · 08-31-2010, 10:30 AM

Quote:

Originally Posted by Sergei Steshenko

How about 'man 1 head' ?

Sorry, my bad. The script has to start with the first entry in "printers.db", compare it to the output from the nmap, then move on to the next entry in "printers.db". This process continues until all 30 or so lines in "printers.db" has been processed. So unfortunately, 'head' will not work in this case.

Asharru · 08-31-2010, 10:56 AM

Quote:

Originally Posted by ghostdog74

Code:

nmap -sP 192.168.0.1/24|awk '/report/{gsub("[()]","",$NF);ip=$NF}/MAC/{map[ip]=$3}END{for(i in map)print i,map[i]}' >result.lst
pname=$(awk 'FNR==2&&FNR==NR{pr=$1;pmac=$2;next} $1~pmac{print $2} ' printers.db result.lst)
# search for product name 
sed "s/<your pattern>/$pname/" html

Ghostdog, thanks for the assist, however this only returns the last match from the "printers.db" file. If I leave 'map[ip]=$3' alone, it shows no ip address for the mac address listed. Also, their is no standard naming pattern for the products so the sed line will not work for me.

I was able to come up with a loop that worked at home using dummy data, however it does not work in testing with the real data:

Code:

#!/bin/sh

while read line
  do
    a=$(cut -f2)
    while read line
      do
        b=$(cut -d" " -f1)
        if [ "$a" = "$b" ]; then
          echo "working!"
        else
          echo "failing!"
        fi
     done < results.lst
done < printers.db

I'm pretty sure it worked because the dummy data I used was a small set and the "results.lst" almost exactly matched the "printers.db"

Asharru · 09-02-2010, 11:30 PM

Alright, I've got the script working to the point that it will search the nmap results using the product database as reference, pull the MAC and product name into two separate arrays, then pulling the IP address into another variable.

Only step left is to modify the index.html to update all of the links for the products on the page. I've played around with sed, got it to work once (before the original post problem) and like a noob didn't save what I had and can't for the life of me remember what I coded.

What I need is this:

Using either sed, awk, or grep, locate a line in my index.html file based on the link text, then change the ip address on that line with the discovered ip address and save the file.

What I have is this:

Code:

grep -i ">${nameArray[$item]}<" index.html | sed 's/192.168.10.[0-9]/'$ipAddy'/' > index.html

nameArray is the array with the product names
product names show up in the html as >3600<
ipAddy is the new ip address associated with the current entry in nameArray array.

grail · 09-02-2010, 11:57 PM

Assuming all your variables are set correctly:

Code:

sed -i.bak "/${nameArray[$item]}/s/192\.168\.10\.[0-9]+/$ipAddy/" index.html

Asharru · 09-04-2010, 05:25 PM

Quote:

Originally Posted by grail

Assuming all your variables are set correctly:

Code:

sed -i.bak "/${nameArray[$item]}/s/192\.168\.10\.[0-9]+/$ipAddy/" index.html

That's actually real close to what I came up with as well

Code:

sed -i "/${nameArray[$item]}/ s/192.168.10.[0-9]*\//$ipAddy\//" index.html

I assume the -i.bak modifies the input file but creates a backup, I didn't bother with that as the tests I ran worked perfectly. I did have to use '*\/' after the [0-9] (and \/ after the $ipAddy) because I noticed in some cases, depending on the final bit of the IP, it would not replace all number {e.g. changing .100 to .23 would come out as .230 or .2300).

Here's the final results (Dummy data of course):

products.db

Code:

00:58:BE:4B:14:87 N-12345
00:58:BE:90:2C:58 N-13579
00:58:BE:96:07:80 L-24680
00:58:BE:9A:B0:49 L-02468
00:58:BE:9E:B0:A6 S-78952
00:58:BE:A2:40:81 S-85476
00:58:BE:A9:D7:9D W-01121
00:58:BE:A0:A1:0B W-13232
00:58:BE:AB:40:2B H-25343
00:58:BE:AC:10:7B H-37454
00:58:BE:AD:C8:AA T-49565
00:58:BE:B1:FF:E9 T-52676
00:58:BE:B7:B6:85 K-64787
00:58:BE:B9:09:78 K-76898
00:58:BE:BC:3F:D5 A-88909
00:58:BE:C2:01:4E A-90010
00:58:BE:C2:10:B6 O-00130
00:58:BE:C2:1D:0C O-18249
00:58:BE:C3:58:3C Q-26358
00:58:BE:CC:58:7E Q-34467
00:58:BE:CE:32:0D R-42576
00:58:BE:DB:02:0F R-59685
00:58:F0:A0:58:00 E-67794
00:45:52:CC:FB:04 E-75803
06:38:32:81:63:B6 V-83542
06:38:32:88:31:D5 V-91871
06:38:32:88:90:09 X-05234
06:38:32:9F:D9:66 X-17748

overwatch.sh

Code:

#!/bin/bash

# ------------------------
# |   Overwatch v2.0    |
# |    By R. Marquardt  |
# ------------------------

# make declarations
declare -a macArray
declare -a nameArray

# get network data and format to remove extraneous data
nmap -sP 192.168.10.0/24 | grep -iv -e "Host 192.168.10.45" -e "Raw" -e "is down" -e "Nmap" -e "Completed" -e "initiating" -e "Scanning" -e "NSE" | sed -e 's/) is.*//' -e 's/ is.*//' -e 's/[ ^I]*$//' -e '/^$/ d' > results.lst

# since nmap places MAC and IP on two lines we need to concatonate lines
sed '/[^)]$/ {N;s/\n//;}' results.lst > results.tmp

# remove final bit of extraneous data
cat results.tmp | sed -e 's/Host.*.192.168.10/''192.168.10/' -e 's/MAC Address: / /' -e 's/ (.*//' > results.lst

# read entries from products.db into macArray and nameArray
let count=0
while read line ; do
  macArray[$count]=$(echo $line | cut -d" " -f1)
  nameArray[$count]=$(echo $line | cut -d" " -f2)
  ((count++))
done < products.db

# search results.lst for entries matching our array and pull out the IP address associated with the entry then update the index.html of the webpage
for item in ${!macArray[*]}
  do
    ipAddy=$(grep -i "${macArray[$item]}" results.lst | cut -d" " -f1)
    sed -i '/>'"${nameArray[$item]}"'</ s/192.168.10.[0-9]*\//'"$ipAddy"'\//' /var/www/ql/index.html
  done

# clean up
if [ -f results.lst ]; then
    rm results.*
fi

Example line in index.html that gets replaced:

Code:

                                                <a href="http://192.168.10.142/" target="_blank"><font size="5">V-91871</font></a> <br>

crts · 09-04-2010, 06:22 PM

Hi,

just wanted to add that you might want to escape the dots in your IP address:

Code:

sed -i "/${nameArray[$item]}/ s/192\.168\.10\.[0-9]*\//$ipAddy\//" index.html

In a regular expression a dot matches any character. So your expression would also match something like
192+168+10+100

I know this is rather unlikely but it is still a good habit.

Asharru · 09-05-2010, 02:08 PM

Quote:

Originally Posted by crts

Hi,

just wanted to add that you might want to escape the dots in your IP address:

Code:

sed -i "/${nameArray[$item]}/ s/192\.168\.10\.[0-9]*\//$ipAddy\//" index.html

In a regular expression a dot matches any character. So your expression would also match something like
192+168+10+100

I know this is rather unlikely but it is still a good habit.

Good point! Even though it is unlikely I suppose I should change it.

grail · 09-05-2010, 06:53 PM

Quote:

it would not replace all number {e.g. changing .100 to .23 would come out as .230 or .2300)

This implies something is wrong with your variable $ipAddy not the regex as the fact that it may add numbers means how you fill this variable is looking for a number
of digits as opposed to setting a replacement value.

Maybe you could show the contents of results.lst that shows what the corresponding line is for .23 address