LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 08-30-2010, 05:03 PM   #1
Asharru
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Rep: Reputation: 0
Question Scripting Help: Using "File1" as list of objects to search for in "File2"


Hello all,

I need help with a project I am working on. The goal is to auto-update a webpage that contains links to network devices. These devices are all set to DHCP, and their hostnames are not guaranteed to stay the same, so static IPs or accessing via hostname is NOT an option.

Process:

Run a script every hour or so to query devices on the local network to generate a list of responding devices by MAC address and what their current IP is (results.lst). Using a "static" MAC address list file "printers.db" as the database search "results.lst" to find specific MAC addresses. When a match is found, grab the IP from "results.lst" and update "index.html" with the new IP address. No comparison needs to be done between "results.lst" IP and "index.html" IP as I just want to overwrite whatever is there.

Here is what I have so far (using dummy data):
PRINTERS.DB
  • Product Mac Address
  • Printer1 00:00:00:00:00:11
  • Printer2 00:00:00:00:11:11
  • Printer3 00:00:00:11:11:11
  • Printer4 00:00:11:11:11:11
  • Printer5 00:11:11:11:11:11
  • Printer6 11:11:11:11:11:11

TEST.SH
Code:
#!/bin/sh

#------------------------------
# Quick Links Updater v2.0    |
# By Rob M.                   |
#  with help from Tim F.      |
#------------------------------

# Clear previous runs
if [ -f results.lst ]
then
rm results.lst
fi

#Get network data and format to remove extraneous data
nmap -sP 192.168.1.0/24 | grep -iv -e "Raw" -e "is down" -e "Nmap" -e "Completed" -e "initiating" -e "Scanning" -e "NSE" | sed -e 's/) is.*//' -e 's/ is.*//' -e 's/[ ^I]*$//' -e '/^$/ d' > results.lst

# concatonate lines and remove more extraneous data
sed '
/([A-Z]/ {
# append a line
        N
# search for 192.168 on the second line
        /\n.*192.168./ {
# found it - now edit making one line
               s/([A-Z].*\n.*192.168./''192.168./
        }
}' results.lst > results.tmp

# remove final bit of extraneous data and delete temp file
cat results.tmp | grep -iv -e "Host " -e "(Company)" | sed -e 's/MAC Address: //' > results.lst
rm results.tmp

# add end catch to file for loop
echo "end" >> results.lst
RESULTS.LST
  • 00:00:00:00:00:00 192.168.1.2
  • 00:00:00:00:00:11 192.168.1.3
  • 00:00:00:00:11:11 192.168.1.4
  • 00:00:00:11:11:11 192.168.1.5
  • 00:00:11:11:11:11 192.168.1.6
  • 00:11:11:11:11:11 192.168.1.7
  • 11:11:11:11:11:11 192.168.1.8
  • 11:00:00:00:00:00 192.168.1.9
  • 11:11:00:00:00:00 192.168.1.10
  • 11:11:11:00:00:00 192.168.1.11
  • 11:11:11:11:00:00 192.168.1.12
  • 11:11:11:11:11:00 192.168.1.13
  • 11:00:11:00:11:00 192.168.1.14
  • 00:11:00:11:00:11 192.168.1.15
  • 11:00:00:00:00:11 192.168.1.16
  • 00:00:11:11:00:00 192.168.1.17

I've tried using awk, grep, sed, cut and while read, until, for, if... However I cannot seem to get it to:

1) Read the first line from the "printers.db" to pull the MAC address from the list
2) Search each line of "results.lst" until it finds a match and pull the IP address for that match
3) Pull the product name from "printers.db"
4) Search "index.html" for the product name
5) Replace the IP address in "index.html" with the IP address pulled from "results.lst"

Any help would be greatly appreciated.
 
Old 08-30-2010, 05:15 PM   #2
Sergei Steshenko
Senior Member
 
Registered: May 2005
Posts: 4,481

Rep: Reputation: 453Reputation: 453Reputation: 453Reputation: 453Reputation: 453
Quote:
Originally Posted by Asharru View Post
...I cannot seem to get it to:

1) Read the first line from the "printers.db"
...
Any help would be greatly appreciated.
How about 'man 1 head' ?
 
Old 08-30-2010, 07:45 PM   #3
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Code:
nmap -sP 192.168.0.1/24|awk '/report/{gsub("[()]","",$NF);ip=$NF}/MAC/{map[ip]=$3}END{for(i in map)print i,map[i]}' >result.lst
pname=$(awk 'FNR==2&&FNR==NR{pr=$1;pmac=$2;next} $1~pmac{print $2} ' printers.db result.lst)
# search for product name 
sed "s/<your pattern>/$pname/" html
 
Old 08-31-2010, 10:30 AM   #4
Asharru
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Sergei Steshenko View Post
How about 'man 1 head' ?
Sorry, my bad. The script has to start with the first entry in "printers.db", compare it to the output from the nmap, then move on to the next entry in "printers.db". This process continues until all 30 or so lines in "printers.db" has been processed. So unfortunately, 'head' will not work in this case.
 
Old 08-31-2010, 10:56 AM   #5
Asharru
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by ghostdog74 View Post
Code:
nmap -sP 192.168.0.1/24|awk '/report/{gsub("[()]","",$NF);ip=$NF}/MAC/{map[ip]=$3}END{for(i in map)print i,map[i]}' >result.lst
pname=$(awk 'FNR==2&&FNR==NR{pr=$1;pmac=$2;next} $1~pmac{print $2} ' printers.db result.lst)
# search for product name 
sed "s/<your pattern>/$pname/" html
Ghostdog, thanks for the assist, however this only returns the last match from the "printers.db" file. If I leave 'map[ip]=$3' alone, it shows no ip address for the mac address listed. Also, their is no standard naming pattern for the products so the sed line will not work for me.

I was able to come up with a loop that worked at home using dummy data, however it does not work in testing with the real data:

Code:
#!/bin/sh

while read line
  do
    a=$(cut -f2)
    while read line
      do
        b=$(cut -d" " -f1)
        if [ "$a" = "$b" ]; then
          echo "working!"
        else
          echo "failing!"
        fi
     done < results.lst
done < printers.db
I'm pretty sure it worked because the dummy data I used was a small set and the "results.lst" almost exactly matched the "printers.db"
 
Old 09-02-2010, 11:30 PM   #6
Asharru
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
Update: Getting closer

Alright, I've got the script working to the point that it will search the nmap results using the product database as reference, pull the MAC and product name into two separate arrays, then pulling the IP address into another variable.

Only step left is to modify the index.html to update all of the links for the products on the page. I've played around with sed, got it to work once (before the original post problem) and like a noob didn't save what I had and can't for the life of me remember what I coded.

What I need is this:

Using either sed, awk, or grep, locate a line in my index.html file based on the link text, then change the ip address on that line with the discovered ip address and save the file.

What I have is this:

Code:
grep -i ">${nameArray[$item]}<" index.html | sed 's/192.168.10.[0-9]/'$ipAddy'/' > index.html
nameArray is the array with the product names
product names show up in the html as >3600<
ipAddy is the new ip address associated with the current entry in nameArray array.
 
Old 09-02-2010, 11:57 PM   #7
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,562

Rep: Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939
Assuming all your variables are set correctly:
Code:
sed -i.bak "/${nameArray[$item]}/s/192\.168\.10\.[0-9]+/$ipAddy/" index.html
 
Old 09-04-2010, 05:25 PM   #8
Asharru
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by grail View Post
Assuming all your variables are set correctly:
Code:
sed -i.bak "/${nameArray[$item]}/s/192\.168\.10\.[0-9]+/$ipAddy/" index.html
That's actually real close to what I came up with as well

Code:
sed -i "/${nameArray[$item]}/ s/192.168.10.[0-9]*\//$ipAddy\//" index.html
I assume the -i.bak modifies the input file but creates a backup, I didn't bother with that as the tests I ran worked perfectly. I did have to use '*\/' after the [0-9] (and \/ after the $ipAddy) because I noticed in some cases, depending on the final bit of the IP, it would not replace all number {e.g. changing .100 to .23 would come out as .230 or .2300).

Here's the final results (Dummy data of course):

products.db
Code:
00:58:BE:4B:14:87 N-12345
00:58:BE:90:2C:58 N-13579
00:58:BE:96:07:80 L-24680
00:58:BE:9A:B0:49 L-02468
00:58:BE:9E:B0:A6 S-78952
00:58:BE:A2:40:81 S-85476
00:58:BE:A9:D7:9D W-01121
00:58:BE:A0:A1:0B W-13232
00:58:BE:AB:40:2B H-25343
00:58:BE:AC:10:7B H-37454
00:58:BE:AD:C8:AA T-49565
00:58:BE:B1:FF:E9 T-52676
00:58:BE:B7:B6:85 K-64787
00:58:BE:B9:09:78 K-76898
00:58:BE:BC:3F:D5 A-88909
00:58:BE:C2:01:4E A-90010
00:58:BE:C2:10:B6 O-00130
00:58:BE:C2:1D:0C O-18249
00:58:BE:C3:58:3C Q-26358
00:58:BE:CC:58:7E Q-34467
00:58:BE:CE:32:0D R-42576
00:58:BE:DB:02:0F R-59685
00:58:F0:A0:58:00 E-67794
00:45:52:CC:FB:04 E-75803
06:38:32:81:63:B6 V-83542
06:38:32:88:31:D5 V-91871
06:38:32:88:90:09 X-05234
06:38:32:9F:D9:66 X-17748
overwatch.sh
Code:
#!/bin/bash

# ------------------------
# |   Overwatch v2.0    |
# |    By R. Marquardt  |
# ------------------------

# make declarations
declare -a macArray
declare -a nameArray

# get network data and format to remove extraneous data
nmap -sP 192.168.10.0/24 | grep -iv -e "Host 192.168.10.45" -e "Raw" -e "is down" -e "Nmap" -e "Completed" -e "initiating" -e "Scanning" -e "NSE" | sed -e 's/) is.*//' -e 's/ is.*//' -e 's/[ ^I]*$//' -e '/^$/ d' > results.lst

# since nmap places MAC and IP on two lines we need to concatonate lines
sed '/[^)]$/ {N;s/\n//;}' results.lst > results.tmp

# remove final bit of extraneous data
cat results.tmp | sed -e 's/Host.*.192.168.10/''192.168.10/' -e 's/MAC Address: / /' -e 's/ (.*//' > results.lst

# read entries from products.db into macArray and nameArray
let count=0
while read line ; do
  macArray[$count]=$(echo $line | cut -d" " -f1)
  nameArray[$count]=$(echo $line | cut -d" " -f2)
  ((count++))
done < products.db

# search results.lst for entries matching our array and pull out the IP address associated with the entry then update the index.html of the webpage
for item in ${!macArray[*]}
  do
    ipAddy=$(grep -i "${macArray[$item]}" results.lst | cut -d" " -f1)
    sed -i '/>'"${nameArray[$item]}"'</ s/192.168.10.[0-9]*\//'"$ipAddy"'\//' /var/www/ql/index.html
  done

# clean up
if [ -f results.lst ]; then
    rm results.*
fi
Example line in index.html that gets replaced:

Code:
                                                <a href="http://192.168.10.142/" target="_blank"><font size="5">V-91871</font></a> <br>
 
Old 09-04-2010, 06:22 PM   #9
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi,

just wanted to add that you might want to escape the dots in your IP address:
Code:
sed -i "/${nameArray[$item]}/ s/192\.168\.10\.[0-9]*\//$ipAddy\//" index.html
In a regular expression a dot matches any character. So your expression would also match something like
192+168+10+100

I know this is rather unlikely but it is still a good habit.
 
Old 09-05-2010, 02:08 PM   #10
Asharru
LQ Newbie
 
Registered: Aug 2010
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by crts View Post
Hi,

just wanted to add that you might want to escape the dots in your IP address:
Code:
sed -i "/${nameArray[$item]}/ s/192\.168\.10\.[0-9]*\//$ipAddy\//" index.html
In a regular expression a dot matches any character. So your expression would also match something like
192+168+10+100

I know this is rather unlikely but it is still a good habit.
Good point! Even though it is unlikely I suppose I should change it.
 
Old 09-05-2010, 06:53 PM   #11
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,562

Rep: Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939Reputation: 1939
Quote:
it would not replace all number {e.g. changing .100 to .23 would come out as .230 or .2300)
This implies something is wrong with your variable $ipAddy not the regex as the fact that it may add numbers means how you fill this variable is looking for a number
of digits as opposed to setting a replacement value.

Maybe you could show the contents of results.lst that shows what the corresponding line is for .23 address
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Telling people to use "Google," to "RTFM," or "Use the search feature" Ausar General 77 03-21-2010 11:26 AM
"Multicolumn" or "tiles", or even "list" icon view on desktop, in any DE? the dsc Linux - Desktop 3 02-20-2010 09:25 AM
"list dynamic dependency" of an executable using command other than "ldd" Amrita@3086 Solaris / OpenSolaris 3 04-04-2007 04:56 AM
LXer: Displaying "MyComputer", "Trash", "Network Servers" Icons On A GNOME Desktop LXer Syndicated Linux News 0 04-02-2007 08:31 AM
Apply "Advanced Search" options to "My LQ" searches PTrenholme LQ Suggestions & Feedback 22 03-10-2007 08:30 AM


All times are GMT -5. The time now is 12:32 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration