[SOLVED] Scripting Help: Using "File1" as list of objects to search for in "File2"
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Scripting Help: Using "File1" as list of objects to search for in "File2"
Hello all,
I need help with a project I am working on. The goal is to auto-update a webpage that contains links to network devices. These devices are all set to DHCP, and their hostnames are not guaranteed to stay the same, so static IPs or accessing via hostname is NOT an option.
Process:
Run a script every hour or so to query devices on the local network to generate a list of responding devices by MAC address and what their current IP is (results.lst). Using a "static" MAC address list file "printers.db" as the database search "results.lst" to find specific MAC addresses. When a match is found, grab the IP from "results.lst" and update "index.html" with the new IP address. No comparison needs to be done between "results.lst" IP and "index.html" IP as I just want to overwrite whatever is there.
Here is what I have so far (using dummy data): PRINTERS.DB
Product Mac Address
Printer1 00:00:00:00:00:11
Printer2 00:00:00:00:11:11
Printer3 00:00:00:11:11:11
Printer4 00:00:11:11:11:11
Printer5 00:11:11:11:11:11
Printer6 11:11:11:11:11:11
TEST.SH
Code:
#!/bin/sh
#------------------------------
# Quick Links Updater v2.0 |
# By Rob M. |
# with help from Tim F. |
#------------------------------
# Clear previous runs
if [ -f results.lst ]
then
rm results.lst
fi
#Get network data and format to remove extraneous data
nmap -sP 192.168.1.0/24 | grep -iv -e "Raw" -e "is down" -e "Nmap" -e "Completed" -e "initiating" -e "Scanning" -e "NSE" | sed -e 's/) is.*//' -e 's/ is.*//' -e 's/[ ^I]*$//' -e '/^$/ d' > results.lst
# concatonate lines and remove more extraneous data
sed '
/([A-Z]/ {
# append a line
N
# search for 192.168 on the second line
/\n.*192.168./ {
# found it - now edit making one line
s/([A-Z].*\n.*192.168./''192.168./
}
}' results.lst > results.tmp
# remove final bit of extraneous data and delete temp file
cat results.tmp | grep -iv -e "Host " -e "(Company)" | sed -e 's/MAC Address: //' > results.lst
rm results.tmp
# add end catch to file for loop
echo "end" >> results.lst
RESULTS.LST
00:00:00:00:00:00 192.168.1.2
00:00:00:00:00:11 192.168.1.3
00:00:00:00:11:11 192.168.1.4
00:00:00:11:11:11 192.168.1.5
00:00:11:11:11:11 192.168.1.6
00:11:11:11:11:11 192.168.1.7
11:11:11:11:11:11 192.168.1.8
11:00:00:00:00:00 192.168.1.9
11:11:00:00:00:00 192.168.1.10
11:11:11:00:00:00 192.168.1.11
11:11:11:11:00:00 192.168.1.12
11:11:11:11:11:00 192.168.1.13
11:00:11:00:11:00 192.168.1.14
00:11:00:11:00:11 192.168.1.15
11:00:00:00:00:11 192.168.1.16
00:00:11:11:00:00 192.168.1.17
I've tried using awk, grep, sed, cut and while read, until, for, if... However I cannot seem to get it to:
1) Read the first line from the "printers.db" to pull the MAC address from the list
2) Search each line of "results.lst" until it finds a match and pull the IP address for that match
3) Pull the product name from "printers.db"
4) Search "index.html" for the product name
5) Replace the IP address in "index.html" with the IP address pulled from "results.lst"
nmap -sP 192.168.0.1/24|awk '/report/{gsub("[()]","",$NF);ip=$NF}/MAC/{map[ip]=$3}END{for(i in map)print i,map[i]}' >result.lst
pname=$(awk 'FNR==2&&FNR==NR{pr=$1;pmac=$2;next} $1~pmac{print $2} ' printers.db result.lst)
# search for product name
sed "s/<your pattern>/$pname/" html
Sorry, my bad. The script has to start with the first entry in "printers.db", compare it to the output from the nmap, then move on to the next entry in "printers.db". This process continues until all 30 or so lines in "printers.db" has been processed. So unfortunately, 'head' will not work in this case.
nmap -sP 192.168.0.1/24|awk '/report/{gsub("[()]","",$NF);ip=$NF}/MAC/{map[ip]=$3}END{for(i in map)print i,map[i]}' >result.lst
pname=$(awk 'FNR==2&&FNR==NR{pr=$1;pmac=$2;next} $1~pmac{print $2} ' printers.db result.lst)
# search for product name
sed "s/<your pattern>/$pname/" html
Ghostdog, thanks for the assist, however this only returns the last match from the "printers.db" file. If I leave 'map[ip]=$3' alone, it shows no ip address for the mac address listed. Also, their is no standard naming pattern for the products so the sed line will not work for me.
I was able to come up with a loop that worked at home using dummy data, however it does not work in testing with the real data:
Code:
#!/bin/sh
while read line
do
a=$(cut -f2)
while read line
do
b=$(cut -d" " -f1)
if [ "$a" = "$b" ]; then
echo "working!"
else
echo "failing!"
fi
done < results.lst
done < printers.db
I'm pretty sure it worked because the dummy data I used was a small set and the "results.lst" almost exactly matched the "printers.db"
Alright, I've got the script working to the point that it will search the nmap results using the product database as reference, pull the MAC and product name into two separate arrays, then pulling the IP address into another variable.
Only step left is to modify the index.html to update all of the links for the products on the page. I've played around with sed, got it to work once (before the original post problem) and like a noob didn't save what I had and can't for the life of me remember what I coded.
What I need is this:
Using either sed, awk, or grep, locate a line in my index.html file based on the link text, then change the ip address on that line with the discovered ip address and save the file.
What I have is this:
Code:
grep -i ">${nameArray[$item]}<" index.html | sed 's/192.168.10.[0-9]/'$ipAddy'/' > index.html
nameArray is the array with the product names
product names show up in the html as >3600<
ipAddy is the new ip address associated with the current entry in nameArray array.
sed -i.bak "/${nameArray[$item]}/s/192\.168\.10\.[0-9]+/$ipAddy/" index.html
That's actually real close to what I came up with as well
Code:
sed -i "/${nameArray[$item]}/ s/192.168.10.[0-9]*\//$ipAddy\//" index.html
I assume the -i.bak modifies the input file but creates a backup, I didn't bother with that as the tests I ran worked perfectly. I did have to use '*\/' after the [0-9] (and \/ after the $ipAddy) because I noticed in some cases, depending on the final bit of the IP, it would not replace all number {e.g. changing .100 to .23 would come out as .230 or .2300).
#!/bin/bash
# ------------------------
# | Overwatch v2.0 |
# | By R. Marquardt |
# ------------------------
# make declarations
declare -a macArray
declare -a nameArray
# get network data and format to remove extraneous data
nmap -sP 192.168.10.0/24 | grep -iv -e "Host 192.168.10.45" -e "Raw" -e "is down" -e "Nmap" -e "Completed" -e "initiating" -e "Scanning" -e "NSE" | sed -e 's/) is.*//' -e 's/ is.*//' -e 's/[ ^I]*$//' -e '/^$/ d' > results.lst
# since nmap places MAC and IP on two lines we need to concatonate lines
sed '/[^)]$/ {N;s/\n//;}' results.lst > results.tmp
# remove final bit of extraneous data
cat results.tmp | sed -e 's/Host.*.192.168.10/''192.168.10/' -e 's/MAC Address: / /' -e 's/ (.*//' > results.lst
# read entries from products.db into macArray and nameArray
let count=0
while read line ; do
macArray[$count]=$(echo $line | cut -d" " -f1)
nameArray[$count]=$(echo $line | cut -d" " -f2)
((count++))
done < products.db
# search results.lst for entries matching our array and pull out the IP address associated with the entry then update the index.html of the webpage
for item in ${!macArray[*]}
do
ipAddy=$(grep -i "${macArray[$item]}" results.lst | cut -d" " -f1)
sed -i '/>'"${nameArray[$item]}"'</ s/192.168.10.[0-9]*\//'"$ipAddy"'\//' /var/www/ql/index.html
done
# clean up
if [ -f results.lst ]; then
rm results.*
fi
it would not replace all number {e.g. changing .100 to .23 would come out as .230 or .2300)
This implies something is wrong with your variable $ipAddy not the regex as the fact that it may add numbers means how you fill this variable is looking for a number
of digits as opposed to setting a replacement value.
Maybe you could show the contents of results.lst that shows what the corresponding line is for .23 address
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.