[SOLVED] Retrieve specific data from html (alternative who is)

individual · 09-28-2018, 02:51 PM

Please see my latest update. I fixed a corner case where there is only the country name available.

scasey · 09-28-2018, 03:08 PM

Quote:

Originally Posted by pedropt

Ok Scasey , here it is another example .

Code:

whois 139.99.118.122 | grep -iE ^country | awk {'print$2'}

hope you understood what i mean with whois tool

Interesting. OVH is a Canadian company, of course, so the CA refrence on the 139.99.0.0/16 block makes sense.
The 139.99.0.0/17 block and the 139.99.118.64/26 blocks are OVH Singapore. Presumably other 139.99. IPs are elsewhere, but it would take multiple queries to figure that out.

For your purposes, I'd think that SG is all you'd need to know, and you'd block the /26 block to start.

Note that the abuse reporting addresses are all the same...which is all I usually care about.

This particular anomaly is probably caused by having SG addresses in the ARIN RIR

pedropt · 09-29-2018, 02:01 AM

Quote:

Originally Posted by individual

Please see my latest update. I fixed a corner case where there is only the country name available.

Thank you , i will check it out .

Quote:

Interesting. OVH is a Canadian company, of course, so the CA refrence on the 139.99.0.0/16 block makes sense.
The 139.99.0.0/17 block and the 139.99.118.64/26 blocks are OVH Singapore. Presumably other 139.99. IPs are elsewhere, but it would take multiple queries to figure that out.

For your purposes, I'd think that SG is all you'd need to know, and you'd block the /26 block to start.

Note that the abuse reporting addresses are all the same...which is all I usually care about.

This particular anomaly is probably caused by having SG addresses in the ARIN RIR
Yesterday 02:51 PM

One of the solutions for me on whois tool for this kind of results where to validate only the first result and ignore the other 2 , but on different ips i notice that sometimes the real country came as a 2nd variable from whois and not as the 1st .

Worst than that is when i get EU (Europe) as result from whois and the real country is in US or anywhere else .
Of course if i wanted the abuse contacts only then whois tool was perfect for the job as you told before , but in my case i only need the country , and in this case the country letter code .
Maybe using the code from "Individual i will post here the full code with the country list hosted on an external server somewhere , ex : github , for those who need it , and the script code will pick up the country list in github and display the full country name in the script when it is not available from dnslytics .

pedropt · 09-29-2018, 03:02 AM

Ok guys , here it is the code on github .
Feel free to give suggestions or optimize it by sending pull requests if you think you need .

https://github.com/peterpt/ipcountry

The script uses a file to retrieve full country name witch is in the git also .

individual · 09-29-2018, 09:53 AM

Quote:

Originally Posted by pedropt

Ok guys , here it is the code on github .
Feel free to give suggestions or optimize it by sending pull requests if you think you need .

https://github.com/peterpt/ipcountry

The script uses a file to retrieve full country name witch is in the git also .

Hey, thank you for giving me credit! If I could, I have a few suggestions:
1. Instead of using nc to check for a network connection, why not use ping?

Code:

ping -c1 [HOSTNAME] &>/dev/null

2. You should put your usage banner in its own function, that way you aren't duplicating code.

Code:

function usage {
    echo "Ipcountry"
    echo "-------------------------------------------------"
    echo "Switches :"
    echo "$0 172.217.168.174 (to retrieve country from IP)"
    echo "$0 -h (This help)"
    echo "-------------------------------------------------"
    echo "Thanks to (Individual) LQ for the help" 
    echo "-------------------------------------------------" 
    echo "This Scripts uses Dnslytics.com\n"
    exit 1
}

3. You can simplify your country name lookup.

Code:

cntname=$(grep -E "^ $country" < "$cntlist" | tail -c+7)

pedropt · 09-29-2018, 10:27 AM

ping in some computers dont work well because uses an echo request to google , some computers have problem with that and i believe that is because of some firewall rules on those computers , i have notice that very often in the past in other github projects i have collaborated .
Instead ping i used netcat to connect directly to port 80 using tcp protocol witch is 100% reliable .

Yup , you are right in the function used to display the help , i did the script very fast and i did not realize that option to use .
Usually on my other scripts when i have plenty of time to look into the code and optimize it i do that , but this one was made in 10 minutes and i just uploaded it to github .
I will do that as soon as i get sometime here because right now i have some issues in my cloud to deal and my web server is right now under fully attack .
I will change the code during next week , or if you want just create a github account , clone ipcountry git , make your changes in your code and send the pull request to me that i will accept it .
Maybe if you have any other ideas for the script then feel free to add them at your will , since basically you did the hardest part of the script witch i would problably wont get there easily .

pedropt · 09-29-2018, 10:27 AM

ping in some computers dont work well because uses an echo request to google , some computers have problem with that and i believe that is because of some firewall rules on those computers , i have notice that very often in the past in other github projects i have collaborated .
Instead ping i used netcat to connect directly to port 80 using tcp protocol witch is 100% reliable .

Yup , you are right in the function used to display the help , i did the script very fast and i did not realize that option to use .
Usually on my other scripts when i have plenty of time to look into the code and optimize it i do that , but this one was made in 10 minutes and i just uploaded it to github .
I will do that as soon as i get sometime here because right now i have some issues in my cloud to deal and my web server is right now under fully attack .
I will change the code during next week , or if you want just create a github account , clone ipcountry git , make your changes in your code and send the pull request to me that i will accept it .
Maybe if you have any other ideas for the script then feel free to add them at your will , since basically you did the hardest part of the script witch i would problably wont get there easily .