|
Retrieve specific data from html (alternative who is)
I have been working with whois tool for some time , but sometimes whois database is not very accurate on countries from specific ip .
tcpiputils.com website looks very accurate and we can retrieve an ip data html with wget , by downloading the normal html displayed on browser . But using grep in the downloaded data to retrieve the country is a mess , because depending on some ips , the html can display the city and then the country on same field . Here it is an example of an ip that have been trying to exploit my mail server on tcpiputils https://dnslytics.com/ip/142.11.199.129 I can download the html with all that data using wget , just by sending the command Quote:
but that line is a mess to pick up something or to pick an unique reference for grep to get it , and then i have another issue ahead witch is in other ips the country could be in other line and it could have only country without city name before . here it is a part of that line Quote:
This one looks a bit hard to figure it out at least for me . thanks |
What specific information are you trying to get? Just the city? All of the location information?
|
Hi , i just want the country .
|
Do you want the full country name, or the two letter country code?
|
The one more easier to get , i believe it is the 2 letter country code .
Using the 2 letter code i can make a search for the full country name on a country list i have here . |
Here you go. This was kind of a fun little script to write. It was easiest to use the PCRE (Perl Compatible regular Expressions) mode of grep to search for the appropriate lines. I've included the option to select the full or short country name.
EDIT: Please let me know if anything isn't clear. EDIT2: I'm sorry, full_country is actually the state name. EDIT3: I updated it to get the full country name. Code:
#!/bin/bash |
An interesting site, and interesting script...but I don't see any real advantage over whois
Code:
whois 142.11.199.129 | grep -i Country |
Wow :hattip: .
Very good programming in so short time . Thank you very much for the code , in my script i already use whois , but this one will popup if whois output is not very reliable or old . Somehow , i believe many people will use this code you wrote in future . Thanks again Edited scasey The problem with whois tool is that sometimes is not very accurate , specially if the server is located in one country and the guy that registered it is in another , in whois you will retrieve multiple countries , one ip here i got 3 countries , US , CN (China) , SG (Singapore) . And also because sometimes whois is overloaded and you can get a timeout from the output . The code "Individual" wrote gives you an alternative way to get some ip country name without having to use whois , and also dnsutils website can reverse ip to hostname and a lot of other informations that whois is not able to get . From anyone who uses it , it may use it to get other variables from the webpage that normally you can not get with whois . |
Quote:
|
Quote:
I use whois to get the reporting address for spam reporting, and it works almost all the time. There are issues with KoreaNIC, and sometimes with JPNIC, and I have to go to the relevant web pages for those...sometimes. Here's a script I use to pull the contact information: Code:
#!/bin/bash |
Ok Scasey , here it is another example .
Code:
whois 139.99.118.122 | grep -iE ^country | awk {'print$2'} |
Quote:
Quote:
|
Quote:
When that doesn't work, I go directly to the managing Network Information Center's website (although, I have also bookmarked https://dnslytics.com -- it certainly can be useful.) |
Quote:
My solution was to implement some rules to deal with DOS attacks , and then lookup on the logs what a specific ip have been doing , and depending on that i can block it directly into the firewall with my script . This way i dont have to worry again with that ip . Sometimes when some ip subnet is trying to hack the server , by this i mean that for ex : one day i get a port scan from 192.168.1.30 , i block it in firewall , then next day i get another portscan or a dos from 192.168.1.35 , same treatment in the firewall , then after 2 days i get an attempt to exploitation or anything else from another ip from same subnet like 192.168.1.50 , best way here is to block that subnet as a whole , because as i notice here , most attacks come from another open websites with services for public , and by this i think that those sites were hacked somehow and the hacker is using the website as a remote shell and redirecting the job using a different ip or the owner of the website did had nothing to do and decided to hammer something on the web . Definitively the best way is to block in the firewall or i have to contact isp providers everyday because of abusing ips on its network , i dont have time for that . |
I definitely agree about blocking netblocks, and do that routinely, but only for email, using ucspi-tcp, which can be configured to drop connections on port 25. We currently block about 75% of connection attempts from spamming providers, mostly in other countries.
We've automated reporting such that we can supply a perl script with the reporting address, the delivering IP address, and the name of the Maildir file containing the spam. The script then composes and send an email to the provider. In our experience, the vast majority of providers welcome the reports, as they allow them to address the source of the UCE and thereby avoid being blacklisted. We used to do all that automatically, but the program we'd found to do the contact lookups stopped working and was no longer maintained. Before that happened, we'd built the very effective block list mentioned above, tho...so we get relatively little UCE anymore...a small enough amount that we can manage doing the lookups and reporting manually. |
| All times are GMT -5. The time now is 02:57 PM. |