script to search and count hits from some country

mad_penguin · 03-25-2011, 04:13 PM

Hi Guys

I need something to make a script that will search some logs and extract IP hits from one country only. Let's say UK. I guess I need to use GeoIP or some database. I just need a very simple bash, perl, php script that will do this job. Just search threw logs (apache) and then give me number of hits found from UK.

I just need some guidance, or if you already have a already made script that will be wonderful.

Thanks in advance.

David the H. · 03-25-2011, 06:35 PM

It would really help if you gave us some details about your requirements, such as a representative example of the log text and the lines you need to extract. Once you have the strings you need to match, it would probably only take a simple grep command to extract them from the logs.

mad_penguin · 03-25-2011, 07:25 PM

Log is from apache ( so, standard apache log ).

I really don't think a 'simple' grep will do the job. How can you tell using a simple grep what's the location of an IP ?

Once again, I need to 'extract' from this apache log all the IP's that are from UK and make a total.

grail · 03-26-2011, 03:38 AM

Fair enough that it may or may not be a simple grep, but without data the argument seems a little moot.

If we are to do it blind, I would suggest a while loop fed by awk, sed or grep to get IPs from log and then whatever program will tell you where an address is
from executed on each iteration of the loop.

David the H. · 03-26-2011, 03:57 AM

Ok, so I had it backwards. You want to extract ip addresses, then use some tool to check where they are from. So you probably still want to use grep (or sed or awk) to extract the addresses, then use some other tool to determine the location.

But since I don't use apache and have never seen an apache log, I can't tell you how to do the first part; and I have no personal reason for wanting to know where ip addresses come from, so I also have no idea what tool would be suitable to use for the second part.

So again, give us some example text, and tell us how you would normally determine an ip address via the shell, and I'm sure it would be fairly easy to wrap it up into a script.

Or perhaps, as I was originally thinking, if you know what would be considered a UK address range (or ranges), then it might be possible to design a grep/sed/awk function to extract only those addresses directly from the log.

corp769 · 03-26-2011, 04:02 AM

I have this script that you could possibly use: (I found it on the internet... name it as ipgeo and chmod it, or just use it within your script)

Code:

#!/bin/sh
URL="http://api.ipinfodb.com/v2/ip_query.php?key=30df9c9eaf73f8348950e325b50cfc583b8c14d6e110c0d59657236cc7c33192&ip="
WGET_OPTION="=-b -q --wait=3 --waitretry=2 --random-wait --limit-rate=9578 "
WGET_AGENT="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
ERROR=0
if [ "$1" = "" ]; then
  ERROR=1
else
  IP=$1
  HTTP_LINK_XML="$URL""$IP""&output=xml&timezone="
  HTTP_LINK_JSON="$URL""$IP""&output=json&timezone="
fi
if [ "$2" != "" ]; then
  if [ "$2" != "json" ] && [ "$2" != "xml" ] ; then
    ERROR=1
  fi
  TYPE="$2"
else
  ERROR=1
fi
if [ "$3" != "" ]; then
  if [ "$3" != "true" ] && [ "$3" != "false" ] ; then
    ERROR=1
  fi
  TIMEZONE="$3"
else
  ERROR=1
fi
if [ "$ERROR" != "0" ]; then
  echo " "
  echo " usage : $0 IP TYPE TIMEZONE"
  echo " Where IP is the IP to check"
  echo " TYPE is the output type (xml|json)"
  echo " TIMEZONE is to show timezone data or not (false|true)"
  echo " Big thanks to the team of IPInfoDB (http://ipinfodb.com)"
  exit
fi
TST_wget=`wget > /dev/null 2>&1`
ErrorLevel=$?
if [ "$ErrorLevel" != 1 ] ; then
  echo " ----"
  echo " wget not found; please install it for proper operation."
  echo " ----"
  exit
fi
if [ "$TYPE" = "json" ]; then
  JSON_Info=`wget -qO- --user-agent="$WGET_AGENT" $HTTP_LINK_JSON"$TIMEZONE" 2>&1`
  echo "$JSON_Info"
else
  XML_Info=`wget -qO- --user-agent="$WGET_AGENT" $HTTP_LINK_XML"$TIMEZONE" 2>&1`
  echo "$XML_Info"
fi