LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
LinkBack Search this Thread
Old 03-08-2012, 04:14 AM   #1
elmo219
LQ Newbie
 
Registered: Mar 2012
Posts: 10

Rep: Reputation: Disabled
Shell Script issue


Hi guys, newbie here :-D

I'm trying to create a script that will pull the 20 top queried domains from my DNS server.

Only issue is i have 99 query logs for any one day, this is what i have so far;

Code:
#!/bin/bash
 
echo "Top 20"
echo ""
for file in $(find /var/dns/log/ -iname "dns-query.log.*"); 
do cat "${file}"|awk '{print $6}'|sed 's/www.//'|awk '{ FS = "." } ; { print $1"."$2"."$3"." }'| sort | uniq -c | sort -nr | head -n 20
done
Now this works in a way.... However it prints the top 20 for all 99 files one after another. What i want to do is some how add these 99 files together then run my command for sorting the queries.

so gives me an output like so;

Quote:
Top 20 Domains

5557 google.com..
4852 api-read.facebook.com.
3817 orcart.facebook.com.
3319 api.facebook.com.
3028 facebook.com..
2577 m.hotmail.com.
2389 pop3.live.com.
2088 profile.ak.fbcdn.
1936 fbcdn-profile-a.akamaihd.net.
1899 mtalk.google.com.
1836 static.ak.fbcdn.
1691 m.facebook.com.
1424 google.co.uk.
1423 ksn2-12.kaspersky-labs.com.
1408 fbcdn-photos-a.akamaihd.net.
1336 google-analytics.com..
1213 android.clients.google.
1177 s-static.ak.facebook.
1168 api.twitter.com.
1095 wpad...
5586 google.com..
4781 api-read.facebook.com.
3638 orcart.facebook.com.

continues for the next 99 files
Anyone have any ideas, or even understand what i'm blabbing on about?
 
Old 03-08-2012, 10:16 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 13,772

Rep: Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346Reputation: 2346
Quote:
Originally Posted by elmo219 View Post
Hi guys, newbie here :-D
I'm trying to create a script that will pull the 20 top queried domains from my DNS server. Only issue is i have 99 query logs for any one day, this is what i have so far;

Now this works in a way.... However it prints the top 20 for all 99 files one after another. What i want to do is some how add these 99 files together then run my command for sorting the queries. so gives me an output like so;

Anyone have any ideas, or even understand what i'm blabbing on about?
Seems like you're most of the way there. If they're just text/log files you're talking about, why not just use a low-tech solution? Something like this:
Code:
#!/bin/bash
 
echo "Top 20"
echo ""
cat `find /var/dns/log/ -iname "dns-query.log.*" >> big-log-file.log 
cat big-log-file.log |awk '{print $6}'|sed 's/www.//'|awk '{ FS = "." } ; { print $1"."$2"."$3"." }'| sort | uniq -c | sort -nr | head -n 20
rm big-log-file.log
Just combine all of them into one big file, then run your operation on it.
 
Old 03-09-2012, 02:36 AM   #3
elmo219
LQ Newbie
 
Registered: Mar 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Hi thanks for replying...

Yeah i got the Bash script working like so;

Code:
 #!/bin/bash
 
echo "Top 20 Domains"
echo ""
cat /var/dns/log/dns-query.log.*|awk '{print $6}'|sed 's/www.//'|awk '{ FS = "." } ; { print $1"."$2"."$3"." }' > /tmp/collectorstats
cat /tmp/collectorstats| sort | uniq -c | sort -nr | head -n 20
rm /tmp/collectorstats
As i stripped out the info first it made the collectorstats file alot smaller.

However it seems after running it i will have to re-write this in perl as the sort is taking 99% usage of one processor :-/

Down side is i'm completely useless at patter matching in perl :-/
 
Old 03-09-2012, 08:59 AM   #4
elmo219
LQ Newbie
 
Registered: Mar 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
ok guys quick update. I'm struggling through converting this simple bash script to perl but having trouble with the sort;

Code:
 #!/usr/bin/perl
#use strict;
#use warnings;
print "Top 20 Queried Domains";
print "\n";
 
#my $txtfile = '/var/dns/log/dns-query.log.1';
my $txtfile = '/var/dns/log/dns-test-2';
 
my $url_queries = {};
my $ip_queries  = {};
#!/usr/bin/perl
#use strict;
#use warnings;
print "Top 20 Queried Domains";
print "\n";
 
#my $txtfile = '/var/dns/log/dns-query.log.1';
my $txtfile = '/var/dns/log/dns-test-2';
 
my $url_queries = {};
my $ip_queries  = {};
 
open (READ, "$txtfile") || die "Can't open logs\n";
while ($line = <READ>){
chomp ($line);
$line =~ s/#/ /g;
$line =~ s/www\./ /g;
($date,$time,$client,$ip,$qn,$query,$dnsname,@d)=split(" ",$line);
 
if ( defined $url_queries->{$dnsname} )
{
  $url_queries->{$dnsname}=$url_queries->{$dnsname}+1;
}
else
{
  $url_queries->{$dnsname}=1;
}
 
if ( defined $ip_queries->{$ip} )
{
  $ip_queries->{$ip}=$ip_queries->{$ip}+1;
}
else
{
  $ip_queries->{$ip}=1;
}
}
close READ || die "Couldn't close logs";
 
# Sort
 
 
# Show 20
$count=5;
while ( (($key, $value) = each(%$url_queries)) && ($count>0) )
#foreach $value (sort{$url_queries{$a} cmp $url_queries{$b}} keys %$url_queries)
{
     print "$key\t\t$value\n";
     #print "$value\t\t$url_queries{$value}\n";
     $count--;
}
 
$count=5;
while ( (($key, $value) = each(%$ip_queries)) && ($count>0) )
{
     print "$key\t\t$value\n";
     $count--;
}
exit (0);
which provides an output like so ;

Quote:
./counter.pl
Top 20 Queried Domains
40-courier.push.apple.com 1
api-read.facebook.com 1
147.66.194.173.in-addr.arpa 1
ssl.google-analytics.com 1
download965.avast.com 13
xxx.xxx.xxx.xxx 1
xxx.xxx.xxx.xxx 3
xxx.xxx.xxx.xxx 1
xxx.xxx.xxx.xxx 1
xxx.xxx.xxx.xxx 1
i've obviously blanked out the IP's

*Sorry edited and cleaned up a bit*

So i can make it clearer, where would i put the sort command here so i can sort on the count value?

Last edited by elmo219; 03-09-2012 at 09:56 AM. Reason: clearing up
 
Old 03-09-2012, 10:08 AM   #5
elmo219
LQ Newbie
 
Registered: Mar 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
ffs

Last edited by elmo219; 03-09-2012 at 10:23 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Issue with shell script zaeem Linux - Networking 10 01-06-2011 09:56 AM
Shell script issue suvra82002 Linux - Enterprise 23 07-26-2008 02:02 PM
Shell Script Exporting Issue trek413 Linux - Software 1 11-01-2006 04:18 PM
issue with shell script chupacabra Linux - General 3 10-18-2002 08:12 PM
Out of guesses! (shell Script issue) chris Linux - General 2 12-10-2001 04:20 PM


All times are GMT -5. The time now is 02:30 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration