LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-26-2011, 04:48 PM   #1
random0munky
LQ Newbie
 
Registered: Jul 2011
Location: Washington, USA
Distribution: Ubuntu, CentOS, FreeBSD
Posts: 22

Rep: Reputation: Disabled
Writing a script that compares two different files


Hi, I'm having trouble figuring out how to match to find matches in two different files when comparing timestamps. The fields I'm wanting to match up are in the format:

Jul 26 09:33:02

I have tried reading the file line by line and using
awk '{print $1,$2,$3}' which only gets and stores the timestamp in one of the files. I've been looking around and saw this example:

awk 'FNR==NR{!a[$3]++;next }{ b[$3]++ }
END{
for(i in a){
for(k in b){
if (a[i]==1 && i ~ k ) { print i }
}
}
}' $FILE $FILE2

Which sorta works but its way over my head at the moment. The two files can be found in your /var/log/syslog and /var/log/auth.log (using Ubuntu 11.04)

Thank You
 
Old 07-26-2011, 05:37 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Hi, welcome to LQ!


Quote:
I have tried reading the file line by line and using
awk '{print $1,$2,$3}' which only gets and stores the timestamp in one of the files. I've been looking around and saw this example:
Code:
awk '
FNR==NR{
  !a[$3]++;
  next 
}
{ 
  b[$3]++ 
}
END{
  for(i in a){
    for(k in b){
      if (a[i]==1 && i ~ k ) {  
        print i 
      }
    }
  }
}' $FILE $FILE2
Which sorta works but its way over my head at the moment. The two files can be found in your /var/log/syslog and /var/log/auth.log (using Ubuntu 11.04)
So if this "sort of works" - what is your question?


Cheers,
Tink

Last edited by Tinkster; 07-26-2011 at 05:39 PM.
 
1 members found this post helpful.
Old 07-26-2011, 05:42 PM   #3
random0munky
LQ Newbie
 
Registered: Jul 2011
Location: Washington, USA
Distribution: Ubuntu, CentOS, FreeBSD
Posts: 22

Original Poster
Rep: Reputation: Disabled
Ah thanks. You know I reference this site a lot but I haven't actually posted anything here =) Anyways, my goal is to take an entry from syslog for instance:

Jul 26 11:35:44 bdouglas kernel: [70761.603498] usb 2-1.1.4: new high speed USB device using ehci_hcd and address 12

and an entry from auth.log:

Jul 26 13:17:01 bdouglas CRON[11888]: pam_unix(cron:session): session closed for user root

And compare the contents of both log files by their timestamps. If their timestamps match the exact hour:min:sec, I want both entries printed.
 
Old 07-26-2011, 05:49 PM   #4
Snark1994
Senior Member
 
Registered: Sep 2010
Location: Wales, UK
Distribution: Arch
Posts: 1,632
Blog Entries: 3

Rep: Reputation: 346Reputation: 346Reputation: 346Reputation: 346
I don't know if this is less over your head, but... (I don't know awk, unfortunately, but I can understand bash):

Code:
while read line;
do
    grep "$(echo $line | grep -o '^[[:alpha:]]\{3\} [[:digit:]]\{2\} [[:digit:]]\{2\}\:[[:digit:]]\{2\}\:[[:digit:]]\{2\}')" /var/log/syslog;
done < /var/log/auth.log
It looks a bit scary, but that whole regular-expression just matches something in the format "Jul 26 11:35:44" at the beginning of the line. So all it's doing is looping through each line in auth.log, finding the bit that matches (the timestamp), and searching through syslog to find any lines which match this, then printing them.

Hope this helps,

Last edited by Snark1994; 07-26-2011 at 05:51 PM. Reason: Coloured regex to make command more readable
 
1 members found this post helpful.
Old 07-26-2011, 05:57 PM   #5
random0munky
LQ Newbie
 
Registered: Jul 2011
Location: Washington, USA
Distribution: Ubuntu, CentOS, FreeBSD
Posts: 22

Original Poster
Rep: Reputation: Disabled
Hmm. That is pretty straight forward. I like what I see so far. I can do an awk $3 which grabs the 3rd field not separated by spaces. I like how you feed a file into another file, I was getting the impression you were overwriting the auth.log file but the alligator is pointing the other way. I'll play with this and see what I can come up with. I would also like to see other variations of doing this since it looks a bit long winded. Thank You
 
Old 07-26-2011, 06:28 PM   #6
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
And an "awk" method ..
Code:
FNR==NR{
  a[$1" "$2" "$3]=$0

}
FNR<NR{
  b[$1" "$2" "$3]=$0
}
END{
  for(i in a){
    #print "I: "i
    for(k in b){
    #print "J: "k
      if ( i == k ) {
        print a[i]
        print b[k]
      }
    }
  }
}
 
1 members found this post helpful.
Old 07-26-2011, 06:34 PM   #7
random0munky
LQ Newbie
 
Registered: Jul 2011
Location: Washington, USA
Distribution: Ubuntu, CentOS, FreeBSD
Posts: 22

Original Poster
Rep: Reputation: Disabled
Where does file 1 and file 2 go exactly. Having a hard time visualizing. Thanks
 
Old 07-26-2011, 06:36 PM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,357

Rep: Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367Reputation: 2367
Here's a Perl version; easy to read regexes and very flexible.
It assumes no more than one match per second, re-opens the 2nd file for each rec in first file, much like post#4 soln.
You could amend it to actually compare the dates in meaningful terms, ie so it knows when its passed the date/time in the 2nd file and doesn't waste time checking further recs; otoh this would mean checking all recs until a match or EOF...
Perl is very quick, so you prob don't need to worry about date matching.
Code:
#!/usr/bin/perl -w
use strict;             # Enforce declarations

my (
    $syslog_file, $s_rec, $s_mth, $s_day, $s_time,
    $auth_file, $a_rec, $a_mth, $a_day, $a_time,
    );

$syslog_file='syslog_tmp';
$auth_file='auth_tmp';

open( S_FILE, "<$syslog_file" ) or
            die "Can't open syslog_file: $syslog_file: $!\n";
while ( defined ( $s_rec = <S_FILE> ) )
{
    chomp($s_rec);
    ($s_mth, $s_day, $s_time) = (split(/\s+/, $s_rec))[0..2];

#DEBUG
#print "$s_mth, $s_day, $s_time\n";

    open( A_FILE, "<$auth_file" ) or
                die "Can't open auth_file: $auth_file: $!\n";
    while ( defined ( $a_rec = <A_FILE> ) )
    {
        chomp($a_rec);
        ($a_mth, $a_day, $a_time) = (split(/\s+/, $a_rec))[0..2];

#DEBUG
#print "$a_mth, $a_day, $a_time\n";

        if( $s_mth eq $a_mth && $s_day eq $a_day && $s_time eq $a_time )
        {
            print "$s_rec\n$a_rec\n\n";
            last;
        }
    }
    close(A_FILE) or die "Can't close auth_file: $auth_file: $!\n";;

}
close(S_FILE) or die "Can't close syslog_file: $syslog_file: $!\n";;
 
1 members found this post helpful.
Old 07-26-2011, 07:23 PM   #9
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Quote:
Originally Posted by random0munky View Post
Where does file 1 and file 2 go exactly. Having a hard time visualizing. Thanks
If you save the above as say munky.awk
Code:
awk -f munky file1 file2
 
1 members found this post helpful.
Old 07-26-2011, 07:55 PM   #10
random0munky
LQ Newbie
 
Registered: Jul 2011
Location: Washington, USA
Distribution: Ubuntu, CentOS, FreeBSD
Posts: 22

Original Poster
Rep: Reputation: Disabled
Ah gotcha gotcha I'll take a look at it thank you for the reply
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Need help writing a script to tar/delete old files aboleth Programming 5 07-23-2009 10:15 AM
Slack-11: script lists installed pkgs, out-of-tree packages, compares versions... GrapefruiTgirl Slackware 0 06-18-2009 12:23 AM
List missing files - help writing a script laki47 Linux - Newbie 7 03-30-2009 12:58 PM
Need help in writing a script to move old files... NYMets91587 Linux - Newbie 9 02-20-2009 05:43 AM
writing awk script files bigmark Linux - Software 1 10-19-2005 10:33 AM


All times are GMT -5. The time now is 01:57 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration