LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-27-2012, 05:02 PM   #1
UnixNewbie91
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Rep: Reputation: Disabled
GREP command help


Hi

Really new to unix so might not be able to explain my problem clearly but here goes.

I have two variables MONTH and YEAR and need to look through a directory containing 15 or so files for find lines that contain both variables. Once this is done I need the output to show the file name and the number of times both variables appear in each file.

So far I have my grep command at:

grep $MONTH ~/webhits/* | grep -c -H $YEAR

When I run this command I get an output of "(standard input):[the number of times both my variables appear in the whole directory] but I need it broken down into the number of times it appears in each file.

Thanks in advance for any help

Last edited by UnixNewbie91; 04-27-2012 at 05:07 PM.
 
Old 04-27-2012, 05:06 PM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,451

Rep: Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050
Easy way is to use your current script in a loop over each file

Code:
for file in ~/webhits/*; do echo "$file: "$(grep $MONTH "$file" | grep -c $YEAR); done

Last edited by suicidaleggroll; 04-27-2012 at 05:08 PM.
 
1 members found this post helpful.
Old 04-27-2012, 05:09 PM   #3
UnixNewbie91
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Brilliant mate thanks so much. Works like a charm
 
Old 04-27-2012, 05:18 PM   #4
UnixNewbie91
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Another question

Going to need to give more details on the files:

Each file contains the number of web hits for a fake website from a number of different made up IP addresses. Some of the IP addresses appear twice in the same file so the hit from this IP address is counted multiple times. If I wanted to only count the hit from each IP once (so get the number unique hits) how would I do this.

Here is an example of the file if it helps
44.184.167.119 Mon May 07 08:11:50 GMT 2007
78.230.158.130 Thu May 10 01:59:33 GMT 2007
78.230.158.130 Thu May 10 05:14:58 GMT 2007

So for these three hits I want to count the hits from IP address 78.230.158.130 as one unique hit.

Any suggestions?
 
Old 04-27-2012, 05:21 PM   #5
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,451

Rep: Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050Reputation: 2050
I believe that's getting out of the realm of bash/grep. Something like awk would probably be powerful enough to do it. You could either leave your current grep in tact and use awk to find the unique IPs from the match, or use awk to do both steps. I'm not an awk expert though, so I'll let somebody else chime in there.
 
Old 04-27-2012, 06:30 PM   #6
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Code:
echo "44.184.167.119 Mon May 07 08:11:50 GMT 2007
78.230.158.130 Thu May 10 01:59:33 GMT 2007
78.230.158.130 Thu May 10 05:14:58 GMT 2007"|cut -d" " -f 1,7|sort -u
44.184.167.119
78.230.158.130

Cheers,
Tink

Last edited by Tinkster; 04-27-2012 at 06:32 PM. Reason: added ,7 since you're looking for years
 
Old 04-27-2012, 08:37 PM   #7
UnixNewbie91
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Not quite sure what you mean there Tink

Basically I have to write a script that looks at the files which are all set up like the example I gave and then put into a table the name of the folder, the number of hits in a given time frame which are sorted in descending order and then the number of unique hits (number of different IP addresses.

Any help?
 
Old 04-27-2012, 09:07 PM   #8
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
If you didn't use grep to count, but to only output records that match your
$YEAR then pipe that into the cut|sort combo above you could then count the
matching lines.
 
Old 04-27-2012, 09:14 PM   #9
UnixNewbie91
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
My current grep command is as follows:
for file in ~/webhits/*; do echo "$file: "$(grep $MONTH "$file" | grep -c $YEAR); done

Your saying to make it into this?
for file in ~/webhits/*; do echo "$file: "$(grep $MONTH "$file" | grep -c $YEAR)|cut -d" " -f 1,7|sort -u; done
 
Old 04-27-2012, 10:21 PM   #10
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Close ...
Code:
for file in ~/webhits/*; do echo "$file: "$(grep $MONTH "$file" | grep $YEAR|cut -d" " -f 1,7|sort -u|wc -l); done

Last edited by Tinkster; 04-27-2012 at 11:08 PM. Reason: my initial version was off
 
1 members found this post helpful.
Old 04-27-2012, 11:18 PM   #11
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
And after a little thought .... maybe this does what you want?
Code:
$ cat webhits/fakelog
44.184.167.119 Mon May 07 08:11:50 GMT 2007
78.230.158.130 Thu May 10 01:59:33 GMT 2007
78.230.158.130 Thu May 10 05:14:58 GMT 2007
$ MONTH=May
$ YEAR=2007
$ awk -v month=$MONTH -v year=$YEAR '$3==month&&$7==year {a[$1]++}END{print FILENAME": "asort(a)}' webhits/*
webhits/fakelog: 2

Back to the drawing board ... this doesn't work when there's more
than one file :} ... as I said: little thought ;D




Cheers,
Tink

Last edited by Tinkster; 04-27-2012 at 11:31 PM. Reason: [b]
 
Old 04-28-2012, 06:28 AM   #12
UnixNewbie91
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Thanks so much for the help Tink,

for file in ~/webhits/*; do echo "$file: "$(grep $MONTH "$file" | grep $YEAR|cut -d" " -f 1,7|sort -u|wc -l); done

that code works perfectly, cant thank you enough
 
Old 04-29-2012, 12:18 AM   #13
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
And a pure awk solution, finally =o)
Code:
$3==month&&$7==year{
  if(FNR==1){
    fn[FILENAME]=FILENAME
  }
  if(fn[FILENAME]!~$1){
    fn[FILENAME]=fn[FILENAME]","$1
  }
}
END{
  for(i in fn){
    split(fn[i],tmp,",")
    print tmp[1]": "asort(tmp)-1
  }
}
Code:
$ ls webhits/
fakelog  fakelog2
$ cat webhits/fakelog
44.184.167.119 Mon May 07 08:11:50 GMT 2007
78.230.158.130 Thu May 10 01:59:33 GMT 2007
78.230.158.130 Thu May 10 05:14:58 GMT 2007
$ cat webhits/fakelog2
44.184.167.119 Mon May 07 08:11:50 GMT 2007
78.230.158.130 Thu May 10 01:59:33 GMT 2009
78.230.158.130 Thu May 10 01:59:33 GMT 2007
78.230.158.130 Thu May 10 05:14:58 GMT 2008
119.167.184.44 Mon May 07 08:11:50 GMT 2007
Code:
$ awk -v month=$MONTH -v year=$YEAR -f awking webhits/*
webhits/fakelog2: 3
webhits/fakelog: 2

I'm sure grail can come up with something more elegant AND shorter.



Cheers,
Tink
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] grep command borgibo Linux - Newbie 7 07-22-2010 10:22 AM
How to pass the result of a command to another command (like grep) desb01 Programming 4 06-25-2009 12:09 PM
Help me in Grep Command + cd command in single line JeiPrakash Linux - Newbie 3 05-27-2008 04:16 AM
how to use grep command sharonyiisl Linux - Newbie 7 05-28-2006 03:46 PM
Help With GREP Command juliettree Linux - Newbie 3 04-08-2004 08:44 AM


All times are GMT -5. The time now is 07:24 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration