LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-28-2017, 09:12 AM   #1
ocrts
LQ Newbie
 
Registered: Mar 2017
Posts: 3

Rep: Reputation: Disabled
find/grep question


I'm using the following command to get the number of *.log files containing "yes" or "no" at the end of the file (the files are large and I'm looking for "yes" or "no" near the end of each file, and each file will contain at most one "yes" OR one "no", but never both):

find ./OUTPUT -type f | grep '\.log' | xargs -l tail -20 {} | grep 'yes\|no' | wc -l

This works fine. Now I need the filenames that DO NOT contain "yes" or "no". How do I modify the above command to do that?

Thanks!
 
Old 03-28-2017, 09:24 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 20,230

Rep: Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836
looks like a homework
at first I would suggest you to check the man page of grep about possible options.
 
Old 03-28-2017, 09:24 AM   #3
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 21 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924
grep -v does inverse matching (see man grep)

Oops, sorry pan64, I posted this at the same time as you. I wasn't wanting to overrule your advice.

@ocrts: Welcome to LQ. Is this a homework question or something you're trying to achieve in the real world? Just interested.

Last edited by hydrurga; 03-28-2017 at 09:26 AM.
 
Old 03-28-2017, 09:34 AM   #4
ocrts
LQ Newbie
 
Registered: Mar 2017
Posts: 3

Original Poster
Rep: Reputation: Disabled
Not a homework. Professional. I'm a little old for homework . Just made the text generic. I have checked the man pages. The -v option will just list all the lines that don't contain "yes" or "no". I need the filenames. I tried the -l option on the second grep and send the output to a file (instead of wc -l) to get the filenames of those that do contain "yes" or "no", but I get a broken pipe on the xargs.
 
Old 03-28-2017, 10:07 AM   #5
JeremyBoden
Senior Member
 
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,937

Rep: Reputation: 498Reputation: 498Reputation: 498Reputation: 498Reputation: 498
How 'near' is "near the end of a file"?
 
Old 03-28-2017, 10:10 AM   #6
ocrts
LQ Newbie
 
Registered: Mar 2017
Posts: 3

Original Poster
Rep: Reputation: Disabled
"near" is within the last 20 lines. Since the log files are >10MB each and there are ~10k files, the -tail significantly reduces the execution time of the command as it isn't searching the entirety of each file.
 
Old 03-28-2017, 11:43 AM   #7
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS,Manjaro
Posts: 4,919

Rep: Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400Reputation: 2400
Quote:
Originally Posted by ocrts View Post
I'm using the following command to get the number of *.log files containing "yes" or "no" at the end of the file (the files are large and I'm looking for "yes" or "no" near the end of each file, and each file will contain at most one "yes" OR one "no", but never both):

find ./OUTPUT -type f | grep '\.log' | xargs -l tail -20 {} | grep 'yes\|no' | wc -l

This works fine. Now I need the filenames that DO NOT contain "yes" or "no". How do I modify the above command to do that?

Thanks!
It strikes me that a small script might be better than a one-liner for this purpose. You could easily have a single small script prepare a list of the log files, and split it into three lists: files with yes, files with no, files without either. From that point generating the counts would be trivial.
Something like
Code:
#!/bin/bash
cd OUTPUT
LIST0=`$(ls *.log)
LISTY=""
LISTN=""
LIST1=""
for foo in ${LIST0} ; do
   if tail -20 ${foo} | grep yes
   then
      LISTY="${LISTY ${foo}"
   elif tail -20 ${foo} | grep no
   then
      LISTN="${LISTN} ${foo}"
   else
      LIST1="${LIST1} ${foo}"
   fi
done
COUNTYES=$(echo "$LISTY"|wc -w)
COUNTNO=$(echo "$LISTN" |wc -w)
COUNT0=$(echo "$LIST1"  |wc -w)
# followed by whatever you want to do with these lists and numbers
This is just off the top of my head, possible with syntax errors, and unlikley to be the most efficient way. Also, as written it will be "noisy" because I have not suppressed output of those grep commands in the if statements. Still, it might be enough to give you ideas.
 
Old 03-28-2017, 11:57 AM   #8
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 20,230

Rep: Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836Reputation: 6836
Quote:
Originally Posted by ocrts View Post
find ./OUTPUT -type f | grep '\.log' | xargs -l tail -20 {} | grep 'yes\|no' | wc -l
itself can be simplified by:
Code:
find ./OUTPUT -type f -name '*.log' | xargs -l tail -20 {} | grep -c 'yes\|no'
which is more efficient, but otherwise should do the same thing (not tested).


I would use a more powerful language, but you can also list the files containing yes and no and removes them from the full list (which is actually only one additional grep and a small script)
 
Old 03-28-2017, 01:02 PM   #9
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 2,452

Rep: Reputation: 1061Reputation: 1061Reputation: 1061Reputation: 1061Reputation: 1061Reputation: 1061Reputation: 1061Reputation: 1061
And you can replace xargs -l with find -exec
Code:
find ./OUTPUT -type f -name '*.log' -exec tail -20 {} \; | grep -c -e 'yes' -e 'no'
find ./OUTPUT -type f -name '*.log' -exec tail -20 {} \; | grep -v -c -e 'yes' -e 'no'
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Find & grep - how to return pathes, not grep phrases ? postcd Linux - General 2 11-25-2014 12:43 PM
[SOLVED] how to use cp find and grep together to copy a list of files using find with grep babhijit Linux - Newbie 10 07-03-2013 12:25 PM
using find or grep to find a group of text strings. tkmsr Linux - Newbie 3 03-04-2011 07:02 AM
Find/grep/wc command to find matching files, print filename and word count dbasch Linux - Newbie 10 09-14-2009 05:55 PM
Bash scripting question with find and grep and .bashrc - a multi year problem brfindla Linux - Software 9 08-24-2009 09:54 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:50 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration