LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 08-02-2021, 05:40 PM   #1
sharky
Member
 
Registered: Oct 2002
Posts: 569

Rep: Reputation: 84
sed or awk command to filter lines from RE to next blank line.


Is there a grep, sed or awk one liner that would remove lines from a regular expression to the next blank line?

Looking at a huge log file that list errors. The log file prints the type of error and then some details. The number of lines in the details can vary but there is always a blank line after.

Example:
Quote:
type1 error
details
details

type2 error
more details
other stuff
details

type2 error
Other stuff
stupid stuff

type3 error
details
Result after filter to remove type2 error and it's details:
Quote:
type1 error
details
details

type3 error
details
 
Old 08-02-2021, 05:43 PM   #2
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,688

Rep: Reputation: Disabled
Code:
sed '/^type2/,/^$/d'
Code:
awk '/^type2/,/^$/{next}1'
Things get a bit more complicated with grep. There are lots of grep-like tools, some of them better suited to the task on hand than others. Broadly speaking, I'd divide tools I deem "well-suited" in this context into three categories. Sadly, GNU grep (or any traditional grep implementation) is in no one of these.

First, tools that can operate on paragraphs rather than on lines of text: greple (written in Perl) and paragrep (written in Python). Their only disadvantage is that they are lesser known and rarely provided by Linux distributions, so you may have to install them from CPAN and PyPI, respectively.
Code:
greple -p -v type2 .
Code:
paragrep -v type2
Then, there's sgrep which is a category in and by itself. It's an old tool, last updated in 2004, but still surprisingly useful at times. Beware though that despite its name, sgrep doesn't understand regular expressions. So, "type2" in the example below is a literal string.
Code:
sgrep -Nao '' '"type2".."\n\n"'
Lastly, there are tools that 1) allow matching across the lines, AND 2) allow -v/--invert-match to be used while doing so. For example, pcregrep that comes with the PCRE library, or ripgrep:
Code:
pcre2grep -Mvx 'type2(?s).*?^'
Code:
rg -Uv '^type2(.+\n)+\n'

Last edited by shruggy; 08-05-2021 at 02:15 AM.
 
Old 08-02-2021, 07:02 PM   #3
sharky
Member
 
Registered: Oct 2002
Posts: 569

Original Poster
Rep: Reputation: 84
Hi Shruggy,

Thanks for the quick reply. Unfortunately our systems do no have pcre2grep, rg, or sgrep installed and I'm not the sysadmin so I can't install them.

The sed and awk commands did not filter out anything. It's probably my fault. I implied that the RE would be at the beginning of the line and that is not necessarily the case.
 
Old 08-02-2021, 08:00 PM   #4
computersavvy
Senior Member
 
Registered: Aug 2016
Posts: 3,345

Rep: Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484
So if you want a detailed answer you need to provide detailed info on what should be the target to be filtered. We can only work on the info you provide and our suggestions will only be as accurate as is the info we have to work with.

Please post an exact input line/ group of lines that the code should work on.
 
Old 08-02-2021, 09:24 PM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,161

Rep: Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125
Quote:
Originally Posted by sharky View Post
The sed and awk commands did not filter out anything. It's probably my fault. I implied that the RE would be at the beginning of the line and that is not necessarily the case.
No probably about it.
shruggy went above and beyond IMHO - how about you make some effort yourself. This should be a good opportunity to learn. 99% of the answer has been given to you. I would suggest sed would be simplest, with least to have to learn along the way.
 
Old 08-03-2021, 01:42 AM   #6
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,688

Rep: Reputation: Disabled
And you were given practically the same sed solution in one of your previous threads.
 
Old 08-04-2021, 03:48 PM   #7
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 2,844

Rep: Reputation: 1222Reputation: 1222Reputation: 1222Reputation: 1222Reputation: 1222Reputation: 1222Reputation: 1222Reputation: 1222Reputation: 1222
The following does not need a last end condition (empty line).
Code:
awk '/type2 error/ {del=1} del==0; NF==0 {del=0}'
 
1 members found this post helpful.
  


Reply

Tags
awk, range, sed, sgrep



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
filter source line based on results line in log using awk and sed samanp Programming 5 04-06-2011 09:42 AM
sed/awk: Three consecutive blank lines in a file, how to delete two of them? recomboDNA Programming 8 06-17-2010 09:50 AM
awk command line: blank line record sep, new line field sep robertmarkbram Programming 4 02-21-2010 05:25 AM
grab the line below a blank line and the line above the next blank line awk or perl? Pantomime Linux - General 7 06-26-2008 08:13 AM
awk/gawk/sed - read lines from file1, comment out or delete matching lines in file2 rascal84 Linux - General 1 05-24-2006 09:19 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:18 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration