How do I grep out lines above till certain pattern is found

blason · 07-06-2019, 12:51 AM

I need to grep out lines matching certain parameter from the file and then above that pattern til the time "#" is found at the beginning of the line.

For example here is the file and I need filter out "00:21:44" and lines above that till # is found.

Any clue?

# CSV: Date,Score,URL,IP #
######################################################################################
"06-07-19 00:23:46","4.5","www.origina-l-diploms.com/wp-includes/fonts/mtbonline/update.htm","198.252.101.174"
"06-07-19 00:23:16","4.5","www.deehhayus.com/sqL/img/home/amazon","202.182.120.120"
"06-07-19 00:22:44","5.9","www.bricktechindia.in/fonts/www.bancoestado.cl/imagenes/comun2008/banca-en-linea-personas.html","43.255.154.40"
"06-07-19 00:22:14","3.7","www.bekenjekleurinstijl.nl/wp-admin/Alibaba/vqcr8bp0gud&lc=1033&id=64855&mkt=en-us&cbcxt=mai&snsc.php?email=nobody@mycraftmail.com","37.46.194.80"
"06-07-19 00:21:44","7.7","www.search-5.com/apx26e/verification/N76C72ED2B98CM9A99BC/qes.php","162.241.130.152"

syg00 · 07-06-2019, 01:05 AM

You can't buffer backwards - you have to buffer reads until they match your criteria. Grep won't do it - maybe awk/perl/ ...
And what on earth does "filter out" mean ? - include or exclude ?.

Turbocapitalist · 07-06-2019, 01:08 AM

I'm not sure of the direction or what is to be removed, but either way it is likely that sed can do the job.

One guess:

Code:

tac file.csv \
| sed -nr -e '/00:21:44/,/^##/p' \
| tac

Can you clarify your requirements a little more?

individual · 07-06-2019, 05:32 AM

Just for fun, here's an AWK one-liner.

Code:

<file.csv awk '{ a[NR]=$0; /^##/ && m=NR+1; /00:21:44/ && n=NR } END { while (m <= n) print a[m++] }'

It prints all lines after ###[...] and stops printing after matching the line with your pattern.

EDIT: If you want to stop altogether after matching your pattern, this one-liner will do that.

Code:

<file.csv awk '{ a[NR]=$0; /^##/ && m=NR+1; if (/00:21:44/) { n=NR; exit } } END { while (m <= n) print a[m++] }'

EDIT2: This is as short as I can get it. An AWK guru could probably come up with a better solution, though.

Code:

<file.csv awk '{a[NR]=$0; /^##/ && m=NR+1; if (/00:21:44/) exit} END {while (m <= NR) print a[m++]}'

astrogeek · 07-06-2019, 04:48 PM

@blason: Please place your code snippets inside [CODE]...[/CODE] tags for better readability. You may type those yourself or click the "#" button in the edit controls.

As others have indicated, a little more clarity about what you are trying to do would be helpful.

In particular, does "filter out" mean delete or extract?

Do you want the # line to be deleted/extracted as well, and which one exactly?

Please review the Site FAQ for guidance in posting your questions and general forum usage. Especially, read the link in that page, How To Ask Questions The Smart Way. The more effort you put into understanding your problem and framing your questions, the better others can help!

blason · 07-07-2019, 11:35 AM

Hi Guys,

Thanks for the input and let me correct or rephrase. filter out as in I need include those lines only and then count backwards till the end of the file where file starts with "#"

Well, eventually I need to find out the recent lines since recent additions get added above it. Hence I was not sure how to include only recent changes or lines which were added yesterday.

blason · 07-15-2019, 05:03 AM

Hey all,

the post by @individual have solved my problem and many thanks for it.

astrogeek · 07-15-2019, 06:08 PM

Good to hear!

If your problem is solved, please use the Thread Tools above top post to mark the thread as solved.

MadeInGermany · 07-16-2019, 03:29 PM

The a[NR]=$0 in the previous solution stores the whole input file in memory.
The following is an improvement attempt

Code:

<file.csv awk '{ a[++n]=$0 } /^#/ { n=0; next } /00:21:44/ { for (m=1; m<=n; m++) print a[m] }'

In sed one can use its hold buffer

Code:

<file.csv sed -n -e 'H; /^#/{s/.*//;x;d;}' -e '/00:21:44/{x;p;}'

BW-userx · 07-16-2019, 08:35 PM

Quote:

Originally Posted by blason

I need to grep out lines matching certain parameter from the file and then above that pattern til the time "#" is found at the beginning of the line.

For example here is the file and I need filter out "00:21:44" and lines above that till # is found.

Any clue?

# CSV: Date,Score,URL,IP #
######################################################################################
"06-07-19 00:23:46","4.5","www.origina-l-diploms.com/wp-includes/fonts/mtbonline/update.htm","198.252.101.174"
"06-07-19 00:23:16","4.5","www.deehhayus.com/sqL/img/home/amazon","202.182.120.120"
"06-07-19 00:22:44","5.9","www.bricktechindia.in/fonts/www.bancoestado.cl/imagenes/comun2008/banca-en-linea-personas.html","43.255.154.40"
"06-07-19 00:22:14","3.7","www.bekenjekleurinstijl.nl/wp-admin/Alibaba/vqcr8bp0gud&lc=1033&id=64855&mkt=en-us&cbcxt=mai&snsc.php?email=nobody@mycraftmail.com","37.46.194.80"
"06-07-19 00:21:44","7.7","www.search-5.com/apx26e/verification/N76C72ED2B98CM9A99BC/qes.php","162.241.130.152"

why cannot it be. I need to start at the # and include, or get the lines until "00:21:44" is reached, by any method that works?

Code:

$ sed -n '/#/,/00:22:44/p' testfile
# CSV: Date,Score,URL,IP #
######################################################################################
"06-07-19 00:23:46","4.5","www.origina-l-diploms.com/wp-includes/fonts/mtbonline/update.htm","198.252.101.174"
"06-07-19 00:23:16","4.5","www.deehhayus.com/sqL/img/home/amazon","202.182.120.120"
"06-07-19 00:22:44","5.9","www.bricktechindia.in/fonts/www.bancoestado.cl/imagenes/comun2008/banca-en-linea-personas.html","43.255.154.40"

MadeInGermany · 07-17-2019, 03:49 AM

After testing, I have corrected my post#9.
I think the O/P wants to print from the last # line before the matched line, up to the matched line.
In contrast, /^#/,/00:22:44/ is from the first # line until the matched line.