LinuxQuestions.org - SED vs. AWK showdown

Page 1 of 3

Show 50 post(s) from this thread on one page

LinuxQuestions.org (/questions/)

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - SED vs. AWK showdown (https://www.linuxquestions.org/questions/programming-9/sed-vs-awk-showdown-768298/)

pixellany

11-10-2009 08:06 PM

SED vs. AWK showdown

1 Attachment(s)

Every so often, there's enthusiastic debate** about which is better: SED or AWK. I'd like to see this be a fun thing---maybe even a learning thing.

Here's a problem:
Given a file with a word list, rearrange into sentences which all begin with a keyword. Remove any leading spaces, and extra spaces between words.

Here is a SED solution (keyword = "the"):

Code:

sed -n '${H;x;s/\n/ /g;s/^ *//;s/ \+/ /g;s/ the/\nthe/g;p};H' words.txt

And the file I used is attached.

The logic used:
while not at the end of file, append each line to the hold register.
when EOF is reached, also append that last line, then bring the hold register into the working register. Now we have the whole file in the register. Then:
replace all linefeeds with spaces
replace all spaces at the beginning
replace all multiple spaces with just one
insert line breaks before all instances of the keyword (the), except at the beginning, where there is no space.
print the result

So:
Is there an AWK solution which is:
faster?
less code?

**many of which I lost......;)

ghostdog74

11-10-2009 09:24 PM

less code doesn't mean its always legible or understandable.

Code:

awk 'NR>1&&$1=="the"{print ""}{ printf "%s ",$0}' words.txt

because printf doesn't insert a newline unless you tell it to, the output you see will be lines concat together, until the key word "the" is found, then print a newline. this is much more simpler to understand than the bunch of sed secret code :)

tuxdev

11-10-2009 09:26 PM

Neither sed or awk, but..

Code:

#!/bin/bash

while read -r LINE ; do

  BUFFER="$BUFFER $LINE"

done

BUFFER="${BUFFER//  / }"

BUFFER="${BUFFER:1}"

if [[ "${BUFFER:$((${#BUFFER}-1))}" == " " ]] ; then

  BUFFER="${BUFFER:0:$((${#BUFFER}-1))}"

fi

BUFFER="${BUFFER// the/$'\n'the}"

echo "$BUFFER"

Isn't parameter expansion fun?

ghostdog74

11-10-2009 09:29 PM

its a small issue. you forget the file name

tuxdev

11-10-2009 09:32 PM

It's stdin, of course! Like any good "filter" type script ought to behave.

ghostdog74

11-10-2009 09:41 PM

Quote:

Originally Posted by tuxdev (Post 3752376)

It's stdin, of course! Like any good "filter" type script ought to behave.

i take it that you mean input redirection. that's fine.

regarding your script, if you are going to buffer every line of the file before processing, its going to be very slow when the file sizes are huge.

pixellany

11-10-2009 09:49 PM

Quote:

Originally Posted by ghostdog74 (Post 3752366)

less code doesn't mean its always legible or understandable.

Code:

awk 'NR>1&&/the/{print ""}{ printf "%s ",$0}' words.txt

Did you run a speed test?

I'll be happy to do it, but later. Now I must watch "V"....

pixellany

11-10-2009 09:50 PM

Quote:

Originally Posted by tuxdev (Post 3752367)

Neither sed or awk, but..

Code:

#!/bin/bash

while read -r LINE ; do

  BUFFER="$BUFFER $LINE"

done

BUFFER="${BUFFER//  / }"

BUFFER="${BUFFER:1}"

if [[ "${BUFFER:$((${#BUFFER}-1))}" == " " ]] ; then

  BUFFER="${BUFFER:0:$((${#BUFFER}-1))}"

fi

BUFFER="${BUFFER// the/$'\n'the}"

echo "$BUFFER"

Isn't parameter expansion fun?

That just might be more obfuscated than my SED solution.....;)

ta0kira

11-10-2009 10:01 PM

Code:

echo `cat words.txt` | sed 's/ the/\nthe/g'

Kevin Barry

ghostdog74

11-10-2009 10:02 PM

Quote:

Originally Posted by pixellany (Post 3752390)

Did you run a speed test?

no, please do. I suspect it will be slower, since I am calling printf every time. Might be better to save them in memory before printing at the right time. But well, i prefer readability more than speed concerns.

NB: please change your sed to take care of things like

Code:

the

thesis is

done

ghostdog74

11-10-2009 10:05 PM

Quote:

Originally Posted by ta0kira (Post 3752401)

Code:

echo `cat words.txt` | sed 's/ the/\nthe/g'

Kevin Barry

ok on small files, but will choke on big files.

ta0kira

11-10-2009 10:10 PM

Quote:

Originally Posted by ghostdog74 (Post 3752403)

ok on small files, but will choke on big files.

It was more of a joke, actually. I think it's funny how people often try to get everything done in one call to one command when it's almost never necessary, because it will probably just go into a script, anyway.
Kevin Barry

tuxdev

11-10-2009 10:13 PM

The bigger problem is if words.txt ends with ";rm -rf ~"

ta0kira

11-10-2009 10:16 PM

Quote:

Originally Posted by tuxdev (Post 3752411)

The bigger problem is if words.txt ends with ";rm -rf ~"

Only if one precedes the line with eval...
Kevin Barry

ghostdog74

11-10-2009 10:19 PM

Quote:

Originally Posted by ta0kira (Post 3752414)

Only if one precedes the line by eval...
Kevin Barry

that's right. we are not "eval"ing each line as it go. just reformatting text.

All times are GMT -5. The time now is 06:20 AM.

Page 1 of 3

Show 50 post(s) from this thread on one page