SED vs. AWK showdown

pixellany · 11-10-2009, 08:06 PM

Every so often, there's enthusiastic debate** about which is better: SED or AWK. I'd like to see this be a fun thing---maybe even a learning thing.

Here's a problem:
Given a file with a word list, rearrange into sentences which all begin with a keyword. Remove any leading spaces, and extra spaces between words.

Here is a SED solution (keyword = "the"):

Code:

sed -n '${H;x;s/\n/ /g;s/^ *//;s/ \+/ /g;s/ the/\nthe/g;p};H' words.txt

And the file I used is attached.

The logic used:
while not at the end of file, append each line to the hold register.
when EOF is reached, also append that last line, then bring the hold register into the working register. Now we have the whole file in the register. Then:
replace all linefeeds with spaces
replace all spaces at the beginning
replace all multiple spaces with just one
insert line breaks before all instances of the keyword (the), except at the beginning, where there is no space.
print the result

So:
Is there an AWK solution which is:
faster?
less code?

**many of which I lost......

ghostdog74 · 11-10-2009, 09:24 PM

less code doesn't mean its always legible or understandable.

Code:

awk 'NR>1&&$1=="the"{print ""}{ printf "%s ",$0}' words.txt

because printf doesn't insert a newline unless you tell it to, the output you see will be lines concat together, until the key word "the" is found, then print a newline. this is much more simpler to understand than the bunch of sed secret code

tuxdev · 11-10-2009, 09:26 PM

Neither sed or awk, but..

Code:

#!/bin/bash
while read -r LINE ; do
   BUFFER="$BUFFER $LINE"
done
BUFFER="${BUFFER//  / }"
BUFFER="${BUFFER:1}"
if [[ "${BUFFER:$((${#BUFFER}-1))}" == " " ]] ; then
   BUFFER="${BUFFER:0:$((${#BUFFER}-1))}"
fi
BUFFER="${BUFFER// the/$'\n'the}"
echo "$BUFFER"

Isn't parameter expansion fun?

ghostdog74 · 11-10-2009, 09:29 PM

its a small issue. you forget the file name

tuxdev · 11-10-2009, 09:32 PM

It's stdin, of course! Like any good "filter" type script ought to behave.

ghostdog74 · 11-10-2009, 09:41 PM

Quote:

Originally Posted by tuxdev

It's stdin, of course! Like any good "filter" type script ought to behave.

i take it that you mean input redirection. that's fine.

regarding your script, if you are going to buffer every line of the file before processing, its going to be very slow when the file sizes are huge.

pixellany · 11-10-2009, 09:49 PM

Quote:

Originally Posted by ghostdog74

less code doesn't mean its always legible or understandable.

Code:

awk 'NR>1&&/the/{print ""}{ printf "%s ",$0}' words.txt

because printf doesn't insert a newline unless you tell it to, the output you see will be lines concat together, until the key word "the" is found, then print a newline. this is much more simpler to understand than the bunch of sed secret code

Did you run a speed test?

I'll be happy to do it, but later. Now I must watch "V"....

pixellany · 11-10-2009, 09:50 PM

Quote:

Originally Posted by tuxdev

Neither sed or awk, but..

Code:

#!/bin/bash
while read -r LINE ; do
   BUFFER="$BUFFER $LINE"
done
BUFFER="${BUFFER//  / }"
BUFFER="${BUFFER:1}"
if [[ "${BUFFER:$((${#BUFFER}-1))}" == " " ]] ; then
   BUFFER="${BUFFER:0:$((${#BUFFER}-1))}"
fi
BUFFER="${BUFFER// the/$'\n'the}"
echo "$BUFFER"

Isn't parameter expansion fun?

That just might be more obfuscated than my SED solution.....

ta0kira · 11-10-2009, 10:01 PM

Code:

echo `cat words.txt` | sed 's/ the/\nthe/g'

Kevin Barry

ghostdog74 · 11-10-2009, 10:02 PM

Quote:

Originally Posted by pixellany

Did you run a speed test?

no, please do. I suspect it will be slower, since I am calling printf every time. Might be better to save them in memory before printing at the right time. But well, i prefer readability more than speed concerns.

NB: please change your sed to take care of things like

Code:

the
thesis is
done

ghostdog74 · 11-10-2009, 10:05 PM

Quote:

Originally Posted by ta0kira

Code:

echo `cat words.txt` | sed 's/ the/\nthe/g'

Kevin Barry

ok on small files, but will choke on big files.

ta0kira · 11-10-2009, 10:10 PM

Quote:

Originally Posted by ghostdog74

ok on small files, but will choke on big files.

It was more of a joke, actually. I think it's funny how people often try to get everything done in one call to one command when it's almost never necessary, because it will probably just go into a script, anyway.
Kevin Barry

tuxdev · 11-10-2009, 10:13 PM

The bigger problem is if words.txt ends with ";rm -rf ~"

ta0kira · 11-10-2009, 10:16 PM

Quote:

Originally Posted by tuxdev

The bigger problem is if words.txt ends with ";rm -rf ~"

Only if one precedes the line with eval...
Kevin Barry

ghostdog74 · 11-10-2009, 10:19 PM

Quote:

Originally Posted by ta0kira

Only if one precedes the line by eval...
Kevin Barry

that's right. we are not "eval"ing each line as it go. just reformatting text.