LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 09-08-2011, 02:53 AM   #1
jgombos
Member
 
Registered: Jul 2003
Posts: 256

Rep: Reputation: 32
sed loop gives unexpected results


Works fine without a loop:
Code:
$ echo -e '1\n2\n3\n4' | sed -ne '/1/,/3/p'
1
2
3
The same output is expected with a loop added as follows:
Code:
echo -e '1\n2\n3\n4' | sed -ne '/1/,/3/{;:loop;N;/[^3]/b loop;p;}'
But there is no output. What am I missing?
 
Old 09-08-2011, 03:13 AM   #2
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Maybe you could explain what you think should be printed? Seems to be working correctly to me.
 
Old 09-08-2011, 03:47 AM   #3
jgombos
Member
 
Registered: Jul 2003
Posts: 256

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by grail View Post
Maybe you could explain what you think should be printed? Seems to be working correctly to me.
The loop eventually exits. When it exits, there is a "p" instruction, at which point the pattern space ("1\n2\n3") must be printed. Yet nothing prints.

*edit*
I solved my problem. A better example would have been:
Code:
echo -e '1\n2\n3\n4' | sed -ne '/1/{;:loop;N;/^[^3]*$/b loop;p;}'
Although I'm still not clear on why the last instruction is skipped.

Last edited by jgombos; 09-08-2011 at 04:10 AM.
 
Old 09-08-2011, 05:26 AM   #4
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Quote:
Although I'm still not clear on why the last instruction is skipped.
Maybe you could explain what you mean here? If by the last instruction you are referring to 'p', if it were skipped you would see nothing at all as you have used '-n'.

The reason I asked you to explain is so you could tell us what you think is happening in the script?
Your second is not a better example but rather one that has worked with the expression you have built. The idea is to build the expression to fit the data and not
the other way around (as the real world would have the data that needs to be manipulated)

Code:
echo -e '1\n2\n3\n4' | sed -ne '/1/,/3/{;:loop;N;/[^3]/b loop;p;}'
-n :- no output unless told to print (won't worry about -e as it serves no purpose)

/1/,/3/ :- only perform the following actions for lines in this range (ie. ignore the row with a 4 on it) { 1 is first number in pattern buffer }

:loop :- start of loop

N :- get the next line into the pattern buffer {pattern now holds '1\n2'}

/[^3]/b loop :- if pattern space contains something which is not a 3 redo loop (this true as pattern space does have a non-3 value in it, so go to start of loop)

N :- get the next line into the pattern buffer {pattern now holds '1\n2\n3'}

So your loop chews up all your data and by the time you get to 'p' there is nothing left to print

Whilst your second example works it is still not doing what you think. It continues the loop until the pattern buffer holds :- '1\n2\n3' and as a 3 is at the end which breaks
the loop from being true it then prints.
 
1 members found this post helpful.
Old 09-08-2011, 07:01 AM   #5
jgombos
Member
 
Registered: Jul 2003
Posts: 256

Original Poster
Rep: Reputation: 32
grail- I appreciate your detailed reply. I struggle with this comment:

Quote:
Originally Posted by grail View Post
N :- get the next line into the pattern buffer {pattern now holds '1\n2\n3'}

So your loop chews up all your data and by the time you get to 'p' there is nothing left to print
I can see that all the data is consumed by the pattern buffer. However, I expect the pattern buffer to print when the loop exits and "p" executes. E.g. I don't see why the "p" is skipped in this case:
Code:
echo -e '1\n2\n' | sed -n '/1/{;:loop;N;/./b loop;p;}'
but not in this case:

Code:
echo -e '1\n2\n' | sed -n '/1/{;:loop;N;/^[^2]*$/b loop;p;}'
1
2
In both those cases the loop terminates with something (everything?) consumed by the buffer, so I think "p" should print something in both cases.

Or is the loop infinite in the first case, and sed simply times out without sending a failing exit status?

Last edited by jgombos; 09-08-2011 at 07:10 AM.
 
Old 09-08-2011, 07:54 AM   #6
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Yes the first gets to the end of the data and never meets the criteria set down so as it has reached the end of the file when it gets to the 'p' it has already
exhausted all the data so there is nothing to print. Whereas your second example gets to a situation where it is true that there is now a '2' before the end of the line
so it leaves the loop and progresses to the next instruction, ie. 'p'
 
Old 09-08-2011, 11:39 AM   #7
ta0kira
Senior Member
 
Registered: Sep 2004
Distribution: FreeBSD 9.1, Kubuntu 12.10
Posts: 3,078

Rep: Reputation: Disabled
What about this:
Code:
echo -e '1\n2\n3\n4' | sed -e ':loop;N;/3/!b loop;!d;q'
Kevin Barry
 
Old 09-08-2011, 01:12 PM   #8
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Quote:
Originally Posted by ta0kira View Post
What about this:
Code:
echo -e '1\n2\n3\n4' | sed -e ':loop;N;/3/!b loop;!d;q'
Kevin Barry
Hi,

can you explain why the marked part should be there?
 
Old 09-08-2011, 09:31 PM   #9
ta0kira
Senior Member
 
Registered: Sep 2004
Distribution: FreeBSD 9.1, Kubuntu 12.10
Posts: 3,078

Rep: Reputation: Disabled
Quote:
Originally Posted by crts View Post
Hi,

can you explain why the marked part should be there?
It probably shouldn't be. I think I added it as a replacement for p when I removed -n and didn't really think about it.
Kevin Barry
 
Old 09-09-2011, 07:08 AM   #10
Reuti
Senior Member
 
Registered: Dec 2004
Location: Marburg, Germany
Distribution: openSUSE 13.1
Posts: 1,320

Rep: Reputation: 252Reputation: 252Reputation: 252
Quote:
Originally Posted by grail View Post
...
N :- get the next line into the pattern buffer {pattern now holds '1\n2\n3'}

So your loop chews up all your data and by the time you get to 'p' there is nothing left to print
...
Does it really get to the 'p'? I would phrase it more like: if 'N' canít read anything, execution will be stopped. When I vary the initial script:
Code:
$ echo -e '1\n2\n3\n4\n5' | sed -ne '/1/,/3/{:loop;N;p;/[^3]/b loop;p;}'
1
2
1
2
3
1
2
3
4
1
2
3
4
5
As it reads inside the loop, the outside range specification /1/,/3/ isnít used at all. It would only be used if the {} exits and a new record needs to be feed into the script.
 
1 members found this post helpful.
Old 09-09-2011, 07:56 AM   #11
jgombos
Member
 
Registered: Jul 2003
Posts: 256

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by Reuti View Post
I would phrase it more like: if 'N' cant read anything, execution will be stopped.
Bingo. That was ultimately the source of my problems (or more accurately: it was the thing preventing me from discovering my problems). Sed croaks when "N" runs out of food. This is mentioned on page 110 of "sed & awk" o'reilly 2nd ed. I suspect it's a crude protection from infinite loops. It's too bad this is not treated as an error condition. Exit status is normal.

Last edited by jgombos; 09-09-2011 at 08:08 AM.
 
Old 09-09-2011, 08:26 AM   #12
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
It is treated as a normal exit as it reached the end of data but none of the expressions terms were met to result in output. This is not an error.
 
Old 09-09-2011, 11:50 AM   #13
crts
Senior Member
 
Registered: Jan 2010
Posts: 1,604

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Quote:
Originally Posted by grail View Post
It is treated as a normal exit as it reached the end of data but none of the expressions terms were met to result in output. This is not an error.
That is very true. Just because the last line has been read it does not mean that you always want to print it.

If the further execution of your script depends on some specific modifications that sed has done or not then you can set the exit status manually; this requires GNU sed, though.

Building on Kevin's example:
Code:
$ echo -e '1\n2\n3\n4' | sed -e ':loop;N;${/3/q;Q 99};/3/!b loop;q'
1
2
3
$ echo $?
0
$ echo -e '1\n2\n3' | sed -e ':loop;N;${/3/q;Q 99};/3/!b loop;q'
1
2
3
$ echo $?
0
$ echo -e '1\n2\n4' | sed -e ':loop;N;${/3/q;Q 99};/3/!b loop;q'
$ echo $?
99
 
1 members found this post helpful.
Old 09-10-2011, 01:33 AM   #14
jgombos
Member
 
Registered: Jul 2003
Posts: 256

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by grail View Post
It is treated as a normal exit as it reached the end of data but none of the expressions terms were met to result in output. This is not an error.
Correct, it's not an error -- that's actually the problem.

It's indeed a poor language choice to take a construct designed for data manipulation, and also give it flow control. It would be like creating a C "scanf" function that behaves like a goto statement in some obscure circumstances. And worse, these are circumstances that arise from inadvertent programming.

In fact having been burnt by this strange behavior and now knowing full well how "N" behaves, I would never deliberately use the N command for flow control. It would be an abuse; it would confuse other readers with poorly constructed code.

Sure, goto statements are appropriate for a quick and dirty language like sed, but this is effectively a goto nowhere -- that is, a goto without a destination label, and the goto statement itself is hiding within a construct that's used to move data.

Last edited by jgombos; 09-10-2011 at 01:52 AM.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Unexpected results: Difference in Perl Script Behavior between manual and cron runs EnderX Programming 3 12-11-2010 12:42 AM
find -mtime gives unexpected results amchargue Linux - Newbie 3 08-12-2010 05:19 PM
[SOLVED] Sort generates unexpected results danielbmartin Linux - Newbie 5 06-08-2010 03:39 AM
find command displays unexpected results helptonewbie Linux - Newbie 5 08-12-2008 03:25 AM
Unexpected result in while do done loop reading file steven.c.banks Linux - General 2 05-01-2008 01:32 PM


All times are GMT -5. The time now is 08:47 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration