Visit Jeremy's Blog.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 04-13-2017, 07:14 AM   #1
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240

Rep: Reputation: 103Reputation: 103
struggling with advanced sed concepts

I'm really at a loss trying to understand sed's advanced functions, such as N, D, P, let alone g, G, H, h, etc. I've read quite a lot, but notions such as pattern space and hold space seem so abstract and weird, I really don't get them. Or maybe I do understand them in theory, but when it comes to using N and, especially, P along with D I simply don't understand what is going on. And I've read about it in three different sources, and all seem to explain it as if it's implicit, but it clearly isn't!

I don't even know where to begin, but I guess I should start with the difference between N and n.

Source lynda: "n prints the content of the pattern space, empties the pattern space, and gets the next line into the pattern space.
N doesn’t print the pattern space and it does not empty it, instead it ends a new line character (\n) to the end of the pattern space, then gets the next line and appends it to the pattern space after the new line. After this has been done, the pattern space, which now contains multiple lines, is available for more editing."

In the second definition, does 'append to the pattern space' mean that N prints the pattern space after it appends the new line?

I don't understand what emptying the pattern space implies exactly either. So n prints the pattern space and then it empties it, but N doesn't empty it. But in both situations lines (be they appended with a newline or not) do show in the final output. So I don't understand what the actual difference is.
Old 04-13-2017, 09:23 AM   #2
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,890
Blog Entries: 13

Rep: Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934Reputation: 4934
What references have you read on this and what experiments have you done with the various options of sed? You cited a great deal of sed command options in your opening statement. I do realize it is a large topic, with a variety of answers, which I can see by just searching for information on "sed N".

Therefore a suggestion is to try each command and evaluate what you find, along with the official sed documentation and then inquire where there is confusion. Much like we see in this question, the OP tried a few things and surmised what they think they were seeing. The top answer cited sed documentation and then went on with a lot of detail to explain the use of N and how it would behave.

I therefore would attack learning these extra details in a similar manner. Try some experiments, re-review the documentation, build up some level of fundamental understanding, and then ask more specific questions.

I also have no idea what "source lynda" represents, which is part of one of your descriptions.
Old 04-13-2017, 09:40 AM   #3
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240

Original Poster
Rep: Reputation: 103Reputation: 103
Lynda is quite a well-known site that offers online courses.
Some other references, if you really need proof:

I've already read that link, too. I'm missing something basic regarding the way sed moves text around, but never mind. I'll probably have to deal with it myself, in time.
Old 04-13-2017, 10:58 AM   #4
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,786

Rep: Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216
Writing a sed script requires a different mindset, and can be quite bewildering if you've never been exposed to the concept of a Turing machine or perhaps taken a look at the Ook! programming language. Once you get the hang of it, it's not hard to understand and is quite powerful (it's Turing-complete), which does not mean that it's easy to use for anything non-trivial.

The sed info document ("info sed" in a terminal) is actually well written and quite complete, though navigating in an info document can be quite a challenge itself at first. Quote from the sed info page: sed programs:: => Execution Cycle::
`sed' maintains two data buffers: the active _pattern_ space, and the auxiliary _hold_ space. Both are initially empty.

`sed' operates by performing the following cycle on each lines of input: first, `sed' reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associated to it: addresses are a kind of condition code, and a command is only executed if the condition is verified before the command is to be executed.

When the end of the script is reached, unless the `-n' option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed. Then the next cycle starts for the next input line.

Unless special commands (like `D') are used, the pattern space is deleted between two cycles. The hold space, on the other hand, keeps its data between cycles (see commands `h', `H', `x', `g', `G' to move data between both buffers).
The "N" command does not, itself, print the pattern space, but if nothing else happens to the pattern buffer before the end of the script is reached, then the whole buffer, now containing two (or more) lines, is going to be printed anyway. (Note that all automatic printing can be disabled by the "-n" option.)

The pattern space usually contains just a single line, but if you have used the "N" command (or other means) to put more than one line in the pattern space, then you can use the "D" or "P" command to delete or print that portion of that buffer up to the first embedded newline.
Old 04-13-2017, 12:36 PM   #5
Senior Member
Registered: Feb 2003
Distribution: debian
Posts: 4,137
Blog Entries: 1

Rep: Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874
About the only one of these I use is g.

$ echo "cat dog cat dog cat dog" | sed 's/cat/dog/'
dog dog cat dog cat dog

$ echo "cat dog cat dog cat dog" | sed 's/cat/dog/g'
dog dog dog dog dog dog
Without the /g it does the first one for the substitution 's///'. With /g it does them ALL.
Old 04-13-2017, 01:24 PM   #6
LQ Guru
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,423
Blog Entries: 3

Rep: Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788
Originally Posted by Shadow_7 View Post
About the only one of these I use is g.
That's a different context. When used elsewhere, g replaces the contents of the pattern space with the contents of the hold space, allowing some juggling of data. Here one line at a time passes through the hold space but swapped out and (unless it is the first line) printed if a pattern is found.

 # print the line immediately before a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{g;1!p;};h'
From :
Old 04-13-2017, 01:52 PM   #7
Registered: Jan 2017
Location: Fremont, CA, USA
Distribution: Trying any&ALL on old/minimal
Posts: 997

Rep: Reputation: 382Reputation: 382Reputation: 382Reputation: 382
You're doing the first step that I do, which is to pick one ... and research it on the web.
But the second step is to try out / experiment with all the possibilities for just that one thing, via web searches, starting from simplest first, until you hit one tiny snippet of code, containing the *MOST* minimal way to demonstrate what you don't understand and then post that in CODE tags. Maybe add key excerpts (quotes) from man/docs or links. Try it: LQ'ers will help you thru getting the knack of doing this process!

Also, step "zero" is to believe! IF you believe you can't, you won't! Research the philosophy: thoughts manifest!

It's great to see studying; just try to keep the LQgurus happy tho
IF you just say: "your darn space shuttle is way too hard for me to pilot", you will just get in-kind emotional responses. If instead, you post fair-good step2above, you will get good-great same in return, and your learning goal will be achieved

Last edited by !!!; 04-13-2017 at 02:19 PM.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
C programming, how to learn advanced concepts Phillyguy Programming 8 09-09-2016 04:32 AM
LXer: Understanding the 5 must have jQuery advanced concepts LXer Syndicated Linux News 0 07-15-2009 01:50 AM
Advanced Sed Question vidyashankara Linux - General 30 07-17-2006 05:25 PM
[sed] "Advanced" sed question(s) G00fy Programming 2 03-20-2006 12:34 AM
Where can I learn more advanced programming concepts? OrganicX Programming 8 03-01-2003 08:08 AM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:32 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration