Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
|
|
03-09-2017, 10:23 AM
|
#1
|
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240
Rep:
|
Next in sed \n vs \n*
cat script_test
cat operator2
Code:
Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.
Look in the Owner and Operator Guide shipped with your system.
Two manuals are provided including the Owner and
Operator Guide and the User Guide.
The Owner and Operator Guide is shipped with your system.
sed -f script_test operator2
Code:
Consult Section 3.1 in the Owner and Operator Guide for a description of the tape drives
available on your system.
Look in the Owner and Operator Guide shipped with your system.
Two manuals are provided including the Owner and Operator Guide and the User Guide.
The Owner and Operator Guide is shipped with your system.
But if I change script_test to:
Then when I run sed -f script_test operator2:
Code:
Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.
Look in the Owner and Operator Guide shipped with your system.
Two manuals are provided including the Owner and
Operator Guide and the User Guide.
The Owner and Operator Guide is shipped with your system.
This behaviour doesn't make much sense to me. The point was, I guess, to see if I can convert it all to one line (I don't need another solution other than sed, I simply want to understand how sed works in this case).
Last edited by vincix; 03-09-2017 at 10:27 AM.
|
|
|
03-09-2017, 11:31 AM
|
#2
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,794
|
The regex " *\n*" (zero or more spaces followed by zero of more newlines) matches the null string at the beginning of each line. The command then replaces that null string with a space. What you would want is " *\n\+" (zero or more spaces followed by one or more newlines). But, since you have only two lines in the pattern space (just one "N" command), there can never be more than one embedded newline to replace. If you want to convert the whole file to one line, you first have to append all the lines and then make your substitution global.
(No, I don't feel like working out the details of doing that just now.)
|
|
1 members found this post helpful.
|
03-10-2017, 04:25 AM
|
#3
|
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240
Original Poster
Rep:
|
I don't understand the 'two lines in the pattern space' part and how this is determined by the fact that there's only one "N". Can you expand on that?
|
|
|
03-10-2017, 05:30 AM
|
#4
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,010
|
N is: append the next line of input into the pattern space.
so if there was only one N you appended only one line to the pattern space, therefore you will have (only) two lines in the pattern space.
|
|
|
03-10-2017, 05:33 AM
|
#5
|
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240
Original Poster
Rep:
|
So basically what N does is to make the newline manipulable so that you can change \n however you wish to, right? Normally, \n is ignored by sed, is that it?
|
|
|
03-10-2017, 06:24 AM
|
#6
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,010
|
no, "normally" \n is not part of the line, therefore will not be "available" in the pattern space. The command N will join lines (= append the next line of input into the pattern space) and will add a \n in between.
|
|
1 members found this post helpful.
|
03-10-2017, 07:23 AM
|
#7
|
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240
Original Poster
Rep:
|
I've been struggling to understand this goddamn mechanism for some time now. And I read several definitions and I simply couldn't imagine it. I might be midly dyslexic or something, as I like to believe that I've understood more difficult things (linux related and so on). I needed a visual cue or some sorts. Anyway, your explanation made it much clearer now I'll tinker a little bit with it and I'll probably get back with some questions regarding the main subject.
|
|
|
03-10-2017, 10:48 AM
|
#8
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,010
|
ok, fine, we are waiting for the next one....
|
|
|
03-11-2017, 04:41 AM
|
#9
|
Member
Registered: Jun 2016
Distribution: any&all, in VBox; Ol'UnixCLI; NO GUI resources
Posts: 999
|
man fmt ? YES! That's IT!!! -w
I spent several DAYS trying [successfully!] to 'grok' N \n !
Here's something: (t&b=goto :a OMy Gosh! [ G-rated!])
seq 6 | sed -e :a -e '$!N; s/\n/ /; ta' ### OR: man > fmt< ? YES!
seq 6 | sed ':a;N;$!ba;s/\n/ /g' ### here with a 1044rating wow!
Best wishes! 'Take your time!' (I do, savoring these puzzles!)
Last edited by Jjanel; 03-11-2017 at 05:15 AM.
|
|
|
03-11-2017, 05:10 AM
|
#10
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,010
|
actually sed handles \n as special char, so if you want to avoid that better to use another tool (for example perl, awk, python, tr, whatever).
|
|
1 members found this post helpful.
|
03-11-2017, 09:54 AM
|
#11
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,794
|
Quote:
Originally Posted by Jjanel
I spent several DAYS trying [successfully!] to 'grok' N \n !
|
You're just going about it the hard way. Joining all the lines in sed is actually pretty easy:
Code:
sed -n 'H;${x;s/\n\+/ /g;p}'
Translation: Append each input line to the hold space. On the last line, exchange the hold space and the pattern space, then replace every occurrence of one or more newline characters with a single space and print the result.
|
|
1 members found this post helpful.
|
03-11-2017, 02:28 PM
|
#12
|
Senior Member
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,240
Original Poster
Rep:
|
I'm happy that this thread is the cause of such fruitful endeavour, but I am going to 'take it down a notch' and ask you something much more trivial.
script_test:
operator3:
Code:
first line
second line
third line
fourth line
fifth line
secs
sieben
acht
neun
zehn
sed -f script_test operator3:
Code:
[*]first line
second line
[*]third line
fourth line
[*]fifth line
secs
[*]sieben
acht
[*]neun
zehn
I understand that if I use it with a simple \n (without *), it takes every two lines and joins them with whatever you tell sed to substitute \n with. So in this case it matches the null character at the beginning of every two lines. But why does it do this every two lines exactly?
For instance, what does the second line have to do with the first line, in order for sed to ignore it and not prefix a "[*]"? I don't understand the exact logic (I hope I was articulate enough).
|
|
|
03-11-2017, 05:42 PM
|
#13
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,794
|
It's not performing the test on every line. It matches null string at the start of the pattern buffer, which, due to the "N" command, contains more than a single line. The newline character actually isn't special to sed, except insofar as it delimits, and is stripped from, the lines as read, and gets inserted as a separator when lines are appended to the pattern buffer or hold buffer.
|
|
|
03-11-2017, 05:48 PM
|
#14
|
LQ Veteran
Registered: May 2008
Posts: 7,053
|
Lets dry run it.
Say we have the input file:
Code:
one
two
three
four
five
six
And we're running sed -e 'N;s/\n*/(*)/' (note: I had to change the square brackets because they were messing up the bbcode in the forum.)
So, sed starts and implicitly reads the next line of the input file into the pattern buffer.
The buffer will now contain: ' one' (note: no terminating '\n' on the buffer)
Next, it does the ' N' operation, which appends the next line of input onto the end of the buffer. The buffer now contains: ' one\ntwo' (sed inserts a '\n' between the two lines in the buffer, but there's still no terminating '\n' at the end of the buffer).
Next, it does the ' s/\n*/(*)/' which instructs it to swap the first occurrence of zero or more '\n' characters to the literal string ' (*)'. The first occurrence of zero '\n' characters is at the beginning of the line, so the buffer now contains: ' (*)one\ntwo'
It now gets to the end of the script and does an implicit print of the buffer. It prints ' (*)one\ntwo' to the terminal, which displays as
It now clears the input buffer, does another implicit read of the next line, loading 'three' into the buffer, and starts again from the 'N' operation, just as before.
Rinse and repeat.
I suspect part of your confusion is stemming from the fact you are confusing the meaning of '*' in a regex, with the meaning of '*' in a shell glob. They are quite different.
Some examples:
Code:
test@ws1:~$ cat /tmp/sed.in
one
two
three
four
five
six
test@ws1:~$ sed -e 'N;s/\n*/(*)/' /tmp/sed.in
(*)one
two
(*)three
four
(*)five
six
test@ws1:~$ sed -e 'N;s/\n.*/(*)/' /tmp/sed.in
one(*)
three(*)
five(*)
test@ws1:~$ sed -e 'N;s/.*\n/(*)/' /tmp/sed.in
(*)two
(*)four
(*)six
test@ws1:~$
I hope that clears it up for you.
Last edited by GazL; 03-11-2017 at 05:54 PM.
|
|
1 members found this post helpful.
|
03-11-2017, 06:03 PM
|
#15
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,794
|
Quote:
Originally Posted by GazL
(note: I had to change the square brackets because they were messing up the bbcode in the forum.)
|
The magic incantation to prevent that when not using [CODE] ... [/CODE] tags is "[NOPARSE] ... [/NOPARSE]".
|
|
1 members found this post helpful.
|
All times are GMT -5. The time now is 03:18 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|