[SOLVED] using the special RE codes \1,...,\9 in bash sed

ghantauke · 11-30-2010, 09:51 AM

I'm using bash. Assume I have the following text in file test.

Code:

Hey there. This is a sample text.
Testing *cough*... Very harsh testing don't.

Lets say I want to replace each word starting with s and ending with e to be replaced by a 2 at the front. I use the following code.

Code:

sed 's/ s.*e / 2\1 /g' test

I get the error

Code:

sed: -e expression #1, char 16: invalid reference \1 on `s' command's RHS

Seems like i can't use \1 to return the first pattern matched. I can do it in perl but not in bash. Need help. Thanks in advance.

GrapefruiTgirl · 11-30-2010, 09:58 AM

Things being referenced with back-references must be found on the LHS inside \(...\) in order to be referenced. Here's an example:

Code:

root@reactor: echo "word1 word2 word3" | sed 's/\(w.*1\)\(.*\)/Q\1Q\2/'
Qword1Q word2 word3
root@reactor:

Hopefully you are not color blind

- I have colored the above. The red stuff is the first of two \(...\) I have used. The second refers to the remainder of the input line, after the match. So, on the RHS of the expression, \1 refers to the red stuff on the LHS, and \2 refers to the green stuff.

Note that this is just a simple example - play around a bit with it in your terminal to see what you can discover.

GrapefruiTgirl · 11-30-2010, 10:03 AM

PS - Have a look at this page for a very good, more comprehensive (and more convoluted perhaps) instruction about back-references, and browse the remainder of that site for lots of good regex instruction:

http://www.regular-expressions.info/brackets.html
http://www.regular-expressions.info/tutorial.html

ghantauke · 11-30-2010, 10:03 AM

Quote:

Originally Posted by GrapefruiTgirl

Things being referenced with back-references must be found on the LHS inside \(...\) in order to be referenced. Here's an example:

Code:

root@reactor: echo "word1 word2 word3" | sed 's/\(w.*1\)\(.*\)/Q\1Q\2/'
Qword1Q word2 word3
root@reactor:

Hopefully you are not color blind

- I have colored the above. The red stuff is the first of two \(...\) I have used. The second refers to the remainder of the input line, after the match. So, on the RHS of the expression, \1 refers to the red stuff on the LHS, and \2 refers to the green stuff.

Note that this is just a simple example - play around a bit with it in your terminal to see what you can discover.

Oh bloody 'ell. Forgot the brackets haha. Thanks for reminding me that.

ghantauke · 11-30-2010, 10:08 AM

Quote:

Originally Posted by GrapefruiTgirl

Things being referenced with back-references must be found on the LHS inside \(...\) in order to be referenced. Here's an example:

Code:

root@reactor: echo "word1 word2 word3" | sed 's/\(w.*1\)\(.*\)/Q\1Q\2/'
Qword1Q word2 word3
root@reactor:

Hopefully you are not color blind

- I have colored the above. The red stuff is the first of two \(...\) I have used. The second refers to the remainder of the input line, after the match. So, on the RHS of the expression, \1 refers to the red stuff on the LHS, and \2 refers to the green stuff.

Note that this is just a simple example - play around a bit with it in your terminal to see what you can discover.

Don't think it matters anymore but I think you should remove the backslash before the brackets in \(w.*1\) and \(.*\). Anyway thanks again

GrapefruiTgirl · 11-30-2010, 10:13 AM

Code:

root@reactor: echo "word1 word2 word3" | sed 's/(w.*1)(.*)/Q\1Q\2/'
sed: -e expression #1, char 20: invalid reference \2 on `s' command's RHS
root@reactor: echo "word1 word2 word3" | sed 's/(w.*1)(.*)/Q\1Q\2/'
sed: -e expression #1, char 20: invalid reference \2 on `s' command's RHS
root@reactor: echo "word1 word2 word3" | sed 's/(w.*1)(.*)/Q\1Q/'
sed: -e expression #1, char 18: invalid reference \1 on `s' command's RHS
root@reactor:

It seems to matter here, in this case. But, it will depend on the tool, the regex style, and the quoting in use, whether you need to escape, or not, or double-escape, certain things in regex's. I tend to always put escapes in situations like this as a habit - usually there's no harm. Maybe if I actually sat down and studied every possible combination of quotes and regex styles, I might really learn when to escape and when not to, but probably I won't invest the time.

I'll just keep escaping things, and if testing poves it to be bad in a certain case, I fix it then.

ghantauke · 11-30-2010, 10:35 AM

Quote:

Originally Posted by GrapefruiTgirl

Code:

root@reactor: echo "word1 word2 word3" | sed 's/(w.*1)(.*)/Q\1Q\2/'
sed: -e expression #1, char 20: invalid reference \2 on `s' command's RHS
root@reactor: echo "word1 word2 word3" | sed 's/(w.*1)(.*)/Q\1Q\2/'
sed: -e expression #1, char 20: invalid reference \2 on `s' command's RHS
root@reactor: echo "word1 word2 word3" | sed 's/(w.*1)(.*)/Q\1Q/'
sed: -e expression #1, char 18: invalid reference \1 on `s' command's RHS
root@reactor:

It seems to matter here, in this case. But, it will depend on the tool, the regex style, and the quoting in use, whether you need to escape, or not, or double-escape, certain things in regex's. I tend to always put escapes in situations like this as a habit - usually there's no harm. Maybe if I actually sat down and studied every possible combination of quotes and regex styles, I might really learn when to escape and when not to, but probably I won't invest the time.

I'll just keep escaping things, and if testing poves it to be bad in a certain case, I fix it then.

Code:

bash-4.1$ sed -r 's/ (s.*e) / 2\1 /g' test
Hey there. This is a 2sample text.
Testing *cough*... Very harsh testing don't.

bash-4.1$ sed -r 's/ \(s.*e\) / 2\1 /g' test
sed: -e expression #1, char 20: invalid reference \1 on `s' command's RHS

It seems to not to require \ before brackets here.

Code:

bash-4.1$ sed 's/([^a-zA-Z])/ \1 /g' test
sed: -e expression #1, char 20: invalid reference \1 on `s' command's RHS

bash-4.1$ sed 's/\([^a-zA-Z]\)/ \1 /g' test
Hey   there .    This   is   a   sample   text . 
Testing    * cough *  .  .  .    Very   harsh   testing   don ' t .

It seems to require \ before brackets here.

Its probably has to do with the fact that I'm using [] in the second code. Well I'm a bit confused but I don't think its gonna be a big problem since I can just use trial and error method when needed. Anyway thank you all for helping me out.

GrapefruiTgirl · 11-30-2010, 10:43 AM

Yes, notice how -r changes the behavior of the regex!

Tinkster · 11-30-2010, 10:45 AM

Quote:

Originally Posted by ghantauke

Its probably has to do with the fact that I'm using [] in the second code. Well I'm a bit confused but I don't think its gonna be a big problem since I can just use trial and error method when needed. Anyway thank you all for helping me out.

Actually it's the -r