SED: how to insert a "&" character in a file

carolflb · 12-16-2009, 10:24 AM

Hi guys,

I'm having a problem with SED and the special character "&". I have a sentence, in which the character "&" appears three times. I saved it as a string:

String="<template>\n<name>InputGen_$inputName#process<\/name>\n<parameter>chan \&$inputName#write#ctrl, int\[-4294967296, 4294967296\] \&$inputName#write#param#val, chan \&process#ctrl<\/parameter>\n<location id=\"id139\" x=\"-1648\" y=\"-40\">\n<urgent\/>\n<\/location>\n<location id=\"id140\" x=\"-1648\" y=\"-144\">\n<\/location><location id=\"id141\" x=\"-1648\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id142\" x=\"-1760\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id143\" x=\"-1888\" y=\"-240\">\n<\/location>\n<location id=\"id144\" x=\"-2024\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id145\" x=\"-2024\" y=\"-392\">\n<\/location>\n<init ref=\"id145\"\/>\n<transition>\n<source ref=\"id139\"\/>\n<target ref=\"id144\"\/>\n<nail x=\"-2024\" y=\"-40\"\/>\n<\/transition>\n<transition>\n<source ref=\"id140\"\/>\n<target ref=\"id139\"\/>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-112\">\n$inputName#write#ctrl?<\/label>\n<\/transition>\n<transition>\n<source ref=\"id141\"\/>\n<target ref=\"id140\"\/>\n<label kind=\"select\" x=\"-1640\" y=\"-224\">\ninterval$a : int\[0,10\]<\/label>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-208\">\n$inputName#write#ctrl"'!'"<\/label>"

And I wanted to insert this string in a text file under the first occurance of the word "</declaration>", using:

sed -i '0,/<\/declaration>/s/<\/declaration>/<\/declaration>\n'"${String}"'/' file.txt

As you can see, I was already inserting an "\" in the string before each special character, but it doesn't help in case I put it in the front of the character "&", i.e. "\&".

Would somebody know what I need to do, to be able to have "&" in my string and insert it in the text file using the command SED?

PS: "InputName" and "a" are variables with the values: "output" and "0"

GrapefruiTgirl · 12-16-2009, 10:44 AM

This *seems to work for me:

Code:

#!/bin/bash

string="<template>\n<name>InputGen_$inputName#process<\/name>\n<parameter>chan \&amp;$inputName#write#ctrl, int\[-4294967296, 4294967296\] \&amp;$inputName#write#param#val, chan \&amp;process#ctrl<\/parameter>\n<location id=\"id139\" x=\"-1648\" y=\"-40\">\n<urgent\/>\n<\/location>\n<location id=\"id140\" x=\"-1648\" y=\"-144\">\n<\/location><location id=\"id141\" x=\"-1648\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id142\" x=\"-1760\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id143\" x=\"-1888\" y=\"-240\">\n<\/location>\n<location id=\"id144\" x=\"-2024\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id145\" x=\"-2024\" y=\"-392\">\n<\/location>\n<init ref=\"id145\"\/>\n<transition>\n<source ref=\"id139\"\/>\n<target ref=\"id144\"\/>\n<nail x=\"-2024\" y=\"-40\"\/>\n<\/transition>\n<transition>\n<source ref=\"id140\"\/>\n<target ref=\"id139\"\/>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-112\">\n$inputName#write#ctrl?<\/label>\n<\/transition>\n<transition>\n<source ref=\"id141\"\/>\n<target ref=\"id140\"\/>\n<label kind=\"select\" x=\"-1640\" y=\"-224\">\ninterval$a : int\[0,10\]<\/label>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-208\">\n$inputName#write#ctrl"'!'"<\/label>"

sed -i.BAK 's/\(<\/declaration>\)/\n\1'"$string"'\n/' ./decla

but if it isn't precisely what you need, tell us what's wrong with it.

PS NOTE: "./decla" is the filename of the test php file I made to test this, so change as needed.

Sasha

berbae · 12-16-2009, 11:10 AM

Code:

#!/bin/bash

String='<template>\n<name>InputGen_$inputName#process</name>\n<parameter>chan \&amp;$inputName#write#ctrl, int[-4294967296, 4294967296] \&amp;$inputName#write#param#val, chan \&amp;process#ctrl</parameter>\n<location id="id139" x="-1648" y="-40">\n<urgent/>\n</location>\n<location id="id140" x="-1648" y="-144">\n</location><location id="id141" x="-1648" y="-240">\n<urgent/>\n</location>\n<location id="id142" x="-1760" y="-240">\n<urgent/>\n</location>\n<location id="id143" x="-1888" y="-240">\n</location>\n<location id="id144" x="-2024" y="-240">\n<urgent/>\n</location>\n<location id="id145" x="-2024" y="-392">\n</location>\n<init ref="id145"/>\n<transition>\n<source ref="id139"/>\n<target ref="id144"/>\n<nail x="-2024" y="-40"/>\n</transition>\n<transition>\n<source ref="id140"/>\n<target ref="id139"/>\n<label kind="synchronisation" x="-1640" y="-112">\n$inputName#write#ctrl?</label>\n</transition>\n<transition>\n<source ref="id141"/>\n<target ref="id140"/>\n<label kind="select" x="-1640" y="-224">\ninterval$a : int[0,10]</label>\n<label kind="synchronisation" x="-1640" y="-208">\n$inputName#write#ctrl"'!'"</label>'

sed -i.bak "0,/<\/declaration>/s+</declaration>+</declaration>\n${String}+" file.txt

Note :
String='\&' single quotes and only the & characters are with backslash
and
sed "s+++" double quotes and the + sign for the s command (instead of /).

carolflb · 12-18-2009, 07:29 AM

Hi people,

both solutions are not working... I need it to insert the "String" just after the first occurance of the word "</declaration>". And $inputName needs to be substituted by its value, so once SED inserts the sentence it needs to write "output" and NOT "$inputName" in the file.

@berbae: your solution is giving me this:

</declaration>
<template>
<name>InputGen_$inputName#process</name>
<parameter>chan </declaration>amp;$inputName#write#ctrl, int[-4294967296, 4294967296] </declaration>amp;$inputName#write#param#val, chan </declaration>amp;process#ctrl</parameter>
<location id="id139" x="-1648" y="-40">
<urgent/>
</location>
<location id="id140" x="-1648" y="-144">
...

It is writing </declaration> where the "&" were... I want to keep the symbol "&" in the sentence. Do you know why it is making that?

And do you also know how to solve the problem with the variable "$inputName"?

ghostdog74 · 12-18-2009, 07:45 AM

Code:

string='"<template>\n<name>InputGen_$inputName#process<\/name>\n<parameter>chan \&amp;$inputName#write#ctrl, int\[-4294967296, 4294967296\] \&amp;$inputName#write#param#val, chan \&amp;process#ctrl<\/parameter>\n<location id=\"id139\" x=\"-1648\" y=\"-40\">\n<urgent\/>\n<\/location>\n<location id=\"id140\" x=\"-1648\" y=\"-144\">\n<\/location><location id=\"id141\" x=\"-1648\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id142\" x=\"-1760\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id143\" x=\"-1888\" y=\"-240\">\n<\/location>\n<location id=\"id144\" x=\"-2024\" y=\"-240\">\n<urgent\/>\n<\/location>\n<location id=\"id145\" x=\"-2024\" y=\"-392\">\n<\/location>\n<init ref=\"id145\"\/>\n<transition>\n<source ref=\"id139\"\/>\n<target ref=\"id144\"\/>\n<nail x=\"-2024\" y=\"-40\"\/>\n<\/transition>\n<transition>\n<source ref=\"id140\"\/>\n<target ref=\"id139\"\/>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-112\">\n$inputName#write#ctrl?<\/label>\n<\/transition>\n<transition>\n<source ref=\"id141\"\/>\n<target ref=\"id140\"\/>\n<label kind=\"select\" x=\"-1640\" y=\"-224\">\ninterval$a : int\[0,10\]<\/label>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-208\">\n$inputName#write#ctrl"'!'"<\/label>"'

awk -v s="$string" '/declaration/{  gsub(/<declaration>/,"<declaration>"s) } {print} ' file

carolflb · 12-18-2009, 07:56 AM

Hi Ghostdog,

nice to have your help again!

The awk command is printing the changes on my terminal, what do I need to do, so that the changes are saved in the file? Otherwise your solution also doesn't help me...

ghostdog74 · 12-18-2009, 08:30 AM

input and output redirection are the very basics of scripting or working with the terminal.
type "man bash" , then type "/" followed by REDIRECTION. otherwise, search google for these terms.

carolflb · 12-18-2009, 08:38 AM

Thanks! Now I could redirect, but the result of your command was inserting the "string" after every occurence of <declaration> in my text. Is there a way for just insert the "string" after the first occurance of </declaration>?

berbae · 12-18-2009, 08:57 AM

There was a problem with the String declaration, at the end of it, with the ctrl"'!'" part (due to the presence of single quotes).
But apart from that and the "$inputName needs to be substituted by its value" which you didn't mention in the first post, it works on my machine, ie the & appear correctly.
Here is what I get now with your new infos.
I tested the command with this file.txt :

Code:

aaa
bbb
ccc
ddd</declaration>eee
fff
ggg
hhh
ddd</declaration>eee
jjj
kkk
lll

The script is now :

Code:

#!/bin/bash

inputName=ABCDE
String='<template>\n<name>InputGen_'"$inputName"'#process</name>\n<parameter>chan \&amp;'"$inputName"'#write#ctrl, int[-4294967296, 4294967296] \&amp;'"$inputName"'#write#param#val, chan \&amp;process#ctrl</parameter>\n<location id="id139" x="-1648" y="-40">\n<urgent/>\n</location>\n<location id="id140" x="-1648" y="-144">\n</location><location id="id141" x="-1648" y="-240">\n<urgent/>\n</location>\n<location id="id142" x="-1760" y="-240">\n<urgent/>\n</location>\n<location id="id143" x="-1888" y="-240">\n</location>\n<location id="id144" x="-2024" y="-240">\n<urgent/>\n</location>\n<location id="id145" x="-2024" y="-392">\n</location>\n<init ref="id145"/>\n<transition>\n<source ref="id139"/>\n<target ref="id144"/>\n<nail x="-2024" y="-40"/>\n</transition>\n<transition>\n<source ref="id140"/>\n<target ref="id139"/>\n<label kind="synchronisation" x="-1640" y="-112">\n'"$inputName"'#write#ctrl?</label>\n</transition>\n<transition>\n<source ref="id141"/>\n<target ref="id140"/>\n<label kind="select" x="-1640" y="-224">\ninterval$a : int[0,10]</label>\n<label kind="synchronisation" x="-1640" y="-208">\n'"$inputName"'#write#ctrl'"\"'"\!"'\""'</label>'

sed -i.bak "0,/<\/declaration>/s+</declaration>+</declaration>\n${String}+" file.txt

Note that I have suppressed the \ before the [ ] and " characters, in the String declaration (not needed inside the single quotes).

When I run it on my machine I get :

Code:

aaa                                                                
bbb                                                                
ccc                                                                
ddd</declaration>                                                  
<template>                                                         
<name>InputGen_ABCDE#process</name>                                
<parameter>chan &amp;ABCDE#write#ctrl, int[-4294967296, 4294967296] &amp;ABCDE#write#param#val, chan &amp;process#ctrl</parameter>
<location id="id139" x="-1648" y="-40">                                                                                           
<urgent/>                                                                                                                         
</location>                                                                                                                       
<location id="id140" x="-1648" y="-144">                                                                                          
</location><location id="id141" x="-1648" y="-240">                                                                               
<urgent/>                                                                                                                         
</location>                                                                                                                       
<location id="id142" x="-1760" y="-240">
<urgent/>
</location>
<location id="id143" x="-1888" y="-240">
</location>
<location id="id144" x="-2024" y="-240">
<urgent/>
</location>
<location id="id145" x="-2024" y="-392">
</location>
<init ref="id145"/>
<transition>
<source ref="id139"/>
<target ref="id144"/>
<nail x="-2024" y="-40"/>
</transition>
<transition>
<source ref="id140"/>
<target ref="id139"/>
<label kind="synchronisation" x="-1640" y="-112">
ABCDE#write#ctrl?</label>
</transition>
<transition>
<source ref="id141"/>
<target ref="id140"/>
<label kind="select" x="-1640" y="-224">
interval$a : int[0,10]</label>
<label kind="synchronisation" x="-1640" y="-208">
ABCDE#write#ctrl"'!'"</label>eee
fff
ggg
hhh
ddd</declaration>eee
jjj
kkk
lll

As you can see the & are there
and the $inputName variable is substituted by its value (here "ABCDE" for the test)
and the ctrl"'!'" is correct now.

So for me what I present answers to your requirements.
So it seems to me...

carolflb · 12-18-2009, 09:25 AM

Hi berbae,

thanks again for your post. I copied exactly what you posted, but I still get "</declaration>amp" on my machine instead of "&amp". It is really strange. Do you know if this could be cause because of the linux or bash version? Anyways thanks again.

ghostdog74 · 12-18-2009, 09:35 AM

Quote:

Originally Posted by carolflb

Thanks! Now I could redirect, but the result of your command was inserting the "string" after every occurence of <declaration> in my text. Is there a way for just insert the "string" after the first occurance of </declaration>?

use a flag to indicate you are done

Code:

awk -v s="$string" '
/declaration/ && !f{
  gsub(/<declaration>/,"<declaration>"s)
  f=1
}{print} ' file

why do you need to escape the & in your string??

berbae · 12-19-2009, 10:48 AM

Some points I could add :

1) I realize from your first post that the end of the String should be ctrl!</label> and not, as I misunderstood, ctrl"'!'"</label>.
So in my proposed code :
ctrl'"\"'"\!"'\""'</label>'
should be replaced by :
ctrl!</label>'

2) It seems that your sed doesn't escaped the & character with the \ before it, as it does on my machine. So for you the & is interpreted laterally to mean the matching string, even with a \ before it.
I don't know exactly why, but probably differences in versions.
For infos I give the versions I used :
GNU bash, version 4.0.35(2)-release
GNU sed version 4.2.1
GNU Awk 3.1.7

3) I looked into the ghostdog74 code, and I could have it to work with these modifications :

Code:

#!/bin/bash                                                                                                                                   

inputName=ABCDE
string="<template>\n<name>InputGen_$inputName#process</name>\n<parameter>chan \\\\&amp;$inputName#write#ctrl, int[-4294967296, 4294967296] \\\\&amp;$inputName#write#param#val, chan \\\\&amp;process#ctrl</parameter>\n<location id=\"id139\" x=\"-1648\" y=\"-40\">\n<urgent/>\n</location>\n<location id=\"id140\" x=\"-1648\" y=\"-144\">\n</location><location id=\"id141\" x=\"-1648\" y=\"-240\">\n<urgent/>\n</location>\n<location id=\"id142\" x=\"-1760\" y=\"-240\">\n<urgent/>\n</location>\n<location id=\"id143\" x=\"-1888\" y=\"-240\">\n</location>\n<location id=\"id144\" x=\"-2024\" y=\"-240\">\n<urgent/>\n</location>\n<location id=\"id145\" x=\"-2024\" y=\"-392\">\n</location>\n<init ref=\"id145\"/>\n<transition>\n<source ref=\"id139\"/>\n<target ref=\"id144\"/>\n<nail x=\"-2024\" y=\"-40\"/>\n</transition>\n<transition>\n<source ref=\"id140\"/>\n<target ref=\"id139\"/>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-112\">\n$inputName#write#ctrl?</label>\n</transition>\n<transition>\n<source ref=\"id141\"/>\n<target ref=\"id140\"/>\n<label kind=\"select\" x=\"-1640\" y=\"-224\">\ninterval$a : int[0,10]</label>\n<label kind=\"synchronisation\" x=\"-1640\" y=\"-208\">\n$inputName#write#ctrl!</label>"

awk -v s="$string" '
/declaration/ && !f{
  gsub(/<\/declaration>/,"</declaration>\n"s)
  f=1
}{print} ' file.txt

In the string declaration, only the " inside the string are escaped with one \, and & needs four \ on my machine. No need of the single quotes, because then the $inputName is not replaced by its value.
Apart from that I get the same output with my file.txt test file as with the sed command.
Post a follow-up if you find a working solution.