[SOLVED] sed gives :sed: -e expression #1, char 1: unknown command: `''

samasat · 05-26-2012, 10:55 PM

I am attempting to automatically replace a output name string in a batchfile prior to its execution to describe settings explicitely in output. However, I am not able to to do that
My Code:

Code:

inFile="$WORK/${SimName}.in"
replaceCmd="sed 's/sim.dat/${SimName}.dat/g' $inFile"
echo $replaceCmd
$replaceCmd

And the output tracked by echo'ing is:

Code:

sed 's/sim.dat/sim0525_PerfCentOS_U032_MAT04_THRD024.dat/g' /home/ssr/Desktop/PerfCentOS/sim0525_PerfCentOS_U032_MAT04_THRD024/sim0525_PerfCentOS_U032_MAT04_THRD024.in
sed: -e expression #1, char 1: unknown command: `''

Any suggestions ?
Thanks,

Ser Olmy · 05-26-2012, 11:11 PM

It seems to be choking on the quote character. Have you tried using escaped double quotes (\") instead of single quotes around the sed expression?

vonbiber · 05-27-2012, 02:50 AM

Quote:

Originally Posted by samasat

inFile="$WORK/${SimName}.in"
replaceCmd="sed 's/sim.dat/${SimName}.dat/g' $inFile"
echo $replaceCmd
$replaceCmd

you're trying to replace 'sim.dat' with the value of $SimName.dat in $inFile
try this

Code:

cmd="sed 's?sim\.dat?$SimName.dat?g' $inFile"
eval "$cmd"

David the H. · 05-27-2012, 08:23 AM

Please use ***[code][/code] tags*** around your code and data, to preserve formatting and to improve readability. Please do not use quote tags, colors, or other fancy formatting.

Oh, hells no! Never, and I mean NEVER, use eval for something as trivial as this. Indeed, you should only very rarely need it at all. It can be quite a security risk, and is almost never really necessary in general scripting.

Eval command and security issues
http://mywiki.wooledge.org/BashFAQ/048

In addition, variables are for storing data, not code. The spaces, quotes, and other special characters cease to be special when in a variable (unless the expansion is unquoted, then some of them still have an effect, but likely not the one you want; see my next point), and in any case leads to unclear, obfuscated code flow. If you need repeated use of the same command, don't put it inside a variable, use a function.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

A second problem here is that if the sed command is surrounded by single quotes, the variable inside them does not get expanded. No quotes is even worse, as the shell attempts to perform word-splitting and glob expansion after the substitution.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Here's what you should be doing:

Code:

replaceCmd(){

	sed "s/sim.dat/$1.dat/g" "$2"

}

inFile="$work/$SimName.in"

replaceCmd "$SimName" "$inFile"

BTW, since environment variables are generally all upper-case, it's good practice to keep your own user variables in lower-case or mixed-case to help differentiate them.

Also, please don't use the full brace-surounded "${var}" form unless absolutely necessary. It does nothing useful, and only adds extra typing and clutter to your code.

samasat · 05-27-2012, 12:35 PM

Quote:

Originally Posted by Ser Olmy

It seems to be choking on the quote character. Have you tried using escaped double quotes (\") instead of single quotes around the sed expression?

Thanks for your suggestion Ser. I did not make any difference in error message overall.

I used following

Code:

replaceCmd="sed \"s/sim.dat/${SimName}.dat/g\" $inFile"
echo $replaceCmd
$replaceCmd

Output was:

Code:

sed "s/sim.dat/sim0502_PerfCentOS_U038_MAT01_THRD01.dat/g" /home/ssr/Desktop/TrashIt_BashTest/sim0502_PerfCentOS_U038_MAT01_THRD01/sim0502_PerfCentOS_U038_MAT01_THRD01.in
sed: -e expression #1, char 1: unknown command: `"'

samasat · 05-27-2012, 01:10 PM

Thanks David. Your help did work.
Only minor correction I had to make was that sed line I was using earlier was not correct. My objective was to replace the string inside the file and save the changes. When I used the old approach it just threw it corrected on screen but changes in file were not saved.
Then I tried this one which also did not work. It may be my ignorance that redirection does not work with sed:

Code:

replaceCmd(){
	sed "s/sim.dat/$1.dat/g" "$2" > "$2"
}

Finally, after googling a bit I found '-i' switch that saved file changes as given below:

Code:

replaceCmd(){
	sed -i "s/sim.dat/$1.dat/g" "$2"
}

I just inserted this function in my code to replace old lines. However, I was wondering if there is any standard/conventional location of such additional function in the code e.g. in C we declare it right at the top or through header file ...

Also, I do agree with approach from your post and the link that:

Quote:

variables are for storing data, not code

and also regarding security risks involved in eval. My only addition is that it also depends on where the code is going to be implemented. e.g. if I am using this code for my research, probably I would think that security is not a concern for me as long as task gets done since anyway my work is going to go open-source. Also, there is a fact that out of 100 things or approaches may be 5-6 do work. I guess, in that case, the ease to try out things quickly gains more importance prior to arriving at final professional level code for finalized approach. There definitely one has to be cautious and perfect and should implement your suggestion.

Thanks David, Ser and vonbiber. I appreciate your help and time !
Thanks,

chrism01 · 05-27-2012, 06:14 PM

Quote:

I just inserted this function in my code to replace old lines. However, I was wondering if there is any standard/conventional location of such additional function in the code e.g. in C we declare it right at the top or through header file ...

The shells read strictly top to bottom, so the fn defn has to come before its called in the 'main' as you can't reference a fn you haven't already 'read' (parsed).

This is in contrast to C or Perl, where the 'compiler/linker' reads the entire src file before trying to action anything.
Shell is interpreted ie parsed+actioned as it is read.

HTH

samasat · 05-28-2012, 10:34 AM

Thanks Chrism01 !

David the H. · 05-28-2012, 10:55 AM

Code:

sed "s/sim.dat/$1.dat/g" "$2" > "$2"

And this doesn't work because the ">" redirection is parsed, and the input file is overwritten, before the command is launched. So you end up with an empty file for input. You have to go through a temporary file, or use the -i option, which actually does the same thing, just in the background.

Incidentally, I've recently been learning how to use ed, and you might consider using that instead. As a real text editor, it can operate on the file directly.

It might also be a good idea to make your function more general in scope, so you can continue to use it even with different starting and ending strings.

Code:

# function: replaceCmd
# globally replace one string with another using ed.
# $1 is the filename, $2 is the string to match
# $3 is the replacement string.  strings are treated as regexes.
# In addition, any forward slashes in the strings must be
# escaped with backslashes, due to the delimiter.

replaceCmd() {

	printf '%s\n' "g/$2/ s/$2/$3/g" 'w' | ed -s "$1"

}

replaceCmd "$inFile" "sim.dat" "$SimName.dat"

How to use ed:
http://wiki.bash-hackers.org/howto/edit-ed
http://snap.nlc.dcccd.edu/learn/nlc/ed.html
(also read the info page)

Finally, re your eval comment. While you are certainly correct that you don't need to worry so much about security in a limited, single-user situation, I'd still recommend against it.

First of all, security isn't just about unauthorized malicious users; it also includes authorized users making mistakes. Don't underestimate the possibility of a malformed string accidentally forming an unwanted command or having other unintended side-effects.

Second, laziness and sloppiness are also risks to beware of. If you're lax in coding when it's safe, then you aren't training yourself to be careful when it isn't. Over-dependence on quick 'n easy, just-get-it-done solutions means you aren't really learning how to use the better alternatives available.

eval is the nuclear weapon of the scripting world; it's only meant to be brought out as a last resort when nothing else works. And since it can nearly always be avoided with just a little bit of work and know-how, any use of it, without a damn good reason and without knowing exactly what you're doing, is really a bad use of it, IMO.

Roken · 05-28-2012, 01:31 PM

I can second (or should that be third) comments about using eval. I'm a reasonably experienced and competent bash scripter, but yesterday I managed to completely trash a system configuration file with the use of eval and a string that didn't end up quite how I intended.

Two lessons learned:

i. Don't use eval
ii. Never test live on system configurations lol

As it happens, it was a fairly trivial matter to re-write the config file - this time. Other times I may not be so lucky.

samasat · 06-09-2012, 05:31 PM

Thanks David and Roken for your advice. I will keep that in mind !

Also, thanks David for the idea of making it generalized. I hope to apply same while using sed as well. I am yet to try 'ed'.

UPDATE:
I have checked that both sed and ed work equally good.

Code:

replaceCmd_sed(){ # $1 -> OldStr,  $2 -> NewStr , $3 -> Name of file to edit
	sed -i "s/$1/$2/g" "$3"
}

replaceCmd_ed(){ # $1 -> OldStr,  $2 -> NewStr , $3 -> Name of file to edit
	printf '%s\n' "g/$1/ s/$1/$2/g" 'w' | ed -s "$3"
}