LinuxQuestions.org - sed problems when run in $( ... )

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - sed problems when run in $( ... ) (https://www.linuxquestions.org/questions/linux-newbie-8/sed-problems-when-run-in-%24-737815/)

sed problems when run in $( ... )

Hi all,

I'm trying to write a script that will setup a new website on my development box - one that automatically creates its Subversion repository, checks out a local working copy, creates folders for the apache scripts and log files, and updates the apache config. It's all going swimmingly, except for the last bit - automatically updating the apache config. The "template" apache config for each new website is stored in a file with dummy entries where the real values should go (so for instance WEBSITE_NAME will be replaced by a display name for the website), and I'm trying to use sed to replace these values in my script.

Because of what the script is doing, I've added a "debug" mode that echo's the commands to be run, rather than actually running them (so I don't mess up my system), so the commands are run from a variable (for instance $( $cmd )). This bit seems to work fine, except when I try and use sed to update the config file, in which case I get the following error that I haven't been able to resolve after several hours of Googling:

sed: -e expression #1, char 1: unknown command: `''

I've managed to whittle the error down to the following:

Code:

# Create the sed command

cmd="sed 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"



# Try running in a sub-shell

output="$( $cmd )" # Tried both like this...

output="`$cmd`"    # ... and in backticks, and both give the above error



# Try piping through bash

output="$( echo "$cmd" | bash )"

echo "$output"    # This works!



# Try running manually

sed 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel

                  # This works too!

Naturally removing the spaces and the single quotes from $cmd makes all the little errors go away, but that's not exactly practical in the real world. I've tried quoting the s/// part of the sed command in both single quotes (as above) and double quotes, and the only difference is that the error complains about double quotes instead. So, is there some kind of extra variable substitution/interpolation that occurs when I run it this way, as the command itself works fine when run manually? Or is there something else wrong that is blatantly obvious to everyone but me?

I'm running an up-to-date Fedora 11, with sed version 4.1.5 and bash 4.0.16(1)-release.

Thanks!

PS. For the record, /svn/svn-admin/httpd-conf-skel contains

Code:

# WEBSITE_NAME

<VirtualHost *:80>

    DocumentRoot /var/www/html/WWW_DIR/htdocs

    ServerName SERVER_NAME

    ErrorLog logs/LOG_DIR/error_log

    CustomLog logs/LOG_DIR/access_log common

</VirtualHost>

The problem is the way the shell quotes the string in $cmd. If you use bash -x you can see it:

Code:

$ cat test.sh

#!/bin/bash

cmd="sed 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"



# Try running in a sub-shell

output="$( $cmd )"

$ bash -x test.sh

+ cmd='sed '\''s/WEBSITE_NAME/Test\ With\ Spaces/g'\'' /svn/svn-admin/httpd-conf-skel'

++ sed ''\''s/WEBSITE_NAME/Test\' 'With\' 'Spaces/g'\''' /svn/svn-admin/httpd-conf-skel

sed: -e expression #1, char 1: unknown command: `''

+ output=

As you can see the extra single quotes at the beginning of the sed command trigger this error. I have tried different combinations of double and single quotes, but I couldn't run it without errors. However, why not keep it simple and do the following?

Code:

$ cat test.sh

#!/bin/bash

output=$(sed 's/WEBSITE_NAME/Test With Spaces/g' /svn/svn-admin/httpd-conf-skel)

$ bash -x test.sh

++ sed 's/WEBSITE_NAME/Test With Spaces/g' /svn/svn-admin/httpd-conf-skel

+ output='# Test With Spaces

<VirtualHost *:80>

    DocumentRoot /var/www/html/WWW_DIR/htdocs

    ServerName SERVER_NAME

    ErrorLog logs/LOG_DIR/error_log

    CustomLog logs/LOG_DIR/access_log common

</VirtualHost>'

Quote:

Originally Posted by colucix (Post 3597205)

If you use bash -x you can see it

That tip looks like it might save me lots of time in the future, so thanks for that!

The reason I'm not running the command directly in a $( ... ) is that there's multiple commands in the script, each of which is run through a common function that checks to see if the debug mode has been set - if it has, it simply echo's the command to STDOUT, otherwise its tries running it.

The "fuller" command that is causing a problem in my script is this:

Code:

# Display command if debugging, otherise run

function check_for_debug() {

  # Display, if debugging

  if [ "x$debug" = "x1" ]

  then

    echo "$cmd"



  # We're not debugging, so run

  else

    output="$( $cmd )"

  fi

}



# State debug mode

debug=1



# Run the commands

cmd="sed -ia 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"

check_for_debug

cmd="sed -ib 's/WWW_DIR/test/g' /svn/svn-admin/httpd-conf-skela"

check_for_debug

cmd="sed -ic 's/SERVER_NAME/test.localdomain/g' /svn/svn-admin/httpd-conf-skelab"

check_for_debug

cmd="sed -id 's/LOG_DIR/test/g' /svn/svn-admin/httpd-conf-skelabc"

check_for_debug

instead of having to duplicate each of the commands in the echo... not to mention the unnecessary extra code that's required

Code:

# State debug mode

debug=1



# Run the commands

if [ "x$debug" = "x1" ]

then

  echo "sed -ia 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"

else

  output=$( sed -ia 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel )

fi



if [ "x$debug" = "x1" ]

then

  echo "sed -ib 's/WWW_DIR/test/g' /svn/svn-admin/httpd-conf-skela"

else

  output=$( sed -ib 's/WWW_DIR/test/g' /svn/svn-admin/httpd-conf-skela )

fi



if [ "x$debug" = "x1" ]

then

  echo "sed -ic 's/SERVER_NAME/test.localdomain/g' /svn/svn-admin/httpd-conf-skelab"

else

  output=$( sed -ic 's/SERVER_NAME/test.localdomain/g' /svn/svn-admin/httpd-conf-skelab )

fi



if [ "x$debug" = "x1" ]

then

  echo "sed -id 's/LOG_DIR/test/g' /svn/svn-admin/httpd-conf-skelabc"

else

  output=$( sed -id 's/LOG_DIR/test/g' /svn/svn-admin/httpd-conf-skelabc )

fi

Or am I trying to do too much with bash?

Thanks again.

Use eval to reverse the bash quoting, that is let bash assign the command using quotes and other stuff, then use eval to evaluate what the bash has written in the cmd string. You should not have any problem anymore.

Code:

output="$(eval $cmd)"

Works perfectly, thanks for your help!