sed problems when run in $( ... )

thierryd_uk · 07-05-2009, 05:09 AM

Hi all,

I'm trying to write a script that will setup a new website on my development box - one that automatically creates its Subversion repository, checks out a local working copy, creates folders for the apache scripts and log files, and updates the apache config. It's all going swimmingly, except for the last bit - automatically updating the apache config. The "template" apache config for each new website is stored in a file with dummy entries where the real values should go (so for instance WEBSITE_NAME will be replaced by a display name for the website), and I'm trying to use sed to replace these values in my script.

Because of what the script is doing, I've added a "debug" mode that echo's the commands to be run, rather than actually running them (so I don't mess up my system), so the commands are run from a variable (for instance $( $cmd )). This bit seems to work fine, except when I try and use sed to update the config file, in which case I get the following error that I haven't been able to resolve after several hours of Googling:

sed: -e expression #1, char 1: unknown command: `''

I've managed to whittle the error down to the following:

Code:

# Create the sed command
cmd="sed 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"

# Try running in a sub-shell
output="$( $cmd )" # Tried both like this...
output="`$cmd`"    # ... and in backticks, and both give the above error

# Try piping through bash
output="$( echo "$cmd" | bash )"
echo "$output"     # This works!

# Try running manually
sed 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel
                   # This works too!

Naturally removing the spaces and the single quotes from $cmd makes all the little errors go away, but that's not exactly practical in the real world. I've tried quoting the s/// part of the sed command in both single quotes (as above) and double quotes, and the only difference is that the error complains about double quotes instead. So, is there some kind of extra variable substitution/interpolation that occurs when I run it this way, as the command itself works fine when run manually? Or is there something else wrong that is blatantly obvious to everyone but me?

I'm running an up-to-date Fedora 11, with sed version 4.1.5 and bash 4.0.16(1)-release.

Thanks!

PS. For the record, /svn/svn-admin/httpd-conf-skel contains

Code:

# WEBSITE_NAME
<VirtualHost *:80>
    DocumentRoot /var/www/html/WWW_DIR/htdocs
    ServerName SERVER_NAME
    ErrorLog logs/LOG_DIR/error_log
    CustomLog logs/LOG_DIR/access_log common
</VirtualHost>

colucix · 07-05-2009, 05:52 AM

The problem is the way the shell quotes the string in $cmd. If you use bash -x you can see it:

Code:

$ cat test.sh
#!/bin/bash
cmd="sed 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"

# Try running in a sub-shell
output="$( $cmd )"
$ bash -x test.sh
+ cmd='sed '\''s/WEBSITE_NAME/Test\ With\ Spaces/g'\'' /svn/svn-admin/httpd-conf-skel'
++ sed ''\''s/WEBSITE_NAME/Test\' 'With\' 'Spaces/g'\''' /svn/svn-admin/httpd-conf-skel
sed: -e expression #1, char 1: unknown command: `''
+ output=

As you can see the extra single quotes at the beginning of the sed command trigger this error. I have tried different combinations of double and single quotes, but I couldn't run it without errors. However, why not keep it simple and do the following?

Code:

$ cat test.sh
#!/bin/bash
output=$(sed 's/WEBSITE_NAME/Test With Spaces/g' /svn/svn-admin/httpd-conf-skel)
$ bash -x test.sh
++ sed 's/WEBSITE_NAME/Test With Spaces/g' /svn/svn-admin/httpd-conf-skel
+ output='# Test With Spaces
<VirtualHost *:80>
    DocumentRoot /var/www/html/WWW_DIR/htdocs
    ServerName SERVER_NAME
    ErrorLog logs/LOG_DIR/error_log
    CustomLog logs/LOG_DIR/access_log common
</VirtualHost>'

thierryd_uk · 07-05-2009, 06:20 AM

Quote:

Originally Posted by colucix

If you use bash -x you can see it

That tip looks like it might save me lots of time in the future, so thanks for that!

The reason I'm not running the command directly in a $( ... ) is that there's multiple commands in the script, each of which is run through a common function that checks to see if the debug mode has been set - if it has, it simply echo's the command to STDOUT, otherwise its tries running it.

The "fuller" command that is causing a problem in my script is this:

Code:

# Display command if debugging, otherise run
function check_for_debug() {
  # Display, if debugging
  if [ "x$debug" = "x1" ]
  then
    echo "$cmd"

  # We're not debugging, so run
  else
    output="$( $cmd )"
  fi
}

# State debug mode
debug=1

# Run the commands
cmd="sed -ia 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"
check_for_debug
cmd="sed -ib 's/WWW_DIR/test/g' /svn/svn-admin/httpd-conf-skela"
check_for_debug
cmd="sed -ic 's/SERVER_NAME/test.localdomain/g' /svn/svn-admin/httpd-conf-skelab"
check_for_debug
cmd="sed -id 's/LOG_DIR/test/g' /svn/svn-admin/httpd-conf-skelabc"
check_for_debug

instead of having to duplicate each of the commands in the echo... not to mention the unnecessary extra code that's required

Code:

# State debug mode
debug=1

# Run the commands
if [ "x$debug" = "x1" ]
then
  echo "sed -ia 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel"
else
  output=$( sed -ia 's/WEBSITE_NAME/Test\ With\ Spaces/g' /svn/svn-admin/httpd-conf-skel )
fi

if [ "x$debug" = "x1" ]
then
  echo "sed -ib 's/WWW_DIR/test/g' /svn/svn-admin/httpd-conf-skela"
else
  output=$( sed -ib 's/WWW_DIR/test/g' /svn/svn-admin/httpd-conf-skela )
fi

if [ "x$debug" = "x1" ]
then
  echo "sed -ic 's/SERVER_NAME/test.localdomain/g' /svn/svn-admin/httpd-conf-skelab"
else
  output=$( sed -ic 's/SERVER_NAME/test.localdomain/g' /svn/svn-admin/httpd-conf-skelab )
fi

if [ "x$debug" = "x1" ]
then
  echo "sed -id 's/LOG_DIR/test/g' /svn/svn-admin/httpd-conf-skelabc"
else
  output=$( sed -id 's/LOG_DIR/test/g' /svn/svn-admin/httpd-conf-skelabc )
fi

Or am I trying to do too much with bash?

Thanks again.

colucix · 07-05-2009, 07:14 AM

Use eval to reverse the bash quoting, that is let bash assign the command using quotes and other stuff, then use eval to evaluate what the bash has written in the cmd string. You should not have any problem anymore.

Code:

output="$(eval $cmd)"

thierryd_uk · 07-05-2009, 10:57 AM

Works perfectly, thanks for your help!