-   Programming (
-   -   Passing variables which contain special characters to sed (

Tech109 06-20-2012 01:10 PM

Passing variables which contain special characters to sed
I've got a for loop like the following:


for i in "textA" "textB" "text/C"; do sed -n "/${i}/,/^$\|pattern/p" targetfile; done
The loop searches targetfile for the items and returns each entry followed by several lines underneath, stopping when it reaches a blank line.

The problem is, it chokes on the "text/C" because it contains a forward slash, and returns an error message similar to the following:

sed -e expression 31, char 14: unknown command: `C'

Any suggestions on how to handle special characters in this instance?


MensaWater 06-20-2012 01:35 PM

You can often escape special characters with a backslash. Have you tried:



Tech109 06-20-2012 02:05 PM

The problem is, this list will vary each time I run the loop. The actual list I'm using contains upwards of 150 items, and the number of items containing special characters varies. I need a way to escape special characters no matter what is passed to the variable.


MensaWater 06-20-2012 02:43 PM

Will it always be preceded by "text"? If so escaping the character after "text" might work - \B would likely still see the character as B and \/C would make it see it as /C.

grail 06-20-2012 03:11 PM

You might need to look at sanitizing each variable for unusual characters and escaping them prior to use in your sed.

Nominal Animal 06-20-2012 06:13 PM

In Bash, to escape characters special to standard sed patterns, you could try something like the following within the loop:

    # start with the original pattern

    # escape all backslashes first

    # escape slashes

    # escape asterisks

    # escape full stops

    # escape [ and ]

    # escape ^ and $

    # remove newlines

    # Now, "$escape" should be safe as part of a normal sed pattern.
    # Note that it is NOT safe if the -r option is used.

Note that since sed cannot match a newline as part of a pattern, I chose above to just remove any newlines altogether; you could just as well replace them with spaces or any other character or string. Any newlines in a pattern will cause the sed to fail with an error message.

David the H. 06-21-2012 12:19 PM

The easiest way is simply to change the delimiter that sed uses.

(Edit: the easiest way to avoid conflicting with the delimiter at least. For regex characters, you'd have to do variable sanitizing as above.)

For the 's' substitution command, just replace the slash with another ascii character. Simply choose something that won't ever be found in the expression itself.

You can do the same thing for the target range brackets, except that the first instance needs to be prefixed with a backslash.

To use the underscore as the delimiter, for example (spaced out a bit for better readability):

for i in "textA" "textB" "text/C"; do

        sed -n "\_$i_ , \_^$\|pattern_ p" targetfile


You can even use non-printing characters, provided you can enter them into the shell. The ansi-c style ($'..') quoting mechanism makes this fairly simple. The only characters in the ascii table that can't be used are null and newline. It's probably easier to store the delimiter in a variable first.


D=$'\e'        #The sed delimiter will be the ascii escape character

for i in "textA" "textB" "text/C"; do

        sed -n "\\${D}$i${D} , \\${D}^$\|pattern${D} p" targetfile


(Edit: gotta escape the backslash in front of the "$"s too, when used in double-quotes)

Tech109 06-22-2012 12:08 PM

Thanks David the H., that's the solution!

I was trying to change the delimiter previously, but I didn't have the syntax correct, so I was getting an error. When I use : as the delimiter in the format you listed above, it's working!

Thanks to everyone else who offered input as well!

All times are GMT -5. The time now is 06:39 AM.