[SOLVED] Passing variables which contain special characters to sed

Tech109 · 06-20-2012, 12:10 PM

I've got a for loop like the following:

Code:

for i in "textA" "textB" "text/C"; do sed -n "/${i}/,/^$\|pattern/p" targetfile; done

The loop searches targetfile for the items and returns each entry followed by several lines underneath, stopping when it reaches a blank line.

The problem is, it chokes on the "text/C" because it contains a forward slash, and returns an error message similar to the following:

sed -e expression 31, char 14: unknown command: `C'

Any suggestions on how to handle special characters in this instance?

Thanks.

MensaWater · 06-20-2012, 12:35 PM

You can often escape special characters with a backslash. Have you tried:

Code:

text\/C

Tech109 · 06-20-2012, 01:05 PM

The problem is, this list will vary each time I run the loop. The actual list I'm using contains upwards of 150 items, and the number of items containing special characters varies. I need a way to escape special characters no matter what is passed to the variable.

Thanks.

MensaWater · 06-20-2012, 01:43 PM

Will it always be preceded by "text"? If so escaping the character after "text" might work - \B would likely still see the character as B and \/C would make it see it as /C.

grail · 06-20-2012, 02:11 PM

You might need to look at sanitizing each variable for unusual characters and escaping them prior to use in your sed.

Nominal Animal · 06-20-2012, 05:13 PM

In Bash, to escape characters special to standard sed patterns, you could try something like the following within the loop:

Code:

    # start with the original pattern
    escaped="$i"

    # escape all backslashes first
    escaped="${escaped//\\/\\\\}"

    # escape slashes
    escaped="${escaped//\//\\/}"

    # escape asterisks
    escaped="${escaped//\*/\\*}"

    # escape full stops
    escaped="${escaped//./\\.}"    

    # escape [ and ]
    escaped="${escaped//\[/\\[}"
    escaped="${escaped//\[/\\]}"

    # escape ^ and $
    escaped="${escaped//^/\\^}"
    escaped="${escaped//\$/\\\$}"

    # remove newlines
    escaped="${escaped//[$'\n']/}"

    # Now, "$escape" should be safe as part of a normal sed pattern.
    # Note that it is NOT safe if the -r option is used.

Note that since sed cannot match a newline as part of a pattern, I chose above to just remove any newlines altogether; you could just as well replace them with spaces or any other character or string. Any newlines in a pattern will cause the sed to fail with an error message.

David the H. · 06-21-2012, 11:19 AM

The easiest way is simply to change the delimiter that sed uses.

(Edit: the easiest way to avoid conflicting with the delimiter at least. For regex characters, you'd have to do variable sanitizing as above.)

For the 's' substitution command, just replace the slash with another ascii character. Simply choose something that won't ever be found in the expression itself.

You can do the same thing for the target range brackets, except that the first instance needs to be prefixed with a backslash.

To use the underscore as the delimiter, for example (spaced out a bit for better readability):

Code:

for i in "textA" "textB" "text/C"; do

	sed -n "\_$i_ , \_^$\|pattern_ p" targetfile

done

You can even use non-printing characters, provided you can enter them into the shell. The ansi-c style ($'..') quoting mechanism makes this fairly simple. The only characters in the ascii table that can't be used are null and newline. It's probably easier to store the delimiter in a variable first.

Code:

D=$'\e'	#The sed delimiter will be the ascii escape character

for i in "textA" "textB" "text/C"; do

	sed -n "\\${D}$i${D} , \\${D}^$\|pattern${D} p" targetfile

done

(Edit: gotta escape the backslash in front of the "$"s too, when used in double-quotes)

Tech109 · 06-22-2012, 11:08 AM

Thanks David the H., that's the solution!

I was trying to change the delimiter previously, but I didn't have the syntax correct, so I was getting an error. When I use : as the delimiter in the format you listed above, it's working!

Thanks to everyone else who offered input as well!