LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 06-20-2012, 12:10 PM   #1
Tech109
LQ Newbie
 
Registered: Jan 2011
Posts: 12

Rep: Reputation: 0
Passing variables which contain special characters to sed


I've got a for loop like the following:

Code:
for i in "textA" "textB" "text/C"; do sed -n "/${i}/,/^$\|pattern/p" targetfile; done
The loop searches targetfile for the items and returns each entry followed by several lines underneath, stopping when it reaches a blank line.

The problem is, it chokes on the "text/C" because it contains a forward slash, and returns an error message similar to the following:

sed -e expression 31, char 14: unknown command: `C'

Any suggestions on how to handle special characters in this instance?

Thanks.
 
Old 06-20-2012, 12:35 PM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,995
Blog Entries: 5

Rep: Reputation: 782Reputation: 782Reputation: 782Reputation: 782Reputation: 782Reputation: 782Reputation: 782
You can often escape special characters with a backslash. Have you tried:

Code:
text\/C
 
Old 06-20-2012, 01:05 PM   #3
Tech109
LQ Newbie
 
Registered: Jan 2011
Posts: 12

Original Poster
Rep: Reputation: 0
The problem is, this list will vary each time I run the loop. The actual list I'm using contains upwards of 150 items, and the number of items containing special characters varies. I need a way to escape special characters no matter what is passed to the variable.

Thanks.
 
Old 06-20-2012, 01:43 PM   #4
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,995
Blog Entries: 5

Rep: Reputation: 782Reputation: 782Reputation: 782Reputation: 782Reputation: 782Reputation: 782Reputation: 782
Will it always be preceded by "text"? If so escaping the character after "text" might work - \B would likely still see the character as B and \/C would make it see it as /C.
 
Old 06-20-2012, 02:11 PM   #5
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,492

Rep: Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892Reputation: 1892
You might need to look at sanitizing each variable for unusual characters and escaping them prior to use in your sed.
 
Old 06-20-2012, 05:13 PM   #6
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
In Bash, to escape characters special to standard sed patterns, you could try something like the following within the loop:
Code:
    # start with the original pattern
    escaped="$i"

    # escape all backslashes first
    escaped="${escaped//\\/\\\\}"

    # escape slashes
    escaped="${escaped//\//\\/}"

    # escape asterisks
    escaped="${escaped//\*/\\*}"

    # escape full stops
    escaped="${escaped//./\\.}"    

    # escape [ and ]
    escaped="${escaped//\[/\\[}"
    escaped="${escaped//\[/\\]}"

    # escape ^ and $
    escaped="${escaped//^/\\^}"
    escaped="${escaped//\$/\\\$}"

    # remove newlines
    escaped="${escaped//[$'\n']/}"

    # Now, "$escape" should be safe as part of a normal sed pattern.
    # Note that it is NOT safe if the -r option is used.
Note that since sed cannot match a newline as part of a pattern, I chose above to just remove any newlines altogether; you could just as well replace them with spaces or any other character or string. Any newlines in a pattern will cause the sed to fail with an error message.
 
Old 06-21-2012, 11:19 AM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
The easiest way is simply to change the delimiter that sed uses.

(Edit: the easiest way to avoid conflicting with the delimiter at least. For regex characters, you'd have to do variable sanitizing as above.)

For the 's' substitution command, just replace the slash with another ascii character. Simply choose something that won't ever be found in the expression itself.

You can do the same thing for the target range brackets, except that the first instance needs to be prefixed with a backslash.

To use the underscore as the delimiter, for example (spaced out a bit for better readability):
Code:
for i in "textA" "textB" "text/C"; do

	sed -n "\_$i_ , \_^$\|pattern_ p" targetfile

done
You can even use non-printing characters, provided you can enter them into the shell. The ansi-c style ($'..') quoting mechanism makes this fairly simple. The only characters in the ascii table that can't be used are null and newline. It's probably easier to store the delimiter in a variable first.

Code:
D=$'\e'	#The sed delimiter will be the ascii escape character

for i in "textA" "textB" "text/C"; do

	sed -n "\\${D}$i${D} , \\${D}^$\|pattern${D} p" targetfile

done
(Edit: gotta escape the backslash in front of the "$"s too, when used in double-quotes)

Last edited by David the H.; 06-21-2012 at 11:27 AM.
 
1 members found this post helpful.
Old 06-22-2012, 11:08 AM   #8
Tech109
LQ Newbie
 
Registered: Jan 2011
Posts: 12

Original Poster
Rep: Reputation: 0
Thanks David the H., that's the solution!

I was trying to change the delimiter previously, but I didn't have the syntax correct, so I was getting an error. When I use : as the delimiter in the format you listed above, it's working!


Thanks to everyone else who offered input as well!
 
  


Reply

Tags
sed, sed bash


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to use sed to find lines with special characters bryanwdly Linux - General 2 02-07-2012 12:54 PM
Help w/ sed parsing special characters clem_c_rock General 8 08-31-2007 04:06 PM
How to export special characters into environment variables? das_Schaf Linux - General 6 06-24-2006 07:17 AM
using sed to insert lines with special characters disorderly Linux - Software 26 04-20-2006 05:30 PM
passing variables to sed jjfate Programming 8 07-31-2003 04:15 AM


All times are GMT -5. The time now is 03:54 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration