I have an output file from a script command that has the string '^[[m' and '^[[1m' that need to be removed. I can not seem to get the sed command to work with those strings. Does anybody else know of a command that can parse those strings out of the file?

AIX version is 5.2

Are you sure it is characters and not simply the way you're seeing a control string. (^M for example is how you would see "carriage returns" in a DOS ascii file). The ^M in that case is actually a single character rather than the two characters ^ and M. You can see that by going to the character and trying to move right - if it jumps past the rest then it is a control character but if it actually goes to the M then you know it is two characters.

Assuming it IS characters you might try escaping the special ones in sed:

\^\[\[m

That way sed knows to treat them as literal characters rather than interpreting special meaning for them (e.g. ^ can mean "beginning of the line" in regular expressions [regex].)

I tried both of these commands...

sed 's/\^\[\[m//g' session.200704061648.log > new.log

...doesn't work. But the command did execute, where as before would give me an error. Any more advice? When I open the file in vi, the characters do not seem to be control characters, I can see other '^M' and those instances behave like you predicted. But, when I use the sed command with an output redirect to a file, it shows the contents of the file without the '^[[m' string.

One other thing I just noticed. When I use a 'more' command on the file, I can see the '^[[m', but when I use 'cat', they're not there.

One of the fun things when dealing with REGEX is you sometimes have to escape things AND quote your escapes. (This is because the shell is interpreting things THEN sed is. Quoting makes the shell pass through the escapes.)

Just did a test here and this worked for me:

sed -e s/'\^''\[''\['m//

Essentailly it is escaping with "\" (e.g. \^) but putting single quotes around the escape it is doing (e.g. '\^'). Since you have to escape each character but have to quote it.

It also worked just to quote the whole thing rather than the individual components:

sed -e s/'\^\[\['m//

Ok, no matter what I do, those characters still remain visible with a more command and vi but not with cat. I'm just going to leave it as it is, and assume that they are just control characters and that they won't be visible with a normal text editor. And even if they are.

Thanks for your help, you've cleared up alot of my questions with sed and special characters in general.

These characters are mostly use to edit appearance of your output. the ^[ character is created by hitting CTRL+v, then Esc key. If you do echo "^[[31mHello^[[0m" it will print "Hello" in read if your terminal is color capable. There is no way to display these characters with a shell command since they mean something to the shell. If you want to get rid of them you have to do it from the vi session. by running your sed command from :%s/\^\[\[m//g

So my original post was ignored apparently. I pointed out that often these characters aren't literal but are rather control strings. I'd asked the OP to check for this before trying anything.

No, I didn't ignore it. They did not behave like the other control characters in the file so I assumed they were literal. I'm still getting the hang of AIX, I'm a Linux guy and I had never seen anything like that in any file under Linux before.

OK my apologies.

One way that might work:

cat -v file >file2

This converts the non-printing characters to literal characters (usually) and you may then be able to strip the characters out with the sed syntax I gave because file2 now has it as literal characters.

I just tested the above using ^B (ctrl-B). In the original file that shows as 1 character. In the output it shows as two (^ and B).

You could probably even just pipe the "cat -v file" into the sed and output that to your result file but I haven't tried that.

Yup, that works. Now that they are converted to literals they can be removed with a sed command. I'm going to mess around with it alittle more today, since the files would be viewed from a Windows system I may not need to remove the control characters. But if I do, I know how to do it now.

Thanks alot for the help.

One more question. I have it working now, but I need to use the 'sed' command 3 times to remove 3 string. The string I need to remove are '^[[m', '^[[1m', and '^M'. What would the sed command be to go through the entire file and remove all instances of those 3?

You can specify "-e" with the s/pattern/replacement/" multiple times - just do it for each pattern. Also put g after the / so it does it "globally" (each time it appears in the line) otherwise it would only do the first time it appeared in the line.

e.g:
sed -e s/'\^\[\['m//g -e s/'\^\[\[1m'//g -e -s/'\^M'/g

By the way ^M is carriage return. If you run into files that have only those in them you can convert it with dos2unix command (or add it by using the unix2dos command). This a common thing when copying text files from Linux to DOS (Windows) or vice-versa.

A good reference for regular expressions is:
http://www.regular-expressions.info/

FJD