CTRL -A representations in Linux terminal log output

casperdaghost · 11-07-2011, 10:36 AM

i have a log file and instead of spaces it has a box with three zeros and a one inbetween the integers

picture the below surrounded by a box - very small, numbers though - the whole thing is only about as big as the monspaced integer.

Code:

00
01

I am pretty sure that these are CRTL-A characters.
i am trying to cat the log file and use sed to take them out, but it is not working -

sed 's/\0001/ /g'

I even tried hitting a CRTL -A manually in the sed replace, but that is not working either -

anybody ever see thse in s alog file

casperdaghost · 11-07-2011, 10:45 AM

when i open up the file in vi they are each translated in to ^A

chrism01 · 11-08-2011, 05:33 PM

Tell us which logfile and provide a few lines for us to look at (in code tags).

SecretCode · 11-09-2011, 11:13 AM

Code:

sed 's/\x01/ /g'

may do it

casperdaghost · 11-09-2011, 12:42 PM

Here ya go - the littel boxes disappear on the code box - however if you open this upin a linux terminal, you will see a box between each value. it is ctrl-a separated

Code:

<8=FIX.4.29=00013335=D57=S56=INET49=MLCORZ34=2000752=20111104-16:13:4921=1114=N47=P11=BBK627255=AAPL59=544=140=238=50054=118=N76=SAVE10=009
>8=FIX.4.29=020835=834=949=INET56=MLCORZ52=20111104-16:13:4911=BBK627237=0109=MLCO17=-120=0150=039=055=AAPL54=138=50059=532=031=0.0151=50014=06=0.018=N44=1.000040=247=P29=476=SAVE58=Open198=1183310=166
<8=FIX.4.29=00013135=G57=S56=INET49=MLCORZ34=2000852=20111104-16:13:5411=BBK627455=AAPL44=140=238=20054=141=BBK627260=20111104-16:13:5410=175

SecretCode · 11-09-2011, 01:03 PM

Which logfile? Which program is writing these lines?

David the H. · 11-09-2011, 01:08 PM

The symbol you're seeing sounds like a unicode replacement symbol, displayed by some programs when the font doesn't support the current character or it's otherwise unrenderable. The numbers inside correspond to that character's unicode codepoint.

SecretCode's sed command appears to handle them, at least if it's gnu sed. You could also use bash's $'' quoting feature (ref: the quoting section of the manpage), with something like tr.

Code:

tr -d $'\x01' <oldfile >newfile

This removes them entirely. You could replace them with a new character instead, of course.

Edit: And here's a description of the replacement character, called the Unicode BMP Fallback:

http://en.wikipedia.org/wiki/Unicode_fallback_font