difference in cat and vi output

keshavk · 07-31-2009, 04:40 AM

Hi,
I'm copying a output to a file. This output is from a terminal.
When I'm opening this file using cat and vi I'm getting diferent output.

e.g.

output of cat is like:
This is your output

while in vi editor output is like:

^M^[[3;0H
^M This is your output
^M^[[4;0H
There are a lot of unwanted character in the file like this

can anybody tell me why it is happening, and how to filter these unwanted character?
I've tried sed its unable to read vi output.

Thanks in advance.

Kesh

EricTRA · 07-31-2009, 04:44 AM

Hello,

Type at console

Code:

file yourfilename

to give you more information about the file itself. I have had these characters in files in the past also, but only when I copied a file from Windows to Linux.
Executing

Code:

dos2unix yourfilename

could solve the problem.

Kind regards,

Eric

karamarisan · 07-31-2009, 04:53 AM

It might be worth telling us what's producing that output. The only thing I can think of (and only because I was reading another thread about this earlier today) is if you're running ls and that's aliased to `ls --color=always` somewhere, vim won't know what to do with the escape sequences that cause text to be colored, but cat will.

Also, dos2unix probably won't help. This is a Linux program producing the output, no? And besides, I wasn't aware dos2unix did anything besides convert CR/LF line terminators to LF.

keshavk · 08-01-2009, 06:12 AM

Hi,
Thanks for your feedback...
The thing is this is the outputs of a linux terminal rather then window, so dos2unix, not helping...

When i checked the file type, the coming filetype is:

: ASCII English text, with very long lines, with CRLF, CR line terminators, with escape sequences

Any idea how to remove these unwanted character,
Thanks in advance..

K.

joeBuffer · 08-01-2009, 01:19 PM

This happens when you use the script command, the output file (default typescript) is exactly like what you're describing. I don't have any strange settings at all, that's just how it is.

Quote:

file typescript
typescript: ASCII text, with CRLF, LF line terminators, with escape sequences

I don't know which programs would do that ...

keshavk · 08-02-2009, 12:06 PM

Hi,
Any idea how to remove CRLF, CR line terminators, from a file in linux.

K.

pwc101 · 08-02-2009, 12:12 PM

See:

man dos2unix

Or, if that doesn't work:

man fromdos

catkin · 08-02-2009, 01:31 PM

Hello keshavk

Your problem is impossible to solve completely. It has been asked many times before in connection with the script command and terminal spying programs. What you have is the sequence of bytes that went to the terminal; what you want to get is a text version of what appeared on the terminal. That is fundamentally impossible; the terminal is a dynamic device while your text file is static.

As long as nothing complex happened in the terminal session (switch to full screen mode followed by user interaction, command recall, command editing, title setting etc) then a "good enough" solution is possible -- filtering out the bytes that controlled colour, cursor position, rang bells etc -- which may render the text sufficiently readable for your purposes.

An alternative approach -- which has the advantage of not being fundamentally impossible -- would be to replay the captured bytes by feeding them to a terminal thus reproducing the session.

Best

Charles

gregorian · 08-02-2009, 01:32 PM

If cat is working for you, just type cat filename > out.txt. You'll get your results in the text file. Hope this helps.

joeBuffer · 08-02-2009, 01:43 PM

Yeah, when I read catkin's post and thought about what you were saying just now, I was going to suggest cat, also.
It would work, with redirection. It works for me, when using the typescript files.

stanlo45 · 08-03-2009, 12:34 AM

This might help with removing "^M" from a file with vi/vim
Type in the following in vi

:%s/<ctrl>v<ctrl>m//g<ENTER>

Stan

keshavk · 08-03-2009, 01:05 AM

Hi Stan,
Thanks for your help.....it helped me in removing many junk...
rest i'll try to solve.

K.

SwaJime · 08-03-2009, 01:14 PM

Why are you using vi? Are you trying to edit the file? Or are you trying to view the file?

If you are only trying to view the file, then you can use "less -fr" and the color sequences will be put to use.

To remove the escape sequences, you can use this:

Code:

$ cat dobit.sh 
#!/bin/bash
#
# Copyright SwaJime's Cove, 2009; all rights reserved
#

# Based on data obtained from http://www.gnu.org/software/teseq/manual/html_node/Escape-Sequence-Recognition.html
#
                                # A control sequence starts with
CSI='\x1b\['			# the two-character csi escape sequence ‘Esc [’, followed by
Rp='[0-9:;<=>?]'		# an optional sequence of parameter bytes in the range x30–x3F,
Ri='[- !\"#$%&'\''()*+,./]'	# an optional sequence of intermediate bytes in the range x20–x2F,
Rf='[]@A-Z[\\^_`a-z{|}~]'	# and a final byte in the range x40–x7e.
				# The set of standard control sequence functions are defined in Ecma-48 / ISO/IEC 6429. 

ls --color=always > myFile.txt
cat myFile.txt | sed -e 's/'"$CSI$Rp"'*'"$Ri"'*'"$Rf"'//g'

karamarisan · 08-03-2009, 01:21 PM

Quote:

Originally Posted by SwaJime

Code:

# Copyright SwaJime's Cove, 2009; all rights reserved

Good of you to provide code, but that's not really in the spirit of this place, is it?

kunstkopf · 08-04-2009, 09:24 AM

In Unix, the character transliteration tool 'tr' exists
almost from the beginning.

To remove CR=13(dec)=15(oct):

tr -d '\015' < infile.txt > outfile.txt

To remove LF=NL=10(dec)=12(oct)

tr -d '\012' < infile.txt > outfile.txt

To remove both:

tr -d '\015\012' < infile.txt > outfile.txt

... all just from the command line, no programming. You can also transliterate
other characters, of course, for instance a->d, b->e, c->f by

tr 'a-c' 'd-f' < infile.txt > outfile.txt

In Linux, there will be mnemonics for meaningful character
subsets like '[:cntrl:]' and '[: punct:]'