[SOLVED] How do I get rid of some weird stuff in a text file?

lethalfang · 11-09-2011, 02:28 PM

I've made a .csv out of some spreadsheet, and it seems like there are a couple of trailing spaces that I need to get rid of.
So here is the weird thing. What appears to be a couple of trailing spaces, aren't exactly space characters.

Code:

cat textfile

AAK1
AATK
ABL1
ABL2
ACTR2

Code:

cat -v textfile

AAK1M-BM-  
AATKM-BM-  
ABL1M-BM-  
ABL2M-BM-  
ACTR2M-BM-

Now, the question is, what the heck is "M-BM-" and how do I get rid of it?

Thanks in advance.

kbscores · 11-09-2011, 02:36 PM

vi the file and do a :set list

It will turn on special characters in file. Then you could do an insert or use regular expressions to delete them.

To turn off special characters use :set nolist

David the H. · 11-09-2011, 03:06 PM

The listing on this page for the solaris version of cat has an extended explanation of the -v option.

http://www.softpanorama.org/Tools/cat.shtml

Code:

-v  Non-printing characters (with the exception of tabs, new-lines and form-feeds) are printed visibly. ASCII control characters (octal 000 - 037) are printed as ^n, where n is the corresponding ASCII character in the range octal 100 - 137 (@, A, B, C, . . ., X, Y, Z, [, \, ], ^, and _); the DEL character (octal 0177) is printed ^?. Other non-printable characters are printed as M-x, where x is the ASCII character specified by the low-order seven bits.

So these two characters appear to be outside of the basic ascii character set, but I have no idea exactly what they are.

Assuming that the pattern is uniform, with every line having two extra characters, you could try removing them with sed:

Code:

sed 's/..$//' file

lethalfang · 11-09-2011, 05:25 PM

Quote:

Originally Posted by David the H.

The listing on this page for the solaris version of cat has an extended explanation of the -v option.

http://www.softpanorama.org/Tools/cat.shtml

Code:

-v  Non-printing characters (with the exception of tabs, new-lines and form-feeds) are printed visibly. ASCII control characters (octal 000 - 037) are printed as ^n, where n is the corresponding ASCII character in the range octal 100 - 137 (@, A, B, C, . . ., X, Y, Z, [, \, ], ^, and _); the DEL character (octal 0177) is printed ^?. Other non-printable characters are printed as M-x, where x is the ASCII character specified by the low-order seven bits.

So these two characters appear to be outside of the basic ascii character set, but I have no idea exactly what they are.

Assuming that the pattern is uniform, with every line having two extra characters, you could try removing them with sed:

Code:

sed 's/..$//' file

Thanks. I've no idea where those weirdo characters come from... but your suggestion worked, so hooray!.