unable to cat log file of type "UTF-8 Unicode (with BOM) text, with no line terminators"

Vernicronz · 08-26-2016, 01:24 AM

Hi all ,

I have some application logs of this type "UTF-8 Unicode (with BOM) text, with no line terminators". I am unable to cat or view it. when i do a vi on it..i get some strange chars.

#vi aplg.log
<feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><fef f><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><f eff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff> <feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><fef f><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><f eff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff> <feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff>

kindly advise. thanks.

HMW · 08-26-2016, 05:59 AM

Quote:

Originally Posted by Vernicronz

Hi all ,

I have some application logs of this type "UTF-8 Unicode (with BOM) text, with no line terminators". I am unable to cat or view it. when i do a vi on it..i get some strange chars.

#vi aplg.log
<feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><fef f><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><f eff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff> <feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><fef f><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><f eff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff> <feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff>

kindly advise. thanks.

Hi!

Try if this helps at all:

Code:

vim aplg.log

then in Vim (or Vi):

Code:

:set nobomb

Best regards,
HMW

Vernicronz · 08-27-2016, 09:18 AM

hi HMW

i have tried it..still no luck

<feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><fef f><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><f eff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff> <feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><fef f><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><f eff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff> <feff><feff><feff><feff><feff><feff><feff><feff><feff><feff><feff>
~
~
~
~
~
~
~
~
~
~
~
~
~
~
:set nobomb

malekmustaq · 08-27-2016, 10:52 AM

Does it contain anything?

Try

Code:

hexdump -C aplg.log

DavidMcCann · 08-27-2016, 11:23 AM

This file seems a bit odd. U+FEFF is the BOM. This is unnecessary (and not recommended) in UTF8. Also the two-byte encoding is UTF-16! And why are there so many — there should only be one BOM at the beginning. The answer to that is probably that it's been used as a zero-width non-breaking space, which is allowed but strongly deprecated.

Try using iconv to switch UFT16 to UTF8 and see if the result is readable.

Vernicronz · 08-29-2016, 09:50 PM

hi malekmustaq ,

[root@hst]# hexdump -C aplg.log
00000000 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
00000010 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
00000020 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|
00000030 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
00000040 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
00000050 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|
00000060 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
00000070 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
00000080 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|
00000090 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
000000a0 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
000000b0 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|
000000c0 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
000000d0 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
000000e0 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|
000000f0 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
00000100 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
00000110 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|
00000120 ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef |................|
00000130 bb bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb |................|
00000140 bf ef bb bf ef bb bf ef bb bf ef bb bf ef bb bf |................|

Vernicronz · 08-29-2016, 10:35 PM

hi DavidMcCan,

did the following ...

god knows what is this .... lol
looks like some japanase characters

[root@fcsdev2 tmp]# iconv -f UTF-16 -t UTF-8 aplg.log
믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯 뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻믯뾻[root@fcsdev2 tmp]#
[root@fcsdev2 tmp]#
[root@fcsdev2 tmp]#
[root@fcsdev2 tmp]#
[root@fcsdev2 tmp]# cat aplg.log
[root@fcsdev2 tmp]#
[root@fcsdev2 tmp]#
[root@fcsdev2 tmp]# cat aplg.log