Help with unreadable comments

jaepi · 01-06-2008, 09:01 PM

Hello there. I'm having some sort of a problem with my text editor. It cannot read Japanese text. Is there any way or any available software in Linux that could convert these text? It all appear as junk.

David the H. · 01-07-2008, 12:37 AM

It might help if you told us exactly which text editor you're using. Perhaps it can handle the text if set up correctly. Do you have Japanese fonts installed? What language environment are you using? It's all in the details.

Well anyway, you should be able to use iconv to translate the files into utf-8, which most of the newer editors should be able to handle. You do need to know the encoding of the original file though.

BTW, this is the non-*nix programming forum. I believe you meant to post to the regular programming forum, no?

jschiwal · 01-07-2008, 12:53 AM

Often, you can use the "file" program to guess which encoding is used in a text file.

Code:

cat >testfile
Здравсвуийте
jschiwal@hpamd64:~> file testfile
testfile: UTF-8 Unicode text

However sometimes it will guess wrong. "iconv -l" will list the supported encoding schemes. Also look in wikipedia. They have a good article explaining different character encoding standards as well as charts of characters for variious standards.

Code:

cat testfile2
������������
jschiwal@hpamd64:~> iconv -f ISO-8859 -t UTF-8 testfile2 -o testfile3
iconv: conversion from `ISO-8859' is not supported
Try `iconv --help' or `iconv --usage' for more information.
jschiwal@hpamd64:~> iconv -f WINDOWS-1251 -t UTF-8 testfile2 -o testfile3
jschiwal@hpamd64:~> cat testfile3
Здравсвуийте

David the H. · 01-07-2008, 01:22 AM

Japanese text is most-commonly encoded in SHIFT-JIS, with JIS (ISO-2022-JP) and EUC-JP also being possible. The last one is mostly found in web documents.

knudfl · 01-07-2008, 06:23 AM

Quote:

Originally Posted by jaepi

Hello there. I'm having some sort of a problem with my text editor. It cannot read Japanese text. Is there any way or any available software in Linux that could convert these text? It all appear as junk.

高鐵賣的好散人潮

1925年高鐵賣的好散人潮

1925年に創業し、80年を越える歴史を持つラックスマンは、プリアンプ、パワーアンプ、プリメインアンプ、真空管アンプ等の高級オーディオ製品の優れたブランドとして
The above is random example text from leafpad and kwrite. Some chinese and the '1925 text' is
japanese.

jaepi · 01-07-2008, 07:46 PM

Quote:

Originally Posted by David the H.

It might help if you told us exactly which text editor you're using. Perhaps it can handle the text if set up correctly. Do you have Japanese fonts installed? What language environment are you using? It's all in the details.

Well anyway, you should be able to use iconv to translate the files into utf-8, which most of the newer editors should be able to handle. You do need to know the encoding of the original file though.

BTW, this is the non-*nix programming forum. I believe you meant to post to the regular programming forum, no?

I'm using gedit in ubuntu..

David the H. · 01-09-2008, 02:48 AM

Ok, apparently for gedit, you have to specify the encoding when you open the file. Start the program, then hit "file open". At the bottom of the file dialog should be a selector for character encoding, probably set to "auto detected". Click on that and use the "add or remove" dialog to add all the Japanese types to the menu. After that it should be able to auto-detect the encoding, or at least it will let you try all the options until you find one that works.

You can also launch it from the command line with a specific encoding using:

gedit --encoding=SHIFT_JIS textfile.txt

I think it's best though to convert and save all your text files to UTF-8 anyway (in addition to iconv, you can use the "save as" dialog in gedit to set the output encoding in the same way as above). It's the standard encoding of modern Linux distros and can handle all languages. The computing world is headed to unicode, and all these old encodings are slowly dying out.

Oh, and don't forget that you need to have a japanese-capable font installed and set in your preferences also.