Linux not reading non-ASCII characters

tailangong · 04-26-2010, 10:58 AM

Hi all,

Well, I have a web application in Linux server. All my Java codes are there.
FYI, whenever user entered non-ASCII characters(e.g. ∞,€,™) in a text field in my web application, and I check the log of my Java code in Linux server, it returns weird characters.

Suppose user entered ∞ in the text field. I should get ∞ in my log too. However, I got weird characters in return.

Any idea? Is this a Linux bug?

MTK358 · 04-26-2010, 11:32 AM

Quote:

Originally Posted by tailangong

Is this a Linux bug?

This has absolutely nothing to do with the Linux OS.

Are you inputting the data using one character encoding and viewing it with another?

Sergei Steshenko · 04-26-2010, 12:05 PM

Quote:

Originally Posted by tailangong

Hi all,

Well, I have a web application in Linux server. All my Java codes are there.
FYI, whenever user entered non-ASCII characters(e.g. ∞,€,™) in a text field in my web application, and I check the log of my Java code in Linux server, it returns weird characters.

Suppose user entered ∞ in the text field. I should get ∞ in my log too. However, I got weird characters in return.

Any idea? Is this a Linux bug?

If you expect a function in a programming language to read something in some manner, your expectations should be based on the official documentation of the function.

So, based on what official documentation, specifically, on which part of it do you expect the characters to be read the way you expect them to be read ?

tailangong · 04-26-2010, 08:57 PM

Quote:

Originally Posted by MTK358

This has absolutely nothing to do with the Linux OS.

Are you inputting the data using one character encoding and viewing it with another?

Everything entered by the user will be passed to my Java program to be inserted into database. On HTML side, I use UTF-8 as my character encoding. In Java, I use request.getParameter to get the value entered by the user. I don't think I have any encoding for that.

ps: Im working from the terminal, can terminal captures non-ASCII characters? Sorry, Im new in Linux.

Sergei Steshenko · 04-26-2010, 09:11 PM

Quote:

Originally Posted by tailangong

...

ps: Im working from the terminal, can terminal captures non-ASCII characters? Sorry, Im new in Linux.

It depends on the terminal and on the way it is configured.

tailangong · 04-26-2010, 09:22 PM

Hi Sergei,

Do you have any idea on how to check this?
What command do I need to check those configurations?

graemef · 04-27-2010, 01:50 AM

There are several environment variable that might give you an idea as to how the terminal is set up. Try the following commands from your terminal:

$ echo $LANG
$ echo $LC_ALL
$ echo $LC_CTYPE

tailangong · 04-27-2010, 01:58 AM

I found out that my page was encoded in iso-8859-1 and echo $LANG in linux returns en_US.UTF-8.
MTK358 was right about it. However, I tried to change my page to use utf-8 as character encoding but to no avail.
Problem still persists. Why??

graemef · 04-27-2010, 02:11 AM

Depending upon the whole application you need to ensure that everything is using the right encoding. From HTTP, Java environemnt, database(?).

Grab something like wireshark and see what is actually being sent along the wire, that should help to narrow the problem down.

tailangong · 04-27-2010, 03:30 AM

Wireshark?? Hmm, I don't have enough time to learn a new tool now.
I was just hoping on how to solve this encoding problem

tailangong · 04-27-2010, 04:40 AM

For your information, when I execute System.out.println("\u20AC"), I should have get € (Euro sign symbol). But what I get is a bunch of weird characters such as ï¿½
Can help?

graemef · 04-27-2010, 05:02 AM

When I run the code I get a Euro sign.

what do you get with the following?

Code:

import java.nio.charset.Charset;

public class unicodeTest
{
   public static void main(String[] args)
   {
      System.out.println(Charset.defaultCharset());
      System.out.println("\u20AC");
   }
}

Sergei Steshenko · 04-27-2010, 05:05 AM

Quote:

Originally Posted by tailangong

Hi Sergei,

Do you have any idea on how to check this?
What command do I need to check those configurations?

To have an idea I need to know what kind of terminal you are using. Also, see http://www.linuxquestions.org/questi...2/#post3948904 .

tailangong · 04-27-2010, 05:25 AM

Hi graemef,

I got this ---> ï¿½

Hi Sergie,

FYI,

LANG=en_US.UTF-8
HOSTTYPE=i386-linux
VENDOR=intel
OSTYPE=linux
MACHTYPE=i386

graemef · 04-27-2010, 05:28 AM

When I run the same program I get:

Code:

UTF-8
€

Does this mean that your Java implementation doesn't have a defaultCharset?