why fwprintf writes chars instead of wchars on Linux system

senthilpeace · 11-18-2009, 04:03 AM

On linux system I noticed that 1 byte characters are written inside the file instead of the 4 bytes of the wchar_t. Actually I wanted to write a data in unicode char format in the output file instead of 1 byte char format.

Could you please through some light if anyone has resolved this kind of issue.

Note: I'm using a gcc compiler to compile the C code. But same code was working fine on Windows OS and got the result as expected.

code:

#include <wchar.h>
#include <stdio.h>

int main( int argc, char *argv[])
{
wchar_t *bigString = L"1234567787955_-+:";
FILE *fp = fopen( "bigString.hex", "wb");
fwprintf( fp, L"%ls", bigString);
fclose( fp);
}

smeezekitty · 11-18-2009, 05:18 PM

4 byte chars are a HUGE waste of space (4x) and i assume that glibc thinks its not required.
thats 4000000000 different chars possable.

ta0kira · 11-18-2009, 05:42 PM

I'm pretty sure there was a thread about the same thing in the last year. I'm not sure what came of it.
Kevin Barry

neonsignal · 11-19-2009, 08:18 PM

The difference is not in the compiler, but in the way in which the locale is handled by the write to the file. On GNU/Linux systems you would typically have utf8 encoding rather than utf16.

You should probably first set the locale to a unicode one (use 'locale -a' to list the ones you have available). You don't need to open the file in binary mode.

Code:

#include <wchar.h>
#include <stdio.h>
#include <locale.h>
int main( int argc, char *argv[])
{
	setlocale( LC_ALL, "en_US.utf8");
	const wchar_t *bigString = L"1234567787955_-+:";
	FILE *fp = fopen( "bigString.hex", "w");
	fwprintf( fp, L"%ls", bigString);
	fclose( fp);
}

If you need the file to be utf16 encoded rather than utf8 for compatibility reasons, you can either call the iconv function from the program, or use the iconv program:

Code:

iconv -f utf-8 -t utf-16 bigString.hex

senthilpeace · 11-19-2009, 11:47 PM

Thanks for your input. My requirement is to read the UTF-16BE file format which has the Japanese unicode char filed in the data field, to process the input data file (ie. UTF-16BE) and to write a data again in UTF-16BE file format.

The issue here is I'm not able to write a data file in UTF-16BE under GNU/Linux and also I have checked the availability of unicode set under GNU/Linux.

ie. used the 'locale -a' command and got the following unicode set list from GNU/Linux.

en_US.iso88591
en_US.iso885915
en_US.utf8

Is there anyway to set the unicode UTF-16BE on GNU/Linux system?

Thanks!
Senthil

Quote:

Originally Posted by neonsignal

The difference is not in the compiler, but in the way in which the locale is handled by the write to the file. On GNU/Linux systems you would typically have utf8 encoding rather than utf16.

You should probably first set the locale to a unicode one (use 'locale -a' to list the ones you have available). You don't need to open the file in binary mode.

Code:

#include <wchar.h>
#include <stdio.h>
#include <locale.h>
int main( int argc, char *argv[])
{
	setlocale( LC_ALL, "en_US.utf8");
	const wchar_t *bigString = L"1234567787955_-+:";
	FILE *fp = fopen( "bigString.hex", "w");
	fwprintf( fp, L"%ls", bigString);
	fclose( fp);
}

If you need the file to be utf16 encoded rather than utf8 for compatibility reasons, you can either call the iconv function from the program, or use the iconv program:

Code:

iconv -f utf-8 -t utf-16 bigString.hex

neonsignal · 11-20-2009, 01:13 AM

Quote:

Is there anyway to set the unicode UTF-16BE on GNU/Linux system?

Microsoft Windows originally used UCS-2 (and now UTF-16) because of their early implementation of Unicode. GNU/Linux uses the UTF-8 alternative. If you wish to use Unicode encodings other than UTF-8 on Linux, you use conversion functions such as iconv, which will handle any encoding you can think of and more beside. Note that UTF-8 and UTF-16 can both be used to represent any of the Unicode code points, and are designed to be functionally equivalent.

If you are porting code from Windows, you also need to be careful of the wchar_t definition, since on gcc it is 32 bits (not 16 bits). See for example the output from this code:

Code:

#include <wchar.h>
#include <stdio.h>
#include <string.h>
int main( int argc, char *argv[])
{
	const wchar_t *bigString = L"1234567787955_-+:";
	FILE *fp = fopen( "bigString.hex", "wb");
	fwrite( bigString, sizeof(wchar_t), wcslen(bigString), fp);
	fclose( fp);
}

There are also wcrtomb and mbrtowc conversion functions, but again, beware of implementation differences between different systems.

senthilpeace · 11-20-2009, 05:52 AM

Thanks for your detail explanation about unicode used in Windows and GNU Linux.

I'll give you little more details about my requirement.

The input data file format (i.e. UTF-16BE) should be like this.

infield1~infield2~infield3~Japanese unicode char (UTF-16BE with CSV of "~")

Would need to feed this data into C component running on GNU Linux and the output file format should be like this

outfield1~outfield2~outfield3~Japanese unicode char (UTF-16BE with CSV of "~")

The question here is how to retain the Japanese unicode char while writing into output file using the wide char library funtion in C on GNU Linux system?

Thanks!
Senthil

Quote:

Originally Posted by neonsignal

Microsoft Windows originally used UCS-2 (and now UTF-16) because of their early implementation of Unicode. GNU/Linux uses the UTF-8 alternative. If you wish to use Unicode encodings other than UTF-8 on Linux, you use conversion functions such as iconv, which will handle any encoding you can think of and more beside. Note that UTF-8 and UTF-16 can both be used to represent any of the Unicode code points, and are designed to be functionally equivalent.

If you are porting code from Windows, you also need to be careful of the wchar_t definition, since on gcc it is 32 bits (not 16 bits). See for example the output from this code:

Code:

#include <wchar.h>
#include <stdio.h>
#include <string.h>
int main( int argc, char *argv[])
{
	const wchar_t *bigString = L"1234567787955_-+:";
	FILE *fp = fopen( "bigString.hex", "wb");
	fwrite( bigString, wcslen(bigString), sizeof(wchar_t), fp);
	fclose( fp);
}

There are also wcrtomb and mbrtowc conversion functions, but again, beware of implementation differences between different systems.