Slackware and UTF-8 encoding

gda · 07-29-2019, 12:48 PM

Hi all,

not sure this is really related to Slackware.... Anyway I have some problem in writing a simple C code handling UTF8 encoded strings properly. As far as I have understood C libraries just use the encoding defined in the environment they live in. In particular for Slackware the default encoding is ISO-8859-2 and so I suppose this encoding is used.

Indeed the following code:

Code:

char test[40];
test[0]=0;
strcat(test,"øèéö");
fprintf(stdout,"test=%s\n",test);

when run in an terminal with ISO-8859-2 encoding returns:

test=řčéö

This is totally expected as the UTF8 characters "ø" and "è" are mapped to different character in the ISO-8859-2 table ("ř" and "č" respectively).

The problem is that if I add at the very beginning of the above code the line

Code:

setlocale(LC_ALL,"es_US.utf8");

and I run the the code in a UFT8 terminal I got:

test=��

So why the UTF8 string is not displayed as written into the code? I have the feeling I'm missing something very basic but I cannot figure out what it is...

Thanks in advance for your help!

Labinnah · 07-29-2019, 02:29 PM

This is not utf-8 string. You written it as ISO-8859-2 and it stay in that form. By setlocale function you only set how terminal interprets strings. It doesn't transcode them to utf-8. If you want display uft-8 string you must write them as utf-8 (in editor you use). Or you can use iconv function

Code:

man 3 iconv

business_kid · 07-29-2019, 03:02 PM

I can't offer much help, but I can sympathise. I'm in Ireland.

Some smart-alec wrote that "War is God's way of teaching Americans Geography!"
Slackware isn't marvellously aware of locale eccentricities. I'm afraid I can only suggest you try some other locales. The various attempted ones for Ireland (across several programs) are:
en_IE
en_IE.utf8
en_IE@euro
en_IE.iso8859-15
etc. etc.

es_US is a dead duck unless it's in glibc locales. But there must be others close. Don't worry about having the basic charset, and don't forget the right-hand 'Alt' key or Alt_Gr. I'd start by changing the 'US' bit. I chose the locale that gave me the best compromise for my situation, where I want mathematical keys and áéíóú for Irish words. I also get '…ºª¯ῶ÷€©™¡≤≥¦×'(a multiply different from x)and no doubt a few others, but I lose the cents sign, and our € (euro) currency has cents.

Be aware also that you can set each LC_ setting differently. You usually hear of setting LC_ALL, but that sets them all. You can pick and choose, but it's up to you.

ehartman · 07-29-2019, 04:50 PM

Quote:

Originally Posted by gda

In particular for Slackware the default encoding is ISO-8859-2 and so I suppose this encoding is used.

No, nowadays the default encoding is UTF-8:

Quote:

# en_US.UTF-8 is the Slackware default locale.
and some lines further
export LANG=en_US.UTF-8

# 'C' is the old Slackware (and UNIX) default, which is 127-bit ASCII
# with a charmap setting of ANSI_X3.4-1968.
#export LANG=C

(from the file /etc/profile.d/lang.sh.new, out of the etc- package).
Note that the 2nd "export" is commented out (as are all other options).

If your system has ISO-8859-2 as a default, you did that yourself, maybe when you installed your system.

Then the "str*" functions are not magical or so: when iso-8859 characters go into them, you will get iso-8859 chars printed, INdependant of your locale.
So to get utf-8 ones: IN an editor with the utf-8 locale, create the characters in the
"strcat(test,"øèéö");"
line, so that those chars are utf-8 (that is: multibyte) and not iso-8859 (which is a single byte character set, it only has 256 values to display, that is why there are multiple iso-8859-* standards).utf-8 has characters that can be UP to 4 bytes, so it can display many more characters, but it does mean the same code(s) mean different things, as values 128 through 255 are used differently in iso-8859 from utf-8
See for more info
man iso-8859-2 or
man utf-8