Letters and chars in Linux

Nominal Animal · 03-05-2011, 01:31 PM

Quote:

Originally Posted by nime

I didn't see any "unsupported" characters until now, everything is showed properly in all 3 my encodings. But I work only with letters, maybe this is applied for special signs like ® © ½ ± ° ¥ £ ¢ € ² ? If not maybe I can //TRANSLIT simply "forget"?

Don't worry, there is no downsides in your case for always using //TRANSLIT for the target encoding. If you don't, and there happens to be a character that cannot be converted, the conversion will fail. You can also use //IGNORE to skip those characters instead.

Here is a function that uses your original interface. Note that it may be slow, because it opens and closes the iconv handle for each string separately. Like I said above, it is always good to append either //TRANSLIT or //IGNORE to the target character set name, otherwise the function will return NULL if there are inconvertible characters. Finally, this will grow and optimize the result string dynamically to exact length. It will always allocate enough additional space for the rest of the input string, plus CONV_EXTRA bytes. If you want, you can set CONV_EXTRA to a larger value, so it will initially allocate more memory. It will still optimize the size via a realloc() call, so there is very little harm in having CONV_EXTRA a bit larger, maybe 256 or 1024.

Code:

#include <stdlib.h>
#include <string.h>
#include <iconv.h>
#include <errno.h>

#define  CONV_EXTRA  16

char *convert(char const *const from, char const *const to, char const *const in)
{
    iconv_t     handle;
    size_t      insize, inleft;
    char       *inends;
    size_t      outsize, outleft;
    char       *out, *outends;

    /* Invalid character set names? */
    if (!from || !to || !*from || !*to) {
        errno = EINVAL;
        return NULL;
    }

    /* No string to convert? */
    if (!in)
        return NULL;

    /* Get a handle for the conversion. */
    handle = iconv_open(to, from);
    if (handle == (iconv_t)-1) {
        /* This conversion is not supported. */
        errno = ENOTSUP;
        return NULL;
    }

    /* Do the conversion. Grow the buffer if not large enough. */
    insize  = strlen(in);
    out     = NULL;
    outsize = insize + (size_t)CONV_EXTRA;
    while(1) {

        /* Prepare for the conversion. */
        inends = (char *)in;
        inleft = insize;

        /* Allocate a new output buffer. */
        out = malloc(outsize + 1);
        if (!out) {
            iconv_close(handle);
            errno = ENOMEM;
            return NULL;
        }
        outends = out;
        outleft = outsize;

        /* Do the conversion. */
        if (iconv(handle, &inends, &inleft, &outends, &outleft) == (size_t)0)
            break;

        /* Error? */
        if (errno != E2BIG) {
            int const   error = errno;
            free(out);
            iconv_close(handle);
            errno = error;
            return NULL;
        }

        /* Grow the output buffer size. */
        free(out);
        outsize = outsize + inleft + (size_t)CONV_EXTRA;
    }

    /* Reallocate the string to optimal size. */
    if (outleft && outsize >= outleft) {
        char *tmp;
        outsize -= outleft;
        tmp = realloc(out, outsize + (size_t)1);
        if (tmp)
            out = tmp;
    }

    /* Append EOS. */
    out[outsize] = 0;

    /* Close the conversion handle. */
    iconv_close(handle);

    return out;
}

Quote:

Originally Posted by nime

Hmm, I think manipulation with locales would be a better (easier) solution here if I can get result with them. Maybe sometimes I would need to do some "sort by letters" in "strange" locale and order in ASCII table is not proportional with place in alphabet order. For example letters in my alphabet is like this: "ABCČĆD(Dž)ĐEFGHIJ..."
So for this reasons temporary change locale can give better results (I think).
Additionally, difference between lowercase 'č' and uppercase 'Č' in ASCII is also 32 (200,232 in cp1250) but not in IBM852 (179, 152). So, better is run away for doing this "by hand" like I use to do in DOS.

First, make sure you have the locale files defined. In Linux, locale -a lists them, and localedef can be used to create or install new ones. The C side is fortunately quite easy; the only limitation is that the locale setting is process wide, so if you use threads, it will change the locale for all threads.

The locale setting is divided into multiple categories, so you can only set e.g. LC_COLLATE category for string collation. Here is an example which uses the locale hr_HR.IBM852 to compare two strings:

Code:

#include <locale.h>

char *oldlocale, *string1, *string2;
int   result;

oldlocale = setlocale(LC_COLLATE, "hr_HR.IBM852");

result = strcoll(string1, string2);

setlocale(LC_COLLATE, oldlocale);

if (result < 0)
    printf("%s < %s\n", string1, string2);
else if (result > 0)
    printf("%s > %s\n", string1, string2);
else
    printf("%s == %s\n", string1, string2);

In the above code, I use the oldlocale variable, so that the locale is only temporarily changed.
In your code, you don't need to do that; you can just set the locale to whatever you happen to need. It is also local to the program, and will not change any system settings or anything, so you can use it in your program pretty freely.

The strcoll function works just like strcmp, except it uses the LC_COLLATE locale category.

Note that the tolower_ibm852() and toupper_ibm852() functions I listed earlier are both thread-safe, and not dependent on the locale settings. If you need case sensitive and insensitive IBM852 comparison functions (strcmp_ibm852() and strcasecmp_ibm852()), I can show them for you; the code is very much like tolower_ibm852() and toupper_ibm852(), except with two different tables.

The difference between these hardcoded functions and locale functions is that these are self-contained, and do not depend on any other things. In fact, if you have issues getting iconv support working for all your target platforms, I could quite easily write hardcoded conversion, sorting, and case changing functions for ISO-8859-2 and IBM852, with conversion to and from UTF-8 and ISO-8859-1, if you like.

Hope this helps!

nime · 03-05-2011, 03:54 PM

Nominal,
I am truly shocked with your deep knowing of this (very complicated) theme. Actually, I shamed to ask any additional help because I understand that I get more than "reasonable" help for free from you.

And also, I hope you see that level of your examples and wide help go far beyond my ability to understand what I am doing. For now I wouldn't like to know about iconv conversions more than is need for my programs because I already have mess in head from this. From more informations now I can only have more damage.
Go to developiong CONV_EXTRA for my strings is also too much. Especially what now I have proper letters on console, file and GTK entry. I can easily now add functions for 8859-2 by myself. What more can I want?
I would simply add //TRANSLIT if this is "has to be".

Thank you for pointing me to potentional problems with changing of locales and I undestand advantages of hardcodins when CP IBM852 is constant. So I try to apply your example but won't work.

I added uppercase and lowercase arrays in header file and declarations for functions, like this:

Code:

const unsigned char uppercase_ibm852[256] = {
      0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,...

static inline char *toupper_ibm852(char *const string);
static inline char *tolower_ibm852(char *const string);
// when I don't even know what 'is static inline char *'
// and what is 'char *const'. I already used only 'const char*'

Then I copy functions to my functions code file and I call functions like this:

Code:

// fstr is string from GTK textbox.
// can be in 'mixedcase' so I try both lower and uppercase..

    char* tp;
    tp = utf8_to_ibm852(fstr);
    static inline char* new_upper = toupper_ibm852(strdup(tp));
    static inline char* new_lower = tolower_ibm852(strdup(tp));

And I get bunch of errors...

Quote:

C:\RTM\gtktext\cHEADER.h|50|warning: 'toupper_ibm852' declared 'static' but never defined|
C:\RTM\gtktext\cHEADER.h|51|warning: 'tolower_ibm852' declared 'static' but never defined|
C:\RTM\gtktext\cCODE.c||In function 'invsearch':|
C:\RTM\gtktext\cCODE.c|20|warning: variable 'new_upper' declared 'inline'|
C:\RTM\gtktext\cCODE.c|20|error: initializer element is not constant|
C:\RTM\gtktext\cCODE.c|21|warning: variable 'new_lower' declared 'inline'|
C:\RTM\gtktext\cCODE.c|21|error: initializer element is not constant|
C:\RTM\gtktext\cCODE.c|21|warning: unused variable 'new_lower'|
C:\RTM\gtktext\cCODE.c|20|warning: unused variable 'new_upper'|
||=== Build finished: 2 errors, 6 warnings ===|

Of course, header is included in code file.
Please help to get this working.

I was also try to reduce code for not have to same function six or more times with your recommendation:

Code:

iconv_t  utf8_to_ibm852 = conversion("UTF-8",      "IBM852//TRANSLIT");
iconv_t  utf8_to_iso2   = conversion("UTF-8",      "ISO-8859-2//TRANSLIT");
iconv_t  ibm852_to_utf8 = conversion("IBM852",     "UTF-8");
iconv_t  ibm852_to_iso2 = conversion("IBM852",     "ISO-8859-2//TRANSLIT");
iconv_t  iso2_to_utf8   = conversion("ISO-8859-2", "UTF-8");
iconv_t  iso2_to_ibm852 = conversion("ISO-8859-2", "IBM852//TRANSLIT");
//-----------
C:\RTM\gtktext\cFUNC.c|195|warning: implicit declaration of function 'conversion'|
C:\RTM\gtktext\cFUNC.c|195|warning: initialization makes pointer from integer without a cast|
C:\RTM\gtktext\cFUNC.c|195|error: initializer element is not constant|
... and so 6 times...
||=== Build finished: 6 errors, 7 warnings ===|

//'implicit declaration' appears mostly when something isn't declared,
//but in same file I already have: static iconv_t iconv_cp1250_to_ibm852 = (iconv_t)-1;
//which works. What do I missed?

Quote:

If you need case sensitive and insensitive IBM852 comparison functions (strcmp_ibm852() and strcasecmp_ibm852()), I can show them for you; the code is very much like tolower_ibm852() and toupper_ibm852(), except with two different tables.

The difference between these hardcoded functions and locale functions is that these are self-contained, and do not depend on any other things. In fact, if you have issues getting iconv support working for all your target platforms, I could quite easily write hardcoded conversion, sorting, and case changing functions for ISO-8859-2 and IBM852, with conversion to and from UTF-8 and ISO-8859-1, if you like.

I would like to see case sensitive and insensitive IBM852 comparison functions (strcmp_ibm852() and strcasecmp_ibm852()).
For now I use following function which I pick up from the net. Works fast and nice, but not on 852 so I must recode first to work only in windows..

Code:

const char *instrnocase(const char *haystack, const char *needle)
{
   if (!*needle){
      return haystack;}
   for (; *haystack; ++haystack){
      if (toupper(*haystack) == toupper(*needle)){
         /* matched starting char -- loop through remaining chars. */
         const char *h, *n;
         for (h = haystack, n = needle; *h && *n; ++h, ++n){
            if (toupper(*h) != toupper(*n)){
               break;}}
         if (!*n) /* matched all of 'needle' to null termination */
          {return haystack;} /* return the start of the match */
      }}
   return 0;
}
//-------------
//Call:
    const char *found = instrnocase(Ip, tp);
    if (found)
        {
          finded ++;
          gtk_entry_append_text (GTK_ENTRY (entry), cp);
          ...

So, for sorting I don't know. Seems complicated. In DOS I made sorts with "lookup" string for properly ordering.
And now I think confused about changing locales...
This is something what should be well tested before any decision.
Thank you for example how to change just a essential part of locale.

Nominal Animal · 03-05-2011, 06:59 PM

Quote:

Originally Posted by nime

And also, I hope you see that level of your examples and wide help go far beyond my ability to understand what I am doing.

Fair enough. I'll try to limit the scope of my answers to fit your needs better.

Quote:

Originally Posted by nime

I would simply add //TRANSLIT if this is "has to be".

Yes. It just makes sure that if there is something unexpected, the results are as close to correct as possible.

Quote:

Originally Posted by nime

I added uppercase and lowercase arrays in header file and declarations for functions, like this:

Code:

const unsigned char uppercase_ibm852[256] = {
      0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,...

static inline char *toupper_ibm852(char *const string);
static inline char *tolower_ibm852(char *const string);
// when I don't even know what 'is static inline char *'
// and what is 'char *const'. I already used only 'const char*'

Sorry!

It's my fault. static inline designates the function to be a local helper function, only used in the current source file. Of course it is wrong here. If you remove the static inline from everywhere, it should work.

As to the types:

char * means "pointer to char": a pointer to a string.
const char * and char const * both mean "pointer to constant char": a pointer to a fixed string. (You can change the pointer, but you cannot change the characters in the string.)
char *const means "a constant pointer to char": a fixed pointer to a string. (You cannot change the pointer, but you can change the individual characters in the string.)
const char *const and char const *const both mean "a constant pointer to constant char": a fixed pointer to a fixed string.

The reason I've marked the pointers fixed is just to help the compiler do better code. The string value is changed, but the pointer is not. This designation should help the compiler.

Current C compilers almost always detect the above situation even without the help, so it is just a detail. However, I've found I understand and remember the intent of the functions better, when the parameters are meticulously marked const. I've even caught a number of bugs that way.

Quote:

Originally Posted by nime

I was also try to reduce code for not have to same function six or more times with your recommendation:

Code:

iconv_t  utf8_to_ibm852 = conversion("UTF-8",      "IBM852//TRANSLIT");
iconv_t  utf8_to_iso2   = conversion("UTF-8",      "ISO-8859-2//TRANSLIT");
iconv_t  ibm852_to_utf8 = conversion("IBM852",     "UTF-8");
iconv_t  ibm852_to_iso2 = conversion("IBM852",     "ISO-8859-2//TRANSLIT");
iconv_t  iso2_to_utf8   = conversion("ISO-8859-2", "UTF-8");
iconv_t  iso2_to_ibm852 = conversion("ISO-8859-2", "IBM852//TRANSLIT");
//-----------
C:\RTM\gtktext\cFUNC.c|195|warning: implicit declaration of function 'conversion'|
C:\RTM\gtktext\cFUNC.c|195|warning: initialization makes pointer from integer without a cast|
C:\RTM\gtktext\cFUNC.c|195|error: initializer element is not constant|
... and so 6 times...
||=== Build finished: 6 errors, 7 warnings ===|

Sorry again!

toupper_ibm852 and tolower_ibm852 are self-standing functions; you can call them anytime you want, without any setup. They do not depend on any other code; everything they needed was in that one code box. (If you remove the static inline from it.)

Similarly, the convert function is self-standing; you only need the code I showed in that code box, and link in iconv when compiling. If you use it, you can replace all the conversion calls you have right now with it. The only iconv_t you should see in all your source code would be the iconv_t handle; line in the convert function.

Quote:

Originally Posted by nime

I would like to see case sensitive and insensitive IBM852 comparison functions (strcmp_ibm852() and strcasecmp_ibm852()).
For now I use following function which I pick up from the net. Works fast and nice, but not on 852 so I must recode first to work only in windows..

I'll put all the conversion functions and the IBM852 functions into one header and one code file, so it'll be easier for you to use. It will replace all the code I've shown you before, so you'll need to modify your existing code a bit, if you decide to use it. It will be very similar, so your changes should be very small and easy to do. I'll do that in a new message in a bit.

nime · 03-06-2011, 03:50 AM

I get yours extra-excellent upper/lower 852 conversion to work with removing "static inline" and putting array in code file. Then I reorganize my program with new header and code file just for conversions because I will need them often.

Now I have this:

Code:

// fstr is UTF8 from Gtk textbox

    char* tp;
    tp = utf8_to_ibm852(fstr);  // convert fstr to 852
    char* new_upper = toupper_ibm852(strdup(tp)); //uppercase 852 string
    ...

    char* n_upper = toupper_ibm852(strdup(Ip));   // uppercase string from data file
    const char *found = strstr(n_upper, new_upper); // pure C strstr
    if (found)   // IT FINDS, IT FINDS!!
        {
          finded ++;

       // fill GTK entry with previously converted UTF8 string 'cp'
       // from 852 data file
          gtk_entry_append_text (GTK_ENTRY (entry), cp);

So, this is it!. No need for additional functions!

nime · 03-06-2011, 07:19 AM

And now, here is my performances!

I have data file over 3MB with 100.000 records which contains 43.761 filled rows with various data, written with QB45 and VB data structures (types) which don't know for null termination so I do this during reading of my records. All string data is in cp852!

I have GTK textbox for input a search string. After, I convert this string from utf8 to 852 and then to uppercase852 all with Nominals functions.
Then I read all 100.000 records and where data exist I isolate and terminate all strings, convert name field (28 chars) to uppercase852 and then search for first occurrence of search string in it with 'strstr' C function.

If search string is finded I write them to console with cp852, write them to windows txt file with cp1250 and write them to GTK textbox with utf8 with following results:

Quote:

found 52 from 43761 'čić' for 0,843 seconds

Assuming many conversions on-the-fly I think that results are very good and fast enough.

Now I have added additional conversions of CP and have 12 of them in my program (everything to anything) and total size of my program (exe) is a little less than 20 kb!

Of course, all of this would not be possible without extremely assistive Nominal Animal selfless help for which I am grateful for a lifetime!

Nominal Animal · 03-06-2011, 09:06 AM

Nime, that sounds excellent! Less than a second to search is not long at all. I'm sure the users are happy.

I know you said you don't need any more code .. but these functions allow you to work directly on the unterminated strings in your data structures.

First, these two functions use the same uppercase_ibm852 array as previously. They allow you to check if the data contains the given substring. The first is case sensitive, the second is case insensitive. If you supply a NULL pointer, or a zero-length area, both functions will just return -1 (no match) without any problems. These should make your code even simpler, I think.

Code:

/* Case sensitive substring search.
 * The haystack is the size bytes of (unterminated) data at data.
 * Both data and string use the IBM852 character set.
 * If string is in data, the function will return 0..(size-1), the offset to the start of the match.
 * If there is no match, the function will return -1.
*/
int find_ibm852(void const *const data, int const size,
                char const *const string)
{
    unsigned char const *const haystack = (unsigned char const *)data;
    unsigned char const *const needle = (unsigned char const *)string;

    int         i, o;

    /* No data? */
    if (size < 1 || !data)
        return -1;

    /* No string to search for? */
    if (!string || !*string)
        return -1;

    /* Case sensitive search loop. */
    for (i = 0; i < size; i++)
        if (haystack[i] == needle[0]) {
            o = 1;
            while (i+o < size && needle[o] && haystack[i+o] == needle[o])
                o++;
            if (!needle[o])
                return i;
        }

    /* No match. */
    return -1;
}

/* Case insensitive substring match.
 * The haystack is the size bytes of (unterminated) data at data.
 * Both data and string use the IBM852 character set,
 * and may be in any case. Both are compared case insensitively.
 * If string is in data, the function will return 0..(size-1), the offset to the start of the match.
 * If there is no match, the function will return -1.
*/
int findcase_ibm852(void const *const data, int const size,
                    char const *const string)
{
    unsigned char const *const haystack = (unsigned char const *)data;
    unsigned char const *const needle = (unsigned char const *)string;

    int         i, o;

    /* No data? */
    if (size < 1 || !data)
        return -1;

    /* No string to search for? */
    if (!string || !*string)
        return -1;

    /* Case insensitive search loop. */
    for (i = 0; i < size; i++)
        if (uppercase_ibm852[haystack[i]] == uppercase_ibm852[needle[0]]) {
            o = 1;
            while (i+o < size && needle[o] && uppercase_ibm852[haystack[i+o]] == uppercase_ibm852[needle[o]])
                o++;
            if (!needle[o])
                return i;
        }

    /* No match. */
    return -1;
}

Finally, convertdata() is a variant of the convert() function. It is used the same way, except that convertdata() takes a pointer and size (number of bytes) to the data, so the data does not need to be terminated. You could use this to create strings out of the data, and do the character set conversion at the same time.

Code:

char *convertdata(char const *const from, char const *const to,
                  void const *const data, size_t const size)
{
    iconv_t     handle;
    size_t      insize, inleft;
    char       *inends;
    size_t      outsize, outleft;
    char       *out, *outends;

    /* Invalid character set names? */
    if (!from || !to || !*from || !*to) {
        errno = EINVAL;
        return NULL;
    }

    /* No string to convert? */
    if (!data || size < (size_t)1)
        return NULL;

    /* Note: If you want this function to return an empty string
     *       instead of NULL, replace the above return NULL; with
     *       return strdup("");
    */

    /* Get a handle for the conversion. */
    handle = iconv_open(to, from);
    if (handle == (iconv_t)-1) {
        /* This conversion is not supported. */
        errno = ENOTSUP;
        return NULL;
    }

    /* Do the conversion. Grow the buffer if not large enough. */
    insize  = size;
    out     = NULL;
    outsize = insize + (size_t)16;
    while(1) {

        /* Prepare for the conversion. */
        inends = (char *)data;
        inleft = insize;

        /* Allocate a new output buffer. */
        out = malloc(outsize + 1);
        if (!out) {
            iconv_close(handle);
            errno = ENOMEM;
            return NULL;
        }
        outends = out;
        outleft = outsize;

        /* Do the conversion. */
        if (iconv(handle, &inends, &inleft, &outends, &outleft) == (size_t)0)
            break;

        /* Error? */
        if (errno != E2BIG) {
            int const   error = errno;
            free(out);
            iconv_close(handle);
            errno = error;
            return NULL;
        }

        /* Grow the output buffer size. */
        free(out);
        outsize = outsize + inleft + (size_t)16;
    }

    /* Reallocate the string to optimal size. */
    if (outleft && outsize >= outleft) {
        char *tmp;
        outsize -= outleft;
        tmp = realloc(out, outsize + (size_t)1);
        if (tmp)
            out = tmp;
    }

    /* Append EOS. */
    out[outsize] = 0;

    /* Close the conversion handle. */
    iconv_close(handle);

    return out;
}

I'm glad to have been of help. If you need any further help, or explanations on exactly why or how any of this stuff works, please let me know!

nime · 03-06-2011, 05:23 PM

Thank you Nominal for more functions but I can't get them to work and probably I don't try enough. Why to, when we already does an excellent job with good results.

But, I am sure, more than half of the world programmers will be happy to find this material here! I searched for this long time. So mighty conversions (relatively simple to make if someone like you help enough) which are independent from M$"cultures" and very slow classes in huge frameworks.

I tested my program more tonight and I see that freeing variables are very necessary. If I don't do this program becomes unstable and significantly slower. So I do best I can. Program seems reliable now but need more testing.

I also try to make new project in Linux to read my data files but I don't do something good. Compiler returns error (-ld 2 or so). But I will do this soon as possible and now I am sure with same good results.
I belive that nothing regarding cp Conversions we did not leave unfinished.
So, thank you for all once more,

nime.