about I/O text files, UTF8 and C

badry · 11-15-2009, 01:03 PM

How can I read and write text files (encoding: UTF8, Programming Lang: C, O.S.: Linux)? Is there any prepared source code? How about working with ‘\n’ or ‘\t’ … when the encoding is UTF8?

MS3FGX · 11-15-2009, 07:52 PM

Not sure what you are asking here. Are you trying to write a program that can read and write to a text file?

paulsm4 · 11-15-2009, 09:52 PM

That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII:

Code:

#include <stdio.h>

int
main (int argc, char *argv[])
{
  char buf[80];

  printf ("Writing file...\n");
  FILE *fp = fopen ("somefile.txt", "w");
  if (!fp)
  {
    perror ("Output File open error!\n");
    return 1;
  }
  fprintf (fp, "Hello 8-bit text!\n");
  fclose (fp);

  printf ("Reading file...\n");
  fp = fopen ("somefile.txt", "r");
  if (!fp)
  {
    perror ("Input File open error!\n");
    return 1;
  }
  fgets (buf, sizeof (buf), fp);
  printf ("buf: %s...\n"), buf);
  fclose (fp);

  return 0;
}

Things get trickier if you want to support (non-UTF8) Unicode (for example, the 16-bit Unicode "WCHAR" used natively by Win32 and Java):

http://www.cprogramming.com/tutorial/unicode.html

http://www.ibm.com/developerworks/li.../l-linuni.html

http://www.linux.org/docs/ldp/howto/...e-HOWTO-6.html

'Hope that helps .. PSM

graemef · 11-16-2009, 03:54 AM

Quote:

Originally Posted by paulsm4

That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII

That's not correct. UTF-8 supports 7-bit ASCII but it also supports much more than that. Beyond the 7bit ASCII UTF-8 will be encoded using a standard algorithm.

To read any UTF-8 file (except in the trivial case when the file only contains 7-bit ASCII) then a UTF-8 library should be used.

Since I normally use C++ rather than C I use the Qt library, however I'm sure that there is a good library within the Gnome project that you could use. For the serious checkout IBM's ICU library.

H_TeXMeX_H · 11-16-2009, 04:22 AM

See here:
http://www.cl.cam.ac.uk/~mgk25/unicode.html#c

paulsm4 · 11-16-2009, 09:27 AM

Hi again, Badry -

Quote:

UTF-8 supports 7-bit ASCII but it also supports much more than that.
<= This is absolutely true

I didn't want to overcomplicate things in my reply - but I hope I didn't oversimplify them, either (at least not too much

).

I want to emphasize - for the majority of string handling you're likely to do in C (and certainly for things like 8-bit '\n' and '\t' characters), you can essentially say "the standard C library gives you UTF-8 for free". Because the majority of string handling in most "classic" C programs *IS* 7-bit ASCII.

Please follow up on some of the links provided. They're all good. I'd especially recommend this one:

http://www.cprogramming.com/tutorial/unicode.html

Your .. PSM

badry · 11-16-2009, 12:41 PM

Quote:

Originally Posted by MS3FGX

Not sure what you are asking here. Are you trying to write a program that can read and write to a text file?

Yes! The files are encoded by UTF8. I know how to read/write ordinary files in C (using char, FILE *, ...), but it does not work for UTF8!

badry · 11-16-2009, 12:59 PM

Thanks! I have written a similar code. However, it does not works. I want to read a Persian text. And after some simple modifications (such as finding tabs, points, ...), write to another file.

Quote:

Originally Posted by paulsm4

That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII:

Code:

#include <stdio.h>

int
main (int argc, char *argv[])
{
  char buf[80];

  printf ("Writing file...\n");
  FILE *fp = fopen ("somefile.txt", "w");
  if (!fp)
  {
    perror ("Output File open error!\n");
    return 1;
  }
  fprintf (fp, "Hello 8-bit text!\n");
  fclose (fp);

  printf ("Reading file...\n");
  fp = fopen ("somefile.txt", "r");
  if (!fp)
  {
    perror ("Input File open error!\n");
    return 1;
  }
  fgets (buf, sizeof (buf), fp);
  printf ("buf: %s...\n"), buf);
  fclose (fp);

  return 0;
}

Things get trickier if you want to support (non-UTF8) Unicode (for example, the 16-bit Unicode "WCHAR" used natively by Win32 and Java):

http://www.cprogramming.com/tutorial/unicode.html

http://www.ibm.com/developerworks/li.../l-linuni.html

http://www.linux.org/docs/ldp/howto/...e-HOWTO-6.html

'Hope that helps .. PSM

paulsm4 · 11-16-2009, 02:25 PM

Whoops, sorry badry!

If you want Persian text, then you're definitely *not* doing 7-bit ASCII.

However, you *can* probably do everything you need with UTF-8. And the links I and others gave you should still be useful.

Also: please look at this exchange:

http://www.mail-archive.com/linux-ut.../msg05595.html

'Hope that helps .. PSM

PS:
You're already going UTF-8. That's definitely the right thing to do, and apologies again for confusing things. Here's a quote from the above link emphasizing this point:

Quote:

The best advice you can get is to steer clear of wide characters.
You should never need to use any wide character functions.
Keep the data in your program internally represented as utf-8.
The standard byte-oriented "strlen", "strcpy", "strstr", "printf" etc
work fine with utf-8.

XML uses utf-8 by default as well, so little if any conversion between
encodings should be needed. You may have to convert your input from a
legacy encoding to utf-8, or you could just externally convert using
something such as this:

cat inputfile | iconv -t utf-8 | myprogram

Being "unicode aware" is trivial in this fashion.

paulsm4 · 11-28-2009, 05:06 PM

Hi, Badry -

One more addendum - a *very* good article I'd like to see *every* developer read:

http://www.joelonsoftware.com/articles/Unicode.html

IMHO .. PSM

PS:
I know it helped clarify more than a couple of things for me!