LinuxQuestions.org - about I/O text files, UTF8 and C

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - about I/O text files, UTF8 and C (https://www.linuxquestions.org/questions/programming-9/about-i-o-text-files-utf8-and-c-769313/)

badry

11-15-2009 01:03 PM

about I/O text files, UTF8 and C

How can I read and write text files (encoding: UTF8, Programming Lang: C, O.S.: Linux)? Is there any prepared source code? How about working with ‘\n’ or ‘\t’ … when the encoding is UTF8?

MS3FGX

11-15-2009 07:52 PM

Not sure what you are asking here. Are you trying to write a program that can read and write to a text file?

paulsm4

11-15-2009 09:52 PM

That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII:

Code:

#include <stdio.h>



int

main (int argc, char *argv[])

{

  char buf[80];



  printf ("Writing file...\n");

  FILE *fp = fopen ("somefile.txt", "w");

  if (!fp)

  {

    perror ("Output File open error!\n");

    return 1;

  }

  fprintf (fp, "Hello 8-bit text!\n");

  fclose (fp);



  printf ("Reading file...\n");

  fp = fopen ("somefile.txt", "r");

  if (!fp)

  {

    perror ("Input File open error!\n");

    return 1;

  }

  fgets (buf, sizeof (buf), fp);

  printf ("buf: %s...\n"), buf);

  fclose (fp);



  return 0;

}

Things get trickier if you want to support (non-UTF8) Unicode (for example, the 16-bit Unicode "WCHAR" used natively by Win32 and Java):

http://www.cprogramming.com/tutorial/unicode.html

http://www.ibm.com/developerworks/li.../l-linuni.html

http://www.linux.org/docs/ldp/howto/...e-HOWTO-6.html

'Hope that helps .. PSM

graemef

11-16-2009 03:54 AM

Quote:

Originally Posted by paulsm4 (Post 3758379)

That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII

That's not correct. UTF-8 supports 7-bit ASCII but it also supports much more than that. Beyond the 7bit ASCII UTF-8 will be encoded using a standard algorithm.

To read any UTF-8 file (except in the trivial case when the file only contains 7-bit ASCII) then a UTF-8 library should be used.

Since I normally use C++ rather than C I use the Qt library, however I'm sure that there is a good library within the Gnome project that you could use. For the serious checkout IBM's ICU library.

H_TeXMeX_H

11-16-2009 04:22 AM

See here:
http://www.cl.cam.ac.uk/~mgk25/unicode.html#c

paulsm4

11-16-2009 09:27 AM

Hi again, Badry -

Quote:

UTF-8 supports 7-bit ASCII but it also supports much more than that.
<= This is absolutely true

I didn't want to overcomplicate things in my reply - but I hope I didn't oversimplify them, either (at least not too much ;)).

I want to emphasize - for the majority of string handling you're likely to do in C (and certainly for things like 8-bit '\n' and '\t' characters), you can essentially say "the standard C library gives you UTF-8 for free". Because the majority of string handling in most "classic" C programs *IS* 7-bit ASCII.

Please follow up on some of the links provided. They're all good. I'd especially recommend this one:

http://www.cprogramming.com/tutorial/unicode.html

Your .. PSM

badry

11-16-2009 12:41 PM

Quote:

Originally Posted by MS3FGX (Post 3758319)

Not sure what you are asking here. Are you trying to write a program that can read and write to a text file?

Yes! The files are encoded by UTF8. I know how to read/write ordinary files in C (using char, FILE *, ...), but it does not work for UTF8!

badry

11-16-2009 12:59 PM

Thanks! I have written a similar code. However, it does not works. I want to read a Persian text. And after some simple modifications (such as finding tabs, points, ...), write to another file.

Quote:

Originally Posted by paulsm4 (Post 3758379)

That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII:

Code:

#include <stdio.h>



int

main (int argc, char *argv[])

{

  char buf[80];



  printf ("Writing file...\n");

  FILE *fp = fopen ("somefile.txt", "w");

  if (!fp)

  {

    perror ("Output File open error!\n");

    return 1;

  }

  fprintf (fp, "Hello 8-bit text!\n");

  fclose (fp);



  printf ("Reading file...\n");

  fp = fopen ("somefile.txt", "r");

  if (!fp)

  {

    perror ("Input File open error!\n");

    return 1;

  }

  fgets (buf, sizeof (buf), fp);

  printf ("buf: %s...\n"), buf);

  fclose (fp);



  return 0;

}

paulsm4

11-16-2009 02:25 PM

Whoops, sorry badry!

If you want Persian text, then you're definitely *not* doing 7-bit ASCII.

However, you *can* probably do everything you need with UTF-8. And the links I and others gave you should still be useful.

Also: please look at this exchange:

http://www.mail-archive.com/linux-ut.../msg05595.html

'Hope that helps .. PSM

PS:
You're already going UTF-8. That's definitely the right thing to do, and apologies again for confusing things. Here's a quote from the above link emphasizing this point:

Quote:

The best advice you can get is to steer clear of wide characters.
You should never need to use any wide character functions.
Keep the data in your program internally represented as utf-8.
The standard byte-oriented "strlen", "strcpy", "strstr", "printf" etc
work fine with utf-8.

XML uses utf-8 by default as well, so little if any conversion between
encodings should be needed. You may have to convert your input from a
legacy encoding to utf-8, or you could just externally convert using
something such as this:

cat inputfile | iconv -t utf-8 | myprogram

Being "unicode aware" is trivial in this fashion.

paulsm4

11-28-2009 05:06 PM

Hi, Badry -

One more addendum - a *very* good article I'd like to see *every* developer read:

http://www.joelonsoftware.com/articles/Unicode.html

IMHO .. PSM

PS:
I know it helped clarify more than a couple of things for me! :)

All times are GMT -5. The time now is 07:47 PM.