LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-15-2009, 01:03 PM   #1
badry
LQ Newbie
 
Registered: Nov 2009
Posts: 3

Rep: Reputation: 0
Question about I/O text files, UTF8 and C


How can I read and write text files (encoding: UTF8, Programming Lang: C, O.S.: Linux)? Is there any prepared source code? How about working with \n or \t when the encoding is UTF8?
 
Old 11-15-2009, 07:52 PM   #2
MS3FGX
Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 351Reputation: 351Reputation: 351Reputation: 351
Not sure what you are asking here. Are you trying to write a program that can read and write to a text file?
 
Old 11-15-2009, 09:52 PM   #3
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII:
Code:
#include <stdio.h>

int
main (int argc, char *argv[])
{
  char buf[80];

  printf ("Writing file...\n");
  FILE *fp = fopen ("somefile.txt", "w");
  if (!fp)
  {
    perror ("Output File open error!\n");
    return 1;
  }
  fprintf (fp, "Hello 8-bit text!\n");
  fclose (fp);

  printf ("Reading file...\n");
  fp = fopen ("somefile.txt", "r");
  if (!fp)
  {
    perror ("Input File open error!\n");
    return 1;
  }
  fgets (buf, sizeof (buf), fp);
  printf ("buf: %s...\n"), buf);
  fclose (fp);

  return 0;
}
Things get trickier if you want to support (non-UTF8) Unicode (for example, the 16-bit Unicode "WCHAR" used natively by Win32 and Java):

http://www.cprogramming.com/tutorial/unicode.html

http://www.ibm.com/developerworks/li.../l-linuni.html

http://www.linux.org/docs/ldp/howto/...e-HOWTO-6.html

'Hope that helps .. PSM
 
Old 11-16-2009, 03:54 AM   #4
graemef
Senior Member
 
Registered: Nov 2005
Location: Hanoi
Distribution: Fedora 13, Ubuntu 10.04
Posts: 2,379

Rep: Reputation: 148Reputation: 148
Quote:
Originally Posted by paulsm4 View Post
That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII
That's not correct. UTF-8 supports 7-bit ASCII but it also supports much more than that. Beyond the 7bit ASCII UTF-8 will be encoded using a standard algorithm.

To read any UTF-8 file (except in the trivial case when the file only contains 7-bit ASCII) then a UTF-8 library should be used.

Since I normally use C++ rather than C I use the Qt library, however I'm sure that there is a good library within the Gnome project that you could use. For the serious checkout IBM's ICU library.
 
Old 11-16-2009, 04:22 AM   #5
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
See here:
http://www.cl.cam.ac.uk/~mgk25/unicode.html#c
 
Old 11-16-2009, 09:27 AM   #6
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Hi again, Badry -

Quote:
UTF-8 supports 7-bit ASCII but it also supports much more than that.
<= This is absolutely true
I didn't want to overcomplicate things in my reply - but I hope I didn't oversimplify them, either (at least not too much ).

I want to emphasize - for the majority of string handling you're likely to do in C (and certainly for things like 8-bit '\n' and '\t' characters), you can essentially say "the standard C library gives you UTF-8 for free". Because the majority of string handling in most "classic" C programs *IS* 7-bit ASCII.

Please follow up on some of the links provided. They're all good. I'd especially recommend this one:

http://www.cprogramming.com/tutorial/unicode.html

Your .. PSM
 
Old 11-16-2009, 12:41 PM   #7
badry
LQ Newbie
 
Registered: Nov 2009
Posts: 3

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by MS3FGX View Post
Not sure what you are asking here. Are you trying to write a program that can read and write to a text file?
Yes! The files are encoded by UTF8. I know how to read/write ordinary files in C (using char, FILE *, ...), but it does not work for UTF8!
 
Old 11-16-2009, 12:59 PM   #8
badry
LQ Newbie
 
Registered: Nov 2009
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks! I have written a similar code. However, it does not works. I want to read a Persian text. And after some simple modifications (such as finding tabs, points, ...), write to another file.


Quote:
Originally Posted by paulsm4 View Post
That's exactly what C does best; essentially UTF-8 *is* 7-bit ASCII:
Code:
#include <stdio.h>

int
main (int argc, char *argv[])
{
  char buf[80];

  printf ("Writing file...\n");
  FILE *fp = fopen ("somefile.txt", "w");
  if (!fp)
  {
    perror ("Output File open error!\n");
    return 1;
  }
  fprintf (fp, "Hello 8-bit text!\n");
  fclose (fp);

  printf ("Reading file...\n");
  fp = fopen ("somefile.txt", "r");
  if (!fp)
  {
    perror ("Input File open error!\n");
    return 1;
  }
  fgets (buf, sizeof (buf), fp);
  printf ("buf: %s...\n"), buf);
  fclose (fp);

  return 0;
}
Things get trickier if you want to support (non-UTF8) Unicode (for example, the 16-bit Unicode "WCHAR" used natively by Win32 and Java):

http://www.cprogramming.com/tutorial/unicode.html

http://www.ibm.com/developerworks/li.../l-linuni.html

http://www.linux.org/docs/ldp/howto/...e-HOWTO-6.html

'Hope that helps .. PSM
 
Old 11-16-2009, 02:25 PM   #9
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Whoops, sorry badry!

If you want Persian text, then you're definitely *not* doing 7-bit ASCII.

However, you *can* probably do everything you need with UTF-8. And the links I and others gave you should still be useful.

Also: please look at this exchange:

http://www.mail-archive.com/linux-ut.../msg05595.html

'Hope that helps .. PSM

PS:
You're already going UTF-8. That's definitely the right thing to do, and apologies again for confusing things. Here's a quote from the above link emphasizing this point:
Quote:
The best advice you can get is to steer clear of wide characters.
You should never need to use any wide character functions.
Keep the data in your program internally represented as utf-8.
The standard byte-oriented "strlen", "strcpy", "strstr", "printf" etc
work fine with utf-8.

XML uses utf-8 by default as well, so little if any conversion between
encodings should be needed. You may have to convert your input from a
legacy encoding to utf-8, or you could just externally convert using
something such as this:

cat inputfile | iconv -t utf-8 | myprogram

Being "unicode aware" is trivial in this fashion.

Last edited by paulsm4; 11-16-2009 at 02:32 PM.
 
Old 11-28-2009, 05:06 PM   #10
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Hi, Badry -

One more addendum - a *very* good article I'd like to see *every* developer read:

http://www.joelonsoftware.com/articles/Unicode.html

IMHO .. PSM

PS:
I know it helped clarify more than a couple of things for me!
 
  


Reply

Tags
file, io, programming, unicode


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Convert DOS text files to UNIX text files ta0kira Linux - Software 7 03-15-2011 11:42 AM
how can i display utf8 format in text area mlibot Linux - Newbie 6 08-12-2009 08:33 AM
utf8 cyrillic text in console emacs tramni1980 Slackware 7 08-25-2008 12:07 PM
Steps needed to convert multiple text files into one master text file jamtech Programming 5 10-07-2007 11:24 PM


All times are GMT -5. The time now is 07:37 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration