Welcome to the most active Linux Forum on the web.
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


Closed Thread
  Search this Thread
Old 01-26-2009, 06:29 PM   #1
Senior Member
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Rep: Reputation: 51
python: how to handle unicode chars in ascii strings?

So, in python, this assignment is legal, but it breaks conversion:

In [44]: sys.getdefaultencoding()
Out[44]: 'ascii'

In [45]: a = 'André'

In [46]: a.encode('ascii','replace')
exceptions.UnicodeDecodeError                        Traceback (most recent call last)

/hosts/soho/v11/users/briank/<ipython console>

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)
however, if I specify that the string is unicode, this all works:

In [47]: a = u'André'

In [48]: a.encode('ascii','replace')
Out[48]: 'Andr??'
The problem I'm having here is that I'm using os.walk to go through a bunch of files... some of those files have paths with unicode chars. I'm unclear on how to get the results of os.walk to be considered unicode such that the encode function works correctly.

At the end of the day, All these paths are going into a database. I just want everything that comes out of that database to be ascii & I'm having a hard time with that when unicode chars make their way into strings that think they are ascii.

Hope that made sense.

So... how can you convert ascii strings with unicode chars into actual ascii strings. Pardon me if my terminology is incorrect - see my first example for what I mean by "unicode chars in ascii strings"
Old 01-26-2009, 10:46 PM   #2
Senior Member
Registered: Feb 2001
Location: Atlanta, GA
Distribution: Slackware
Posts: 1,823

Rep: Reputation: 120Reputation: 120
Continue discussion here:

Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
python: how do you replace unicode chars in large text files? BrianK Programming 1 12-19-2008 01:54 AM
Is it really worth it to use chars, especialy since I can strings? RHLinuxGUY Programming 6 07-23-2006 01:52 PM
cyrillic chars - unicode or koi8_ru ojav Linux - Newbie 1 05-29-2005 03:51 PM
can fedora handle ascii strings for wep keys? ehawk Linux - Wireless Networking 1 10-13-2004 07:34 PM
Fun with strings & chars in C Scrag Programming 4 05-19-2004 04:03 PM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:36 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration