python: how to handle unicode chars in ascii strings?

BrianK · 01-26-2009, 05:32 PM

So, in python, this assignment is legal, but it breaks conversion:

Code:

In [44]: sys.getdefaultencoding()
Out[44]: 'ascii'

In [45]: a = 'André'

In [46]: a.encode('ascii','replace')
---------------------------------------------------------------------------
exceptions.UnicodeDecodeError                        Traceback (most recent call last)

/hosts/soho/v11/users/briank/<ipython console>

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)

however, if I specify that the string is unicode, this all works:

Code:

In [47]: a = u'André'

In [48]: a.encode('ascii','replace')
Out[48]: 'Andr??'

The problem I'm having here is that I'm using os.walk to go through a bunch of files... some of those files have paths with unicode chars. I'm unclear on how to get the results of os.walk to be considered unicode such that the encode function works correctly.

At the end of the day, All these paths are going into a database. I just want everything that comes out of that database to be ascii & I'm having a hard time with that when unicode chars make their way into strings that think they are ascii.

Hope that made sense.

So... how can you convert ascii strings with unicode chars into actual ascii strings. Pardon me if my terminology is incorrect - see my first example for what I mean by "unicode chars in ascii strings"

unSpawn · 01-26-2009, 06:48 PM

Please post your thread once and in only one forum. Posting a single thread in the most relevant forum will make it easier for members to help you and will keep the discussion in one place. This thread should be closed because it is a duplicate of http://www.linuxquestions.org/questi...trings-700041/.

niknah · 01-26-2009, 07:52 PM

I've double posted before too when the site was down and I hit reload.

What you want is maybe encode('latin-1'... instead of 'ascii'.

BrianK · 01-26-2009, 08:43 PM

To add to the mystery, I tried this, which worked in one python session, worked in the script, though I can't seem to make another example of it, so I suspect it's not totally correct...

Code:

codecs.unicode_internal_decode(a)[0].encode('ascii','replace')

I'm open to better solutions if anyone has one.

BrianK · 01-26-2009, 08:45 PM

Quote:

Originally Posted by unSpawn

Please post your thread once and in only one forum. Posting a single thread in the most relevant forum will make it easier for members to help you and will keep the discussion in one place. This thread should be closed because it is a duplicate of http://www.linuxquestions.org/questi...trings-700041/.

yeah, must have been a site error (or something on my end... not my finger

). It took forever to post & I only clicked "submit" once. After a few minutes, it came back with a 503 (I think) gateway error.

Duly noted, however.

unSpawn · 01-27-2009, 04:20 PM

If it was due to a server timeout then NP, in any case the other thread was closed so no efforts were wasted.