[SOLVED] replace ascii characters using sed

sopier · 12-09-2011, 11:12 AM

I'm using gnome terminal, when I run this command:

Code:

echo Michael Bublé | sed 's/[éèêë]/e/g'

the results is fine => Michael Buble

but when I run this code:

Code:

cat artist.txt | sed 's/[éèêë]/e/g'

the result is Michael Bubl

Any solutions on this?

Regards

druuna · 12-09-2011, 11:16 AM

Hi,

Have you tried the more elegant and resource friendly:

Code:

sed 's/[éèêë]/e/g' artist.txt

Hope this helps.

corp769 · 12-09-2011, 11:17 AM

Works for me. Can you post the contents of that file?

sopier · 12-09-2011, 11:19 AM

Yes, I already tried that too, the result is still the same: Michael Bubl

sopier · 12-09-2011, 11:21 AM

the content is only one line:

Code:

Michael Bublé

druuna · 12-09-2011, 11:23 AM

Hi,

What's the output of the following command:

Code:

$ file artist.txt

The output of the above command should show if this is a "normal" ascii file or not.

sopier · 12-09-2011, 11:25 AM

Quote:

Originally Posted by druuna

Hi,

What's the output of the following command:

Code:

$ file artist.txt

The output of the above command should show if this is a "normal" ascii file or not.

artist.txt: ISO-8859 text

druuna · 12-09-2011, 11:37 AM

Hi,

That might be the problem.

Mine says UTF-8 Unicode text and both sed 's/[éèêë]/e/g' artist.txt and cat artist.txt | sed 's/[éèêë]/e/g' work on my side.

If I change my locale to anything else then UTF8 the output is incorrect.

Check your locale setting (locale -a), try setting it to, for example, en_US.UTF-8 (export LANG=en_US.utf8) and try again.

Hope this helps.

BTW: The above changes made are only set in your current terminal.

sopier · 12-09-2011, 11:48 AM

Quote:

Originally Posted by druuna

Hi,

That might be the problem.

Mine says UTF-8 Unicode text and both sed 's/[éèêë]/e/g' artist.txt and cat artist.txt | sed 's/[éèêë]/e/g' work on my side.

If I change my locale to anything else then UTF8 the output is incorrect.

Check your locale setting (locale -a), try setting it to, for example, en_US.UTF-8 (export LANG=en_US.utf8) and try again.

Hope this helps.

BTW: The above changes made are only set in your current terminal.

Thank you, I finally convert the file first using this command

Code:

iconv -f ISO-8859-1 -t UTF-8 artist_old.txt > artist_new.txt

and the sed replacing goes fine...

Regards

druuna · 12-09-2011, 11:50 AM

You're welcome

Nominal Animal · 12-09-2011, 12:35 PM

Note that you can use

Code:

iconv -f ISO-8859-1 -t ASCII//TRANSLIT input-file > output-file

to convert all accented letters to their nearest ASCII equivalents, for example é è ê ë to e, æ to ae, and so on.