Vim & UTF-8

mattca · 08-26-2010, 01:43 PM

Please help

I'm working on a French website and having a horrible time getting Vim to edit the files without mangling the characters.

I can enter accented characters fine.. they appear properly in Vim, and when I save the file with fileencoding=utf-8 they show up in the browser properly as well.

The problem occurs when I have to edit a file with previously entered accented characters, they appear mangled. If I save a file with these mangled characters, they still appear properly in the browser.

However, when I enter new characters they appear properly in Vim, but NOT the browser. The old characters still appear properly. When I save the file with fileencoding=utf-8, then the new characters appear properly in the browser and the old characters are now broken.

It seems like I cannot enter new accented characters in a file without breaking the accented characters already present.

What am I doing wrong?

Hidden Windshield · 08-26-2010, 03:42 PM

Sounds to me like the files with previously entered accented characters aren't in UTF-8. So, when VIM tries to load them, they come through mangled.

What does the "file" command say when you run it on one of the files with previously entered accented characters? If it says anything other than "UTF-8 Unicode text", that's the problem. Use iconv to convert the files from whatever encoding they're in to UTF-8.

mattca · 08-27-2010, 09:23 AM

Code:

	[Fri Aug 27, 10:06:35] [~/mnt/m2w/dev.<snip>.com/system/application/controllers]
	[matt@turing] $ file -i awards.php
awards.php: text/x-php; charset=utf-8

I got it to work using this script, but I'm not sure what specifically made it work.

I also still cannot enter utf characters into the database, using the mysql command line or Vim's database plugin. I'm using rxvt-unicode and slackware 13.0 if it makes a difference.

Hidden Windshield · 08-27-2010, 02:21 PM

Hmm. I took a look at that script, and nothing jumped out at me except the terminal encoding. Maybe the terminal settings were wrong somehow? Anyway, glad you got it fixed.

As for MySQL, it uses Latin1 (ISO-8859-1) for most things, unless you specifically set it to something else. Running the command "SHOW VARIABLES LIKE 'character_set_client';" in the interactive client will show you what character set it's using to communicate with the client, for instance. You can find other such variables to check here. For individual tables, you have to use the "CHARACTER SET" attribute of char or varchar columns to get utf8, unless the table default is already utf8.

Hope this helps.