convert file from UTF8 to ASCII encoding

graemef · 06-27-2006, 12:58 PM

I have been writing a VBA script (not my area of expertise) to extract some data from an excel spreadsheet and generate an XML file. The XML file is going to be read in by an existing program. The file will include some Unicode characters and so VBA needs it to be saved as an Unicode (UTF-8) file but the program that will read the file needs it to be saved in ASCII format.

I have opened the file with Notepad++ switched the encoding to ASCII and saved the file and this works.

However I am going to generate about 5000 files, I don't really want to do this for each one by hand! So does anyone know of a way in which I can automate this process.

Ideas, solutions welcome.

Thanks very much.

unSpawn · 06-27-2006, 01:09 PM

iconv? From "apropos iconv": iconv (1) - Convert encoding of given files from one encoding to another

jim mcnamara · 06-27-2006, 03:03 PM

There is a file, usually /usr/lib/nls/iconv/config.iconv, that lists all of the possible (allowable conversions)

On my box

Code:

iconv -f utf8 -t iso89 oldfile > newfile

does the job. You need to find config.iconv, then read it to find what your commands are.

graemef · 06-27-2006, 10:02 PM

Excellent response, thanks unSpawn & jim mcnamara. The iconv is supported in php so I can put together a quick script to locate each file and convert it. I'm a little confused by the encodings that I needed to use but I've tested it and it works, so I'm relieved.

Once again thanks very much.

jlliagre · 06-28-2006, 01:38 AM

I'm surprised it worked, as iso89 isn't a known encoding.

Didn't you run instead "iconv -f utf8 -t ascii oldfile > newfile" ?

What Unicode characters were causing a problem in your source file ?

graemef · 06-28-2006, 02:00 AM

The php function I've written is as follows:

PHP Code:


			
function encodeFiles($rootPath)

{

    $directories = scandir($rootPath);

    // Loop through each directory in the rootPath

    foreach ($directories as $dir)

    {

        $fileName = $rootPath."\\".$dir. "\\metadata.xml";

        // Check to see if the current directory has a file called metadata.xml

        if (file_exists($fileName))

        {

            // Read in the contents

            $data = file_get_contents($fileName);

            // Just display on the screen the file being modified

            echo "Converting " . $fileName . "...\n";

            // Convert the contents

            $data = iconv("UCS-2","UTF-8", $data);

            // Write back out to the same file

            file_put_contents($fileName,$data);

        }  // end if file exists

    }  // end loop for each directory in the root directory

}  // end of function encodeFiles()

When I open the file in Notepad++ it tells me the encoding is ANSI, which is what I needed and the program I feed the file into accepts it. So I'm now making progress in the conversion job.

jlliagre · 06-28-2006, 02:56 AM

Okay, so instead of converting from UTF-8 to ASCII, you are converting from UTF-8 to UTF-16.

Your posting title was misleading, UTF-8 is close to ASCII, while UTF-16 is definitely not.

graemef · 06-28-2006, 03:07 AM

My title was describing what I thought I needed to do at the time. The program I'm creating the xml files for is fairly poor on documentation and I had spent the whole afternoon trying to figure out why they were being rejected, no errors, no log files, not much in the way of assistance.

But I'm now over that hurdle, I wonder what it will throw at me next.

kalimat · 12-15-2008, 04:45 AM

hi....i have a question:
and if i want to do a script for converting every file in a folder,what can i do?

i started like this:

#!/bin/bash
cd folder
n=$(ls | wc -l)
i=1
p=... //path of the file

//and now..for every file in the folder (?)

while test i -le n; do
iconv -f UTF-16 -t ASCII $p
((i++))
p=.....//path of the next file
done

How i make i to retain the path of the file?..
i am waiting for an answer..
thanks a lot