file conversion issue

weebo1 · 06-24-2013, 03:25 PM

Hello

After a few hours of unsuccessful google-ing, I decided to ask to pros.
This is my problem:
The result of an annotation tool is saved as an .xml file
I need to extract some lines, which contain a certain pattern, from the many files in a folder.
Here is the trick: for some files it works, for some not. I tried to break-down the problem.
My "detective" work (aka GOOGLE like crazy) has led me to the following conclusions:
1. The output of

Code:

grep "start" test.anvil

is:

Code:

      <el index="0" start="0" end="0.93332">
      <el index="1" start="0.93332" end="1.93331">
      <el index="2" start="1.93331" end="3.1333">

So, my grep command works.

2. The output of

Code:

file -bi test.anvil

is:

Code:

text/xml

3. The output of

Code:

grep "start" test_not_working.anvil

is:
nothing.

4. The output of

Code:

file -bi test_not_working.anvil

is:

Code:

text/plain; charset=utf-16

I tried iconv in any possible way. No success (I get the error:

Code:

iconv: illegal input sequence at position 0.

I tried messing with the xml file itself. Nothing.
My only mistake was that when I saved the file in the annotation tool, I didn't choose from beginning ISO-8859-1 coding, and I left the default value: UTF-8
I really don't know what else I can try. What I found on google was related to iconv and nothing worked for me. not even the \\IGNORE option.
Any help is more than appreciated.

Thanks a lot

szboardstretcher · 06-24-2013, 03:41 PM

have you tried opening a copy of the file in vim?

then try this and write out the file?

Code:

:set fileencoding=UTF-8

weebo1 · 06-27-2013, 02:58 AM

Thanks for your reply. I tried and I still get errors.
Any other suggestions?

Thanks

Madhu Desai · 06-27-2013, 04:19 AM

Code:

iconv -f UTF-16 -t UTF-8 test_not_working.anvil | grep 'start'

weebo1 · 06-28-2013, 12:45 AM

Thanks, but iconv was one of the first I tried and no success

szboardstretcher · 06-28-2013, 12:52 AM

Is there any way you can upload a small file that is giving you problems? After scrubbing it of proprietary/confidential information of course.

Id like to see whats what.

David the H. · 06-28-2013, 12:56 PM

Try checking for the existence of a Byte Order Mark, and perhaps dos-style line-endings.

In any case, xml is not a very easily greppable data format. I suggest running each file through the hxpipe utility, from the html-xml-utils package. It will convert the xml into a line-based format that's safer and easier for grep/sed/awk to parse. You will have to clean up the lines a bit though to remove the formatting flags.

The package also has a few other tools like hxclean and hxnormalize which can help tidy up broken xml.

And finally, I agree with szboardstretcher. It would help to see a larger sample of the data, and preferably the original.

Edit: Here's an example of how hxpipe can be used. With the lines given in the OP, you can extract the "start" data like this:

Code:

$ text='<el index="0" start="0" end="0.93332">
<el index="1" start="0.93332" end="1.93331">
<el index="2" start="1.93331" end="3.1333">'

$ hxpipe <<<"$text"
-
Aindex CDATA 0
Astart CDATA 0
Aend CDATA 0.93332
(el
-\n
Aindex CDATA 1
Astart CDATA 0.93332
Aend CDATA 1.93331
(el
-\n
Aindex CDATA 2
Astart CDATA 1.93331
Aend CDATA 3.1333
(el
-\n

$ hxpipe <<<"$text" | sed -n '/^Astart/ s/.*CDATA //p'
0
0.93332
1.93331