Is there a utility to convert UTF to HTML metacharacters?

RandomTroll · 11-15-2017, 02:01 PM

I convert pages I download from UTF to HTML (I prefer ISO-8859). I use utf8trans, but it relies on a character table, which is incomplete. I've been adding custom entries for years, but that's lame. Converting UTF's 23f0 to ⏰ (for example) would be good enough for me; I could write a script that added all those entries to my character table, which would be inelegant and make it huge. Hmmmm... there's probably a C function to do that, which would make an easy program.

Turbocapitalist · 11-15-2017, 02:24 PM

There's a CPAN module for perl that does that already. You just need to specify the range of characters to be encoded. Here's a one-liner to encode x0100 through x2fa1f:

Code:

perl -MHTML::HTML5::Entities -p \
 -e 'print encode_entities($_,"\x{0100}..\x{2FA1F}")'; \
 < input.html > output.html

The package for that module with APT-based distros is libhtml-html5-entities-perl and something similar for RPM

RandomTroll · 11-15-2017, 11:34 PM

Thanks.

I have 2 problems: output.html has everything in input.html doubled, like the problem -n cures for sed.

It converts periods to &#x2e, which is correct, but unwanted, and an easy fix with sed.

It isn't in Slackware, but I'm used to building my own Perl modules.

Turbocapitalist · 11-16-2017, 04:29 AM

Sorry. The mistake was on my end. map() is needed to build the replacement string. The -C is needed to force I/O as UTF-8. Unicode is still a bit unfamiliar.

Also the -p should be a -n instead:

Code:

perl -CSD -MHTML::HTML5::Entities -n  \
      -e '$unicode=join("",map ({chr} 0x100 .. 0x2FA1F)); \
          print encode_entities($_,$unicode)' \
< input.html > output.html

RandomTroll · 11-16-2017, 11:45 AM

Thanks. I have 1 last question: how do I set an environment variable? The man page for this function says,

Quote:

$HTML::HTML5::Entities::hex

This variable controls whether numeric entities will use hexadecimal or decimal notation. It is TRUE (hexadecimal) by default, but can be set to FALSE.

, which I'd like to switch to decimal, but all of the tries I have made to set this variable to false are rejected.

Turbocapitalist · 11-16-2017, 11:55 AM

It's just a regular variable for perl, though one in the namespace of a specific module. Just include it in the one-liner at the beginning.

Code:

perl -CSD -MHTML::HTML5::Entities -n \
  -e '$HTML::HTML5::Entities::hex=0; \
      $unicode=join("",map ({chr} 0x100 .. 0x2FA1F)); \
      print encode_entities($_,$unicode)' \
  < input.html > output.html

RandomTroll · 11-16-2017, 03:15 PM

While the perl script worked okay for a short test file, it took minutes on a real-life 80K file (that had only 6 UTF characters to convert). I wrote a program to emit all the entries for a complete character table for utf8trans and it only takes a second longer than before. I had feared that a 65K-entry table would slow utf8trans down a lot but I don't notice.

Turbocapitalist · 11-16-2017, 03:27 PM

I also only tested on a small data set and the -n and -p options for perl make a loop and the join() and map() functions were inside that loop.

Out of curiosity, does moving them out of the loop speed things up noticeably for the larger data set?

Code:

#!/usr/bin/perl -CSD

use HTML::HTML5::Entities;

use strict;
use warnings;

$HTML::HTML5::Entities::hex=0;

my $unicode=join("",map({chr} 0x100 .. 0x2FA1F));
my $file = shift || '/dev/stdin';

open(my $in, "<", $file)
    or die("Cannot open '$file' : $!\n");

while (my $line = <$in>) {
    print encode_entities($line, $unicode);
}

close($in);

exit(0);

RandomTroll · 11-17-2017, 09:56 PM

Quote:

Too late for "-CSD" option at bin/TestIt line 1.

26 seconds, which is a lot faster than the 255 the previous version took, but still a lot more than 1. Thanks for trying. https://www.nytimes.com/2017/11/16/b...me-warner.html was the test file

Turbocapitalist · 11-18-2017, 02:06 PM

No problem. Scripts are for a quick solution from the writing perspective not necessarily from the run speed. Can you go into more detail about how you solved it? For some people, C is comfortable and they can write something equivalent quickly.

RandomTroll · 11-18-2017, 02:44 PM

utf8trans translates utf-8 characters according to a table. Each entry in the table has 2 fields, the number of the utf character to replace, written as a hex string (e.g., 2d0a), a tab, the string to translate it into. The Slackware package comes with a translation table that doesn't have all the characters I want to translate. I had added individual characters as I encountered them. I got tired of this. So I wrote a C program that emitted a complete set of entries and appended this to my table. I didn't do this in the first place because I feared that so-large a table would slow utf8trans down. That turned out not to be the case.