[SOLVED] Apply GREP to replace invalid UTF-8 characters in files

SparceMatrix · 11-02-2011, 05:02 PM

jthill, I've applied your snippet of code in a script combined with Nominal Animal's test for valid UTF-8:

Code:

#/bin/bash
echo "THIS IS A TEST";
LIST=`ls /my/test/files`;
echo "THIS IS A TEST TOO";
for i in $LIST 
do
	if !(iconv -f UTF-8 -t UTF-8 $i &>/dev/null) ; 
	then
		echo "$i is not valid UTF-8"
		ex -b -u NONE -c 'set backup' -c '/^<body>$/,/^<\/body>$/!iconv -f windows-1252 -t utf-8' -c 'wq' "$i" 
	fi
done

I am getting back some errors:

Code:

[xxx@abc123 bashScripts]# bash TESTbash2
THIS IS A TEST
THIS IS A TEST TOO
100525ED.htm.bak is not valid UTF-8
"100525ED.htm.bak" [New File]
Error detected while processing command line:
E486: Pattern not found: ^<body>$
"100525ED.htm.bak" [New File] 0 lines, 0 characters written
100601ED.htm is not valid UTF-8
"100601ED.htm" [New File]
Error detected while processing command line:
E486: Pattern not found: ^<body>$
"100601ED.htm" [New File] 0 lines, 0 characters written
[xxx@abc123 bashScripts]#

My backups are in the script directory, I think, instead of the source directory. But they are empty files.

I have one reference for bash, 3rd Edition O'Reilly Linux in a Nutshell. I see no reference to the 'set backup' in the options for 'set'. I also don't see -b or -u as options for the 'ex' command. Is my reference too old?

Despite the errors, the script works and I hope to advance it to search and change sub-directories. What is the significance of the errors?

jthill · 11-02-2011, 09:04 PM

Quote:

But they are empty files.

What's in the files they're backups of?

Quote:

I have one reference for bash, 3rd Edition O'Reilly Linux in a Nutshell. I see no reference to the 'set backup' in the options for 'set'. I also don't see -b or -u as options for the 'ex' command. Is my reference too old?

You're running vim 7.0 from 2006, I have 7.3 from 2011, I looked up that O'Reilly 3rd and the book's talking about version 5.8 or 5.7 which were current as of the copyright date on that, 2000. Vim's builtin docs are so complete it might be daunting, you could buy a current book. 'backup' is a vim option, it's what leads to the tilde-suffix backups, nothing to do with bash.

To make that script work on multiple directories I'd use find not ls. find prints full pathnames and you can make it find only what you want. Its doc is delivered as an info node, 'info find' (no quotes) at a prompt to see if you've got it installed, if it's not the manpage and the find commands in my original reply will get you started.

SparceMatrix · 11-03-2011, 01:50 PM

Quote:

What's in the files they're backups of?

They are long HTML files. There is something in the originals, the files that show up mysteriously in the script's directory are empty. I'd really like to have some backups.

Yes, 'find' is the preferred command.

There is a handy on line reference for Linux commands at O'Reilly,

http://oreilly.com/linux/command-directory/

There is also a handy VIM on line reference,

http://vimdoc.sourceforge.net/htmldoc/help.html

I looked up 'backup', but could see no reference to an empty file in backup.

http://vimdoc.sourceforge.net/htmldo...ml#'backup'

There is also a handy reference for the command line for Linux,

http://linuxcommand.org/index.php

SparceMatrix · 11-03-2011, 05:23 PM

kindlegen is now digesting all my UTF-8 files after having processed them with this:

Code:

#/bin/bash
echo "THIS IS A TEST";
LIST=`find . -name '*.html'`;
for i in $LIST 
do
	if !(iconv -f UTF-8 -t UTF-8 $i &>/dev/null) ; 
	then
		echo "$i is not valid UTF-8"
		# ex -u NONE -b -c 'set backup' -c '1G' -c '/body>/,/body>/;/body>/!iconv -f windows-1252 -t utf-8' -c 'wq' "$f"
		ex -b -u NONE -c 'set backup' -c '/^<body>$/,/^<\/body>$/!iconv -f windows-1252 -t utf-8' -c 'wq' "$i" 
	# else 
		# echo "$i is valid UTF-8"
	fi
done

Thanks to jthill and Nominal Animal for your invaluable help. I couldn't have done it without you.

I now move on to the source of the problem which is based in some Perl scripts.

jthill · 11-03-2011, 06:45 PM

[edit: whoops, I somehow missed that last post, duh. glad it all worked out!]