grep on Unicode files

overbored · 07-16-2004, 12:10 AM

I installed Cygwin and I'd like to grep some Unicode files to see what lines it's used on, but all I get is the following:

Code:

$ grep -ni '\bI\b' Pronouns.txt
Binary file Pronouns.txt matches

$ grep -niU '\bI\b' Pronouns.txt
Binary file Pronouns.txt matches

$ grep -V
grep (GNU grep) 2.5

Copyright 1988, 1992-1999, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Does anybody know how to make grep work with Unicode files? Or are there any command-line tools that can convert Unicode to/from ASCII?

Thanks in advance.

/bin/bash · 07-18-2004, 05:26 AM

Try grep -a or grep --binary-files=text, they are the same thing.

wipe · 07-18-2004, 01:12 PM

The current stable version of GNU grep seems to (still) be 2.5.1. It's a bugfix update from 2.5. I don't know what the bugs were though. Both versions should be able to handle all usual flavours of Unicode. The problem might be in Cygwin or your text files (try other editors).

There is a standard tool for encoding conversions called iconv. It's distributed with the GNU C library but it's a commandline utility as well. I believe pretty much every programming and scripting language has an iconv interface of some sort.

Simon