Perl help - recursively find non-ascii characters in file

zeky · 01-07-2009, 03:56 AM

Hi!

I have some nasty, non-ascii character in some files that contains php code. What I want to do here is to recursively find all the files that contains a specific non-ascii character in the file. And most importantly - i need to know the name of that file.

So far, I found a script that looks into a file for non-ascii characters:

Code:

while (<>) { 
    s/([\x80-\xff])/sprintf "\\x{%02x}",ord($1)/eg; 
    print; 
}

Ok, this is good, the non-ascii character that I'm looking for is:

Code:

x{ef}\\x{bb}\\x{bf}

The problem here is that i can can't run this script to run recursively and I don't get the name of the file that contains this characters.

I've tried with bash, but since it's standard output, I can't get any resault on this. Here is what I've tried:

Code:

find |xargs /usr/local/bin/check_for_non-ascii_characters.sh  |grep -l 'x{ef}\\x{bb}\\x{bf}'

So, I need a way to recursively find non-ascii characters (a specific pattern, mentioned before) in all files and I need the name of the files containing it.

Thanks

eco · 01-07-2009, 04:01 AM

why not simply use the '-R' option in grep?

Code:

# grep -Rl 'x{ef}\\x{bb}\\x{bf}' *

zeky · 01-07-2009, 06:44 AM

Quote:

Originally Posted by eco

why not simply use the '-R' option in grep?

Code:

# grep -Rl 'x{ef}\\x{bb}\\x{bf}' *

The problem is that I can's use grep directly on files, because the non-ascii characters are not recognized by it. The Perl script recognize it, so i need to use some perl "hack" in order to see what file contains this characters.

allend · 01-07-2009, 09:46 AM

Gnu grep has an experimental -P switch for perl regular expressions.
Perhaps 'grep -lR -P \\x{EF}\\x{BB}\\x{BF} .' run from top directory will give what you want.

chrism01 · 01-07-2009, 07:38 PM

You should be able to adapt this:

Code:

#!/usr/bin/perl -w
use File::Find;

use strict;

@ARGV = ('.');
my ($str, $fname, $rec);

sub check
{
    $fname = $_;
    open(FILE, "<", $fname) or die "unable to open $fname: $!\n";
    while(defined($rec=<FILE>))
    {
        chomp($rec);
        if( $rec =~ /zxc/ )
        {
            print "FNAME: $fname\n";
            print "FULLPATH: $File::Find::name\n";
        }
    }
    close(FILE) or die "unable to open $fname: $!\n";
}
find(\&check, @ARGV);

It actually searches for a string (zxc) but you can change that.