Search for exact repetitions in text file

unihiekka · 11-23-2010, 11:20 PM

Hi!

I've got a big text file in which I know have probably made some typos (LaTeX). Sometimes I rewrite sentences several times and then end up with double pieces like "the the" or "is is" without noticing it. Most spell checkers that I can use in LaTeX are very basic so they do not notice these grammar errors. Is there a way that I can search for these repetitions by hand using sed or awk or something along these lines? Is there an app for that?

Thanks.

grail · 11-24-2010, 12:17 AM

Well its crude but you could try something like:

Code:

awk '{for(x=1;x<=NF;x++)for(y=x+1;y<=NF;y++)if($x == $y){print NR,$x;break}}' file

Obviously if you have something like 'is is is' it will show 2 duplications occurred on this line.
If you change break to next then you will only be advised of each line once.

Kenhelm · 11-24-2010, 03:48 PM

Code:

echo 'This looks for for repeated words
within a line and prints the the
line number if if any are found' |
grep -nE  '\b(\w+)[[:space:]]+\1\b' 

1:This looks for for repeated words
2:within a line and prints the the
3:line number if if any are found

Code:

echo 'This looks for repeated
words separated by a newline. It uses
uses a sliding window of two lines 
through the text and prints the
line number of the second
second line when a match is found.'| 
sed -rn 'N; /(\b\w+)[[:space:]]*\n[[:space:]]*\1\b/{=;p}; D'

3
words separated by a newline. It uses
uses a sliding window of two lines
6
line number of the second
second line when a match is found.

chrism01 · 11-24-2010, 10:23 PM

The 2nd Ed Perl Cookbook suggests

Code:

$/ = '';                      # paragrep mode
while (<>) {
    while ( m{
                \b            # start at a word boundary (begin letters)
                (\S+)         # find chunk of non-whitespace
                \b            # until another word boundary (end letters)
                (
                    \s+       # separated by some whitespace
                    \1        # and that very same chunk again
                    \b        # until another word boundary
                ) +           # one or more sets of those
             }xig
         )
    {
        print "dup word '$1' at paragraph $.\n";
    }
}