Got a lot of text that contains some duplicate sentences. How do I detect them? Can sed do the job if the lines are first merged into one big line using: tr '\n' _ < input.txt > output.txt?

Let's say the text looks like this:

echo "hello my dear friend_how are you my dear John_" > temp

We want to detect that "my dear" is repeated twice. Why doesn't the following work?

sed "s/\(.*\).*\1/duplicate:\1/g" < temp

This outputs "duplicate:" followed by new line.

How do you define duplicate? The word "my" appears twice as does the word "dear". Or it could be counted as the phrase "my dear" appearing twice. And should the words or phrases to be checked occur consecutively or anywhere in a single line? Or are you looking for whole sentences which might be spread over several lines?

1 members found this post helpful.

The longest match would be ideal. Could specify a minimum length if needed. I am really looking for sentences made up of dozens of words.

Definitely anywhere in a single line.

Yes but all lines can be merged into one big line as shown above so this is not an issue.

Just to clarify. Given your flexible approach to the definition "line", do you effectively only want to find duplicates in a given sentence?

For example:

What would you like to define as a duplicate in the above?

i would keep them in separate lines and run something likeedit:

this complicates things considering "my| |dear" is not a word delimited by spaces nor a record delimited by lines but a phrase containing a space (similar problem as csv's containing a comma in a field).

also, most parsers would consider friend_how to be 1 word.

the thing is that it needs a basis for comparison, as you know already. but what would that be? it could be anything as this is what you are looking for, anything that matches something else. the letter "a" even being within that file more than once. Seeing that you are stating "some duplicate sentences".

You could write an elaborate two loop algorithm that reads one line at a time, then stores that line and then goes through the entire file looks for a duplicate line then do with it what you will, then goto next line and do the same. Using the period . as a means of identification of a line. then break it down to do three words or two words or even one word as you already have your two loops written to do this. Just a minor change to your code inside to take "words" and not complete sentences then check against the hold-for-comparison variable. It is a variable because it changes with each new value to be compared to.

hypothetical
#outside loop gets next line, runs inner loop again doing same process. when outer loop hits EOF - it quits.

You've already read about some of the pitfalls in doing this with sed and stuff,

when you get the basic working you than can get more elaborate with it in what to check for and how to deal with it.
( I might be backwards logic in how I told the thing is working ) better read this too this shows 4 loops but you get the idea yes?
http://tldp.org/LDP/Bash-Beginners-G...ect_09_03.html

Gosh, I do not know why this is so difficult to communicate. Hydrurga, a line is text with a \n at the end (hex code 0Dh). Text comes in multiple lines, but it doesn't have to, they can all be merged into one if all the \n's are deleted or replaced with _ or whatever character you prefer, even space will do.

So it's all a single line of great length. The letter e may be duplicated thousands of times in this line, but more of interest are long phrases being duplicated.

In the above example, "Ulysses" is duplicated, and also "Ulysses wrote" is duplicated, and "wrote", and "it", and "morning", and "this morning". All are acceptable output. But the long ones are preferred. So how do you exclude the short ones?

I think the easiest way to do it is match a minimum length of 30 in the first pass (ie .............................. in sed syntax), then edit out the duplication manually if we want to which is the purpose of this exercise, then 29 dots in the next pass, edit out this duplication too with an editor, and so on to 10, say.

file
MOD:
correct me if I am wrong
from my understanding of sed it is a simple find / replace tool . you have to tell it what to find first in order for it to replace with something else. it is like you want sed to search a given pattern that is and can only be one pattern
"then edit out the duplication manually if we want to which is the purpose of this exercise"
which supersedes sed because it is suppose to match a pattern then replace not match let you know it has something that matches then move on looking for the next match.

what output are you looking for in this example:
i think you should define how greedy you want your matching to be ?
counting duplicate lines or words would be trivial. counting duplicates of randomly identified groupings of words would take some pseudo-coding.

pseudo : not genuine; sham.

so pseudo-coding still would not work. terminology : pseudo-coding is
This has been Just a friendly fyi announcement.

What is the "ec" in that grep command meant for?

Here's what works on schneidz's text:

The output is:

But bummer, it does not work on large files. It does not match at all on my 1.13 Mb file. But it matches if the file is first truncated to 32000 bytes with head -c 32000.

words starting with 'ec'
hence echo being the only matches.

regular expressions and grep
https://www.digitalocean.com/communi...terns-in-linux

you could capture the match in a var then slip it into sed to be replaced?
pseudo-coding
$1 being off the command line

First you find the unknown duplicate text, then you go to your gedit or leafpad or whatever manual editor you have and decide which of the two instances you discovered you want to keep.