how to use regexp in diff

lexit · 02-10-2011, 04:30 PM

I need to compare 2 files using diff. The problem I've encountered is that I need to exclude certain lines that contain certain phrases. I know that diff supports the -I switch but no matter how I try to form the regexp it doesn't seem to work the way I expect it to. If anyone has used the -I switch before could you please post some examples of how it is used. Thanks.

diff -I "\[skipthisline\]" file1 file2 > output.diff

I need to exclude lines that contain the string "[skipthisline]" but I have no idea what syntax is used after the -I switch. Is is supposed to be included in quotes or slashes /\[skipthisline\]/ or entered without either? I need to include a backslash before each bracket so that it's not interpreted as a set of characters like [a-z] but is instead interpreted as a string. Do I need to use 2 backslashes? "\\[skipthisline\\]"

Is it sufficient to simply type the string I want to match or do I need to match the entire line in order to exclude it from the output?

.*\[skipthisline\].*

or

^.*\[skipthisline\].*$

David the H. · 02-11-2011, 08:11 AM

Apparently, the regex has to match the corresponding line in both files for it to work. Otherwise it still shows it as a difference.

Code:

$ cat fileA.txt
[Fry]
[Leela]
[Bender]
[Farnsworth]
[Amy]
[Hermes]
[Zoidberg]
[Nibbler]

$ cat fileB.txt
[Fry]
[Leela]
[Bender]
[Prof. Farnsworth]
[Amy]
[Hermes]
[Dr. Zoidberg]
[Nibbler]

$ diff fileA.txt fileB.txt
4c4
< [Farnsworth]
---
> [Prof. Farnsworth]
7c7
< [Zoidberg]
---
> [Dr. Zoidberg]

$ diff -I '^\[Dr.*\]' fileA.txt fileB.txt
4c4
< [Farnsworth]
---
> [Prof. Farnsworth]
7c7
< [Zoidberg]
---
> [Dr. Zoidberg]

$ diff -I '^\[.*Zoid.*\]' fileA.txt fileB.txt
4c4
< [Farnsworth]
---
> [Prof. Farnsworth]

$ diff -I 'Zoid.*\]$' fileA.txt fileB.txt
4c4
< [Farnsworth]
---
> [Prof. Farnsworth]

Inserting a blank line somewhere so that the line numbers don't match up also foils the regex match.

As for escaping characters, first realize that protecting them from the shell and from the regex pattern are two separate things. To fully protect from the shell, you usually should use single-quotes. See here for a full discussion on shell quoting: http://www.tldp.org/LDP/abs/html/quoting.html

After you've protected the string from the shell and passed it to the program, then you have to escape the regex-reserved characters, including brackets. There are a couple of ways to do this. Backslashing is one way, as I demonstrated above. You can also use brackets to list a range of acceptable characters, including restricted ones such as brackets.

Code:

diff -I '[[].*Zoid.*[]]$' fileA.txt fileB.txt

But to match a ']', it has to be the first character in the list. This doesn't work:

Code:

$ diff -I '[[].*Zoid.*[ ]]$' fileA.txt fileB.txt

More on regex syntax here: http://www.tldp.org/LDP/abs/html/x16775.html
and here: http://www.regular-expressions.info/posixbrackets.html

lexit · 02-12-2011, 05:02 PM

Excellent tutorial and examples on how to use regexp with diff. Many thanks for your helpful reply.