Well, I'd say you want to keep the < and > cause they indicate which file the line resides in. It would get sort of messy looking at more lines with differences.
But to remove the other stuff, you could easily grep them out.
Code:
diff file1 file2 | grep -v "^---" | grep -v "^[0-9c0-9]"
Code:
[trickykid@your-momma ~]$ diff file1 file2 | grep -v "^---" | grep -v "^[0-9c0-9]"
< blah
> blah 2