yeah, the collision thing did cross my mind, I don't know what the statistical probability is I guessed it was very low.
using md5sum was just a hack

The thing is.
If all this data is in a database then it is probably better to query the database directly instead of exporting data and messing about with grep, awk, cut, uniq, diff.

I'm sure a solution can be achieved with either bash, awk, perl, python.
( not a script using all of those, but a script written for either one )
Ultimately executing a report run on the database is going to be far more powerful/flexible and easier once the time has been invested.

i agree, the mainframe dataset is pulled from db2, vsam, i.d.m.s., websphere mqueues and other datasets.

to get an additional column added, business solutions delivery would have to create a service request; then, app-dev would need to design it; and, enterprise release management would need to approve the budget.

ahh,

yeah I understand ... Bureaucracy

I have to say that, to me, this is screaming out to be done in Perl (eg Firerat post #16).

This sort of data munging is something it's excellent at (I used to do a lot of this sort of thing) and it may help to know that both arrays and hashes are both first class data types in Perl.
Regexes are also built-in.

CSVs will import directly into SQLite

then you can just mess about with SQL to find what you want.
it's monster fast too

1 members found this post helpful.

^ big up; but, this particular server doesnt have sqlite3 installed.

it is a shame you can not install sqlite3, or get the CSV data to a machine that does have it installed.

I assume some policy is in place limiting the installation of programs or the export of data files.

if you provide data samples which resemble the real data, and fully explain the "rules of the game" we can come up something.

until then, you are on your own.

1 members found this post helpful.

oh dear, clone and build it? it builds easy, i have even built it on M$

i actually have a sqlite instance on a redhat virtual server that isnt locked down as far as hipaa data is concerned. this a.i.x. server doesnt have it and i aint got r00t.