LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-08-2019, 07:05 PM   #16
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781

yeah, the collision thing did cross my mind, I don't know what the statistical probability is I guessed it was very low.
using md5sum was just a hack

The thing is.
If all this data is in a database then it is probably better to query the database directly instead of exporting data and messing about with grep, awk, cut, uniq, diff.

I'm sure a solution can be achieved with either bash, awk, perl, python.
( not a script using all of those, but a script written for either one )
Ultimately executing a report run on the database is going to be far more powerful/flexible and easier once the time has been invested.
 
Old 11-09-2019, 08:42 AM   #17
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,267

Original Poster
Rep: Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914
Quote:
Originally Posted by Firerat View Post
yeah, the collision thing did cross my mind, I don't know what the statistical probability is I guessed it was very low.
using md5sum was just a hack

The thing is.
If all this data is in a database then it is probably better to query the database directly instead of exporting data and messing about with grep, awk, cut, uniq, diff.

I'm sure a solution can be achieved with either bash, awk, perl, python.
( not a script using all of those, but a script written for either one )
Ultimately executing a report run on the database is going to be far more powerful/flexible and easier once the time has been invested.
i agree, the mainframe dataset is pulled from db2, vsam, i.d.m.s., websphere mqueues and other datasets.

to get an additional column added, business solutions delivery would have to create a service request; then, app-dev would need to design it; and, enterprise release management would need to approve the budget.
 
Old 11-09-2019, 01:59 PM   #18
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Quote:
Originally Posted by schneidz View Post
i agree, the mainframe dataset is pulled from db2, vsam, i.d.m.s., websphere mqueues and other datasets.

to get an additional column added, business solutions delivery would have to create a service request; then, app-dev would need to design it; and, enterprise release management would need to approve the budget.
ahh,

yeah I understand ... Bureaucracy
 
Old 11-18-2019, 11:57 PM   #19
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,685

Rep: Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492Reputation: 2492
I have to say that, to me, this is screaming out to be done in Perl (eg Firerat post #16).

This sort of data munging is something it's excellent at (I used to do a lot of this sort of thing) and it may help to know that both arrays and hashes are both first class data types in Perl.
Regexes are also built-in.
 
Old 11-19-2019, 05:23 AM   #20
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Void, Debian, Puppy, Raspbian
Posts: 3,476

Rep: Reputation: 222Reputation: 222Reputation: 222
CSVs will import directly into SQLite

then you can just mess about with SQL to find what you want.
it's monster fast too
 
1 members found this post helpful.
Old 11-23-2019, 05:49 PM   #21
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,267

Original Poster
Rep: Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914
^ big up; but, this particular server doesnt have sqlite3 installed.
 
Old 11-23-2019, 05:57 PM   #22
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Quote:
Originally Posted by schneidz View Post
^ big up; but, this particular server doesnt have sqlite3 installed.
it is a shame you can not install sqlite3, or get the CSV data to a machine that does have it installed.

I assume some policy is in place limiting the installation of programs or the export of data files.

if you provide data samples which resemble the real data, and fully explain the "rules of the game" we can come up something.

until then, you are on your own.
 
1 members found this post helpful.
Old 11-25-2019, 04:21 PM   #23
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Void, Debian, Puppy, Raspbian
Posts: 3,476

Rep: Reputation: 222Reputation: 222Reputation: 222
oh dear, clone and build it? it builds easy, i have even built it on M$
 
Old 11-26-2019, 04:51 PM   #24
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,267

Original Poster
Rep: Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914
Quote:
Originally Posted by Firerat View Post
it is a shame you can not install sqlite3, or get the CSV data to a machine that does have it installed.

I assume some policy is in place limiting the installation of programs or the export of data files.

if you provide data samples which resemble the real data, and fully explain the "rules of the game" we can come up something.

until then, you are on your own.
i actually have a sqlite instance on a redhat virtual server that isnt locked down as far as hipaa data is concerned. this a.i.x. server doesnt have it and i aint got r00t.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] bash suggestions to convert horizontal columns to vertical columns Guyverix Programming 14 01-24-2013 12:03 PM
Comparing Directories (diff..?) with its File's Sizes, Ignoring Filenames? mrm5102 Linux - Newbie 2 04-23-2012 09:42 AM
way of ignoring out of order lines in diff? Geneset Linux - Software 2 06-04-2009 11:01 AM
Selecting certain parts of a list of columns in BASH mikejreading Linux - Newbie 6 05-07-2009 05:48 AM
diff / patch ignoring changes to particular lines Kikazaru Linux - General 2 03-09-2009 11:57 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration