LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-12-2004, 04:03 PM   #1
mychl
Member
 
Registered: Jul 2001
Location: Earth
Posts: 164

Rep: Reputation: 30
Flat File QC Scripting


Hi All, need some help.

I have a linux server that runs php/mysql etc... I'm trying to automate the QC process on various Flat File formats (txt, csv).

All I really need to do is verify the total record length for aeach line of data.

Anyone have any ideas about what type of script I will be able to use to do this. Also, if it's a script I can have run via the web server, that would be great also.

Thanks for any input in advance----

mychl
 
Old 02-13-2004, 03:10 AM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,965
Blog Entries: 11

Rep: Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865Reputation: 865
I don't even know what QC is :}


Cheers,
Tink
 
Old 02-13-2004, 08:19 AM   #3
mychl
Member
 
Registered: Jul 2001
Location: Earth
Posts: 164

Original Poster
Rep: Reputation: 30
Quality Control.... basically just verifying the data....

 
Old 02-13-2004, 11:18 AM   #4
jim mcnamara
Member
 
Registered: May 2002
Posts: 964

Rep: Reputation: 34
This is a start:
Code:
#!/bin/sh
# script: linecnt
# the $1 parameter is the file name
# usage: linecnt <file name>

echo "Line counts for $1"
let counter=0
while read rec
do
    counter=`expr $counter + 1`
    echo "$counter: `echo $rec | wc -c`"    
done  < $1
# eof
 
Old 02-15-2004, 02:19 PM   #5
mychl
Member
 
Registered: Jul 2001
Location: Earth
Posts: 164

Original Poster
Rep: Reputation: 30
Thanks Jim, that gave me a good start.....

Works great for TEXT format, now I'm working on CSV.

I'll post what I work out....

Thanks again!
 
Old 02-20-2004, 11:53 AM   #6
mychl
Member
 
Registered: Jul 2001
Location: Earth
Posts: 164

Original Poster
Rep: Reputation: 30
Here is what I came up with, using perl...

[code]
#!/usr/bin/perl
########################################################################
open(LOGFILE, ">QC_TXT_LOG.log");
print LOGFILE "Begin Parsing All TXT Files\n";
print LOGFILE "========================================\n\n";
# Scan Parent Directory for Sub-Folders
opendir(MAIN, ".");
@dirs=(readdir(MAIN));
closedir(MAIN);
#######################################################################
# Recurse into sub-folders
foreach $dir(@dirs){
if(-d "$dir"){
print LOGFILE "Profile: $dir\n";
print LOGFILE "===============================\n";
opendir(DIR, "$dir");
@files=grep(/\.TXT$/|/\.txt$/,readdir(DIR));
closedir(DIR);
$linecount=0;
foreach $file(@files){
@fileinfo=stat("$dir/$file");
$filesize=((@fileinfo[7])/1000);
$filedate=localtime(@fileinfo[9]);
print LOGFILE "\tFilename: $file\n";
print LOGFILE "\tFile Size: $filesize Kb\n";
print LOGFILE "\tFile Date: $filedate\n";
$linecount=0;
open FILE, "$dir/$file";
while (<FILE>){
$linecount++;
if($.==1){
$first_length=length($_);
print LOGFILE "\t\tRecord Length:\t$first_length\n";
}
$lcount=length($_);
if($lcount!=$first_length){
print LOGFILE "***Line:$. Count:$lcount DOES NOT MATCH PROFILE***\n";
}
}
close FILE;
print LOGFILE "\t\tRecord Count:\t$linecount\n";
}
print LOGFILE "===============================\n";
}
}
[\code]

This script saves my staff and I about 4 hours of work every week......

wahoooooooooooooooo
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
permissions on flat file php authentication blizunt7 Linux - Security 2 11-23-2005 06:13 AM
Manipulating Fields Of Flat File Prasun1 General 10 09-29-2005 11:42 AM
Delete line from flat text file in C zaichik Programming 6 01-26-2005 06:16 PM
Making VSAM file from a flat file in AIX Raghothaman AIX 2 08-25-2004 04:55 PM
creating a flat binary file with gcc wsimmons Programming 2 01-08-2002 11:33 AM


All times are GMT -5. The time now is 02:10 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration