script to create a file

smturner1 · 10-27-2015, 02:09 PM

I am trying to create a file from an existing file or a file list and I am grep'ing an item out of that file to get the records to populate that new file. I have determined if the files are compressed and whether the file is a list of file names or if it is a single file.

The issue that I am having is constructing the loop around the correct items so that my file is created by grabbing each item out of the file/files.

CL Syntax: file_create.sh input_file(File_List) item(item_list)

Any help you can provide is greatly appreciated.

Code:

#!/bin/ksh
set -x


#-------------------------------------------#
#Check to see if the file(s) are compressed #
#-------------------------------------------#
#---------------------------------------#
#Is this an invoice file or a file list #
#---------------------------------------#
#-------------#
#invoice file #
#-------------#

if [ $1 == "ABC*\.txt*" ]; then
   inv_file=$1

#------------------------#
#Is this file compressed #
#------------------------#

   file $inv_file | grep -i "compress" | read chk

   if [ -z "$chk"  ]; then
      display=cat
   else
      display=zcat
   fi
else

#----------#
#file list #
#----------#
#--------------------------------------------------#
#Is this file compressed. Create cat/zcat variable #
#--------------------------------------------------#
for x in `cat $1`;do
   file $x | grep -i "compress" | read chk

   inv_file=$x
done

   if [ -z "$chk"  ]; then
      display=cat
   else
      display=zcat
   fi

fi

#---------------------------------------#
#Is this a list of items or a single item#
#---------------------------------------#

if [-f "$2"];then
for i in `cat $2`;do
   item=$i
   done
else
  item=$2
fi

#-------------------------------------------------#
# Create the file from the inputs passed into the #
# Commandline                                     #
#-------------------------------------------------
while
$display $inv_file | head -1 >> invoice_output
$display $inv_file | grep $item >> invoice_output
wc -l invoice_output | nawk '(print $1)' | read count
echo "T|$count">>invoice_output

michaelk · 10-28-2015, 08:05 AM

Code:

#----------#
#file list #
#----------#
#--------------------------------------------------#
#Is this file compressed. Create cat/zcat variable #
#--------------------------------------------------#
for x in `cat $1`;do
   file $x | grep -i "compress" | read chk

   inv_file=$x
done

   if [ -z "$chk"  ]; then
      display=cat
   else
      display=zcat
   fi

fi

Do you want to grab the contents of the file list or the contents of each file within the list?

Code:

$display $inv_file | grep $btn >> invoice_output

What is $btn?

zhjim · 10-28-2015, 08:09 AM

Quote:

#----------#
#file list #
#----------#
#--------------------------------------------------#
#Is this file compressed. Create cat/zcat variable #
#--------------------------------------------------#
for x in `cat $1`;do
file $x | grep -i "compress" | read chk

inv_file=$x
done

if [ -z "$chk" ]; then
display=cat
else
display=zcat
fi

If I get that correct $1 contains the file list but you only set cat|zcat once after looping through it all. This might lead to trouble if one of the files is not compressed. You only check the last file in the list for compression.
You would also only $display the last one of the file list due to overwriting $inv_file.

Either use an array to save all files and $display them at the end. Or $display them right within the for loop of the file list.

Not touching on the item_list cause i just dont get it.

smturner1 · 10-29-2015, 08:47 AM

I am trying to account for a couple scenarios. First scenario, an input file ($1) that contains a list of files and a file ($2) that contains a list of items. The second scenario, is a single file with one or more records containing an item from $2. $2 can be a single item (ABC123) or a file containing many items in a list form. $1 can be a single compressed file or a list of compressed/uncompressed files. $2 is never compressed.

I know that my issue is related to my looping constructs. The reason I know this is because every single command works when I execute it on the CLI. The issue as you guys have noted is that the commands are only being executed on the last inv_file and item.

BTW - $btn shoul dhave been $item. Sorry for the confusion.

michaelk · 10-29-2015, 12:06 PM

Code:

while
$display $inv_file | head -1 >> invoice_output

More clarification is required. If $1 is a list that includes compressed files are you trying to grab the first line of every file?

smturner1 · 10-29-2015, 12:39 PM

michaelk,

Yes, I am grabbing the header record from the source file in the list of files/file ($1) and pushing it into the invoice_output file. Then I need to loop through the

Code:

$display $inv_file | grep $item >> invoice_output

to find each record that has the $item (usoc), grab the entire record and add it to the invoice_output. Then it needs to go back to the outer loop, where the header portion exists and add the Trailer which should be "T|(line_number)"

michaelk · 10-29-2015, 02:14 PM

Worst case scenario from your description I think...

Code:

source file
   header
   file 1 (compressed)
        -> file1
        -> file2
   file 3 (not compressed 
        -> file1
        -> file2

Are the lists one item per line? What about a loop that iterates the source file and writes all the list items to a temporary file.

To compare the $2 to the temporary file look at the grep -f option.

onebuck · 11-01-2015, 08:58 AM

Moved: This thread is more suitable in <Programming> and has been moved accordingly to help your thread/question get the exposure it deserves.

grail · 11-01-2015, 10:59 AM

Well I am glad someone can follow this because I am very confused???

I think it would help if you would provide an idea of what a file might look like inside and also the item list in file2?

Also, would you please explain what a 'file list' is? Should this mean there are multiple parameters on the command line (ie more than 2)?

smturner1 · 11-03-2015, 11:52 AM

Here we go...

A file ($1) is an invoice file compressed/uncompressed. There is a header record, detail records, summary records, and a trailer record.

A file List ($1) is a list of invoice files (see above for definition) that may or may not be compressed. Since this script will be executed in the home dir where all of the files reside file path unnecessary.

An $item is a serial number that exists in both the detail record and the summary record. If I grep out abc123 from a source file I could get multiple detail records, but only a single summary record. I want both record types.

The concept: I want to be able to create an invoice file from a production source file based on an invoice file or list of invoice files. I also would like to use a single item or a list of items as the criteria for the detail and summary records.

The lists are one item per line. Both of them.

grail · 11-03-2015, 12:17 PM

Cool, thanks for the details

So if I understand correctly, the process would be:

1. Confirm you received 2 arguments (currently not done)

2. Figure out is $1 is compressed or not ... it would seem here we make an assumption on what type of compression or you can guarantee always the same type?

3. Based on the first line of the file will tell you if you are looking at a single file or one with a list (currently not done). I am also assuming the first line will tell us this

4. Using info from 3 above, we either loop over files or execute the necessary commands, including $2, to get the details

Does this sound about right? If so, some will need answers from you to proceed

michaelk · 11-03-2015, 02:07 PM

From the first post it appears that "ABC*\.txt*" defines an invoice file versus a file list.

It is unclear if the file list also has a header record etc. I assume that you are creating a summary or trailer record by counting the lines in the file i.e.
wc -l invoice_output | nawk '(print $1)' | read count
echo "T|$count">>invoice_output

I think you need to process the invoice file separately from the file list.

Using the eval command you can iterate through the file list whether compressed or uncompressed. The ifs or pseudo code but the loop is real but untested.

Code:

# if $1 is a file list
if [ $1=="compressed" ] 
 cmd=zcat
else
 cmd=cat
fi

for line in $(eval $cmd $1)
do
   if [ $line==file]
      if [ $line=="compressed] 
        zcat $line >> temp.file
     else
        cat $line >> temp.file
     fi
  else
     echo $line >> temp.file  # this is just a guess.
  fi
done 

create header record ...
if [$2 == file]
   grep -f $2 $temp.file >> invoice_output
else
   grep $2 $temp.file >> invoice_output
create summary and trailer record ...