Help w/ script to read file and parse variables

cslink23 · 11-22-2006, 06:33 PM

I need to split a file into 3 smaller files. Specifically,
I'm trying to write a shell (bash) script to:
1. read a file
2. parse the variables in it
3. depending on the value of the 3rd field write the record to 1 of three files.

Here's the input data:
"0123456789",1," 2"
"0123456789",1," 1"
"0123456789",1," 3"
...

I'll ask the user for the input file name. The output files will have the same name as the input with a number added. The number comes from the last field in the input file (ie. 1,2,3). And, the records will be written to each output file based on the value of that last field.

Here's what I'm working with (piecing together from other web samples):

#!/bin/bash
echo -n "Please enter the IQ filename:"
read filename
# test filename and file exists

# loop through input file
branch=`echo $inputline | cut -d"," -f 3`
branch=`awk -F , '{ print $3 }' $inputline`
while branch=`cat $filename | awk -F, '{ print $3 }' $filename`
Note: None of these worked for me BUT I may be mixing recipes.

#while branch=`cat $inputline | cut -d, -f3`

do
case "$branch" in
" 1")
# write " 1" records to output.1
# count output records so I'll know input matches output.
;;
" 2")
# write " 2" records to output.2
;;
" 3")
# write " 3" records to output.3
;;
*)
echo "Bad Branch $branch"
;;
esac
done
exit 0

Thanks for your help.
Link

Sepero · 11-22-2006, 07:06 PM

I don't know how to use awk, but I can see some obvious flaws in your script. First, you are trying to use $inputline, and you haven't even defined it. Second, you define $branch, but then you immediately define it again, making the first define useless. Then you define $branch a third time in your loop, making the second define useless.

cslink23 · 11-22-2006, 07:15 PM

I assumed that it would be viewed as the types of things I've tried. These lines are out of context since they would only "work" with the proper corresponding "while" loops. The whole script is just meant to be pseudocode. Thanks for the reply, though.

Sepero · 11-23-2006, 01:32 AM

Done. Please pay all thanks to the GNU licenses. Without them, Linux could not exist.
EDIT: Oh yes, I did test this code before posting it.

Code:

#!/bin/bash

#Global count variable for counting lines output.
count=0

if [ -f output.1 -o -f output.2 -o -f output.3 ];
then
  echo "One or all output files already exist. Please move or delete them."
  exit 1
fi

echo -n "Please enter the IQ filename: "
filename="test.dat"
echo $filename
#read filename

# test filename and file exists
if [ ! -f "$filename" ];
then
  echo "Sorry, that file does not exist. Please make sure the name is correct."
  exit 1
fi

# loop through input file
while read inputline;
do
  branch=$(echo "$inputline" | cut -d "," -f 3)
  case "$branch" in
    '" 1"')
      echo "$inputline" >> output.1
      let count=count+1
      ;;
    '" 2"')
      echo "$inputline" >> output.2
      let count=count+1
      ;;
    '" 3"')
      echo "$inputline" >> output.3
      let count=count+1
      ;;
    *)
      echo "Bad Branch: $branch"
      let count=count+1
      ;;
esac
done < "$filename"

echo "$count lines processed."

cslink23 · 11-23-2006, 09:33 AM

Sepero -

Thank you so much for putting this together for me. The line:
branch=$(echo "$inputline" | cut -d "," -f 3)
is giving me a syntax error - "syntax error at line 27: `branch=$' unexpected"

Is it possible that my version of /bin/sh is too old? I don't know how to tell the version. This is a Point of Sale server that is supported by another company. If there is a way to update it (and maybe that would be for another forum) I can give it a try.

But, for simplicity, do you know of any way to modify that one statement to work around this problem?

Thanks

Sepero · 11-23-2006, 06:41 PM

Try the backticks, like you had in your original. I believe backticks are generally a bad way to go, because someday you might need to use them inside your statement.

Code:

branch=`echo "$inputline" | cut -d "," -f 3`

matthewg42 · 11-23-2006, 06:53 PM

It seems very strange that bash should be used for this sort of task - I would think perl or awk would be much more suitable.

Perhaps a Haiku can help to answer the question why...

Code:

task to do with bash
it is inappropriate
must be a homework

Sepero · 11-23-2006, 07:38 PM

I agree with you mathew. Especially when I noticed that the script is to output to 3 separate files. Since it was a short script (pretty much a hack), I figured I'd test my Bash skills a little bit. (sort of a self-challenge.

)

cslink23 · 11-23-2006, 08:50 PM

OK - Now that's nice. One more problem (for now)...the "let" statement doesn't work.
"link2.bat: let: not found"

Is there another way to verify all records are accounted for?

Signed, picky

Sepero · 11-23-2006, 09:23 PM

I'm not at home right now, so I can't test this, but you could try:

Code:

count=$(($count + 1))
or perhaps
count=$((count + 1))

It seems awful strange that your shell doesn't support these things. I'm guessing that you're either using a really old version of bash, or you're using a shell that is trying to emulate bash. My bet is on the latter.

cslink23 · 11-24-2006, 04:52 PM

I have no way to convince you that this is not homework. But, I can explain its use. Our Point of Sale server (SCO Unix) has a piece of software called IQ used to extract data from a UniBASIC database. We run several queries using IQ to extract files which are used to create purchase orders. The problem is that we have to run this IQ 3 times - once for each store - generating 3 different files. Each run takes over 1 hour. MY GOAL: I'd like to run this program only once (1 hour) to create a file containing data for all three stores then split the data into 3 files in a script (or whatever works).

I do know some PERL and could probably get it done in that but we don't have PERL on this server and I can't add it and keep our support contract. Also, you'll remember that in one of my posts above, I mentioned that my version of /bin/sh may be old. I can tell from my /etc/passwd that /bin/sh is what is being used - not bash. But, I don't know how to check the version of /bin/sh. If you know how please tell me and I'll report that to you.

I tried your last suggestion for increasing $count and received this message for both formats:
syntax error at line 35: `count=$' unexpected

I appreciate your continued help and assure you that it will be greatly appreciated.

Link

cslink23 · 11-24-2006, 04:56 PM

BTW, I could probably live with this and just add some "wc -l" statements to check the file sizes. But, that's a little less tidy.

Sepero · 11-24-2006, 08:59 PM

Ok, so it's not bash. What shell is it?
If I know, I might be able to figure out how to do math with it.

cslink23 · 11-24-2006, 09:01 PM

The only clue I have is that in the /etc/passwd file the shell assigned to each user is "/bin/sh". I have no idea how to find out further. Please tell me where to look to find out the shell and the version (if you know).

Thanks.

Sepero · 11-25-2006, 02:33 AM

whereis sh
Then when you find sh. Type this:
ls -l [location of sh]

I think the second command should tell you which shell you have.