[SOLVED] variable string file base

marq_new · 05-01-2011, 09:36 PM

Hi,
I'm a new linux programmer. can you help me with my script, please.

this is may script:

Code:

IDP_CODE=`cat $RefTbl"idp_code.txt"|sort|uniq`

#dialed digit with "*" character

cat $INPUTRaw$DDay1$CHour1$CMin2|awk -F~ '{print $22":"$11","$12","$13","$22","$
7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44}'|awk -F, '$3 ~ /^639|^9
/'|awk -F, '$1 ~ /*/'|sort >>  $OUTPUTDir"IDP/Temp.txt"

cat $INPUTRaw$PDay1$PHour1$PMin1|awk -F~ '{print $22":"$11","$12","$13","$22","$
7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44}'|awk -F, '$3 ~ /^639|^9
/'|awk -F, '$1 ~ /*/'|sort >>  $OUTPUTDir"IDP/Temp.txt"


#dialed digit with "#" character
cat $INPUTRaw$DDay1$CHour1$CMin2|awk -F~ '{print $22":"$11","$12","$13","$22","$
7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44}'|awk -F, '$3 ~ /^639|^9
/'|awk -F, '$1 ~ /#/'|sort >>  $OUTPUTDir"IDP/Tempb.txt"
cat $INPUTRaw$PDay1$PHour1$PMin1|awk -F~ '{print $22":"$11","$12","$13","$22","$
7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44}'|awk -F, '$3 ~ /^639|^9
/'|awk -F, '$1 ~ /#/'|sort >>  $OUTPUTDir"IDP/Tempb.txt"

for d in $IDP_CODE
 do
   #dialed digit with "*" character
   cat $OUTPUTDir"IDP/Temp.txt"|grep -av ^$d"0907"|grep -av ^$d"0908"|grep -av ^
$d"0909"|grep -av ^$d"0910"|grep -av ^$d"0912"|grep -av ^$d"0918"|grep -av ^$d"0
919"|grep -av ^$d"0920"|grep -av ^$d"0921"|grep -av ^$d"0928"|grep -av ^$d"0929"
|grep -av ^$d"0930"|grep -av ^$d"0938"|grep -av ^$d"0939"|grep -av ^$d"0948"|gre
p -av ^$d"0949"|grep -av ^$d"0999"|uniq >> $OUTPUTDir"IDP/re_extract_"$DDay1$CHo
ur1$CMin".txt"
   mv $OUTPUTDir"IDP/re_extract_"$DDay1$CHour1$CMin".txt" $OUTPUTDir"IDP/Temp.tx
t"
done

#apped all trans with # character
cat $OUTPUTDir"IDP/Tempb.txt" >> $OUTPUTDir"IDP/Temp.txt"
cat $OUTPUTDir"IDP/Temp.txt"|awk -F: '{print $2}'|awk -F~ '{print $1":"$2}'|uniq > $OUTPUTDir"IDP/re
_extract_"$DDay1$CHour1$CMin".txt"

Problem: Output file doest not exclude the values in grep -av

thanks in advance.

David the H. · 05-02-2011, 08:08 AM

I'm not even going to try to guess without some more background info. Care to explain what your script is supposed to do?

How about giving us an example of the input it's supposed to be processing? What is the expected output, and what is the actual output you get? What else do we need to know to understand what the script is doing? Be specific.

Also, your formatting is highly unreadable. I had to take several minutes just to break it up into something more easily understandable. Here it is with each command given its own line, but with no other changes made.

Code:

#!/bin/bash

IDP_CODE=` cat $RefTbl"idp_code.txt" | sort | uniq `

#dialed digit with "*" character

cat $INPUTRaw$DDay1$CHour1$CMin2 | \
awk -F~ '{ print $22":"$11","$12","$13","$22","$7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44 }' | \
awk -F, '$3 ~ /^639|^9/' | \
awk -F, '$1 ~ /*/' | \
sort >>  $OUTPUTDir"IDP/Temp.txt"

cat $INPUTRaw$PDay1$PHour1$PMin1 | \
awk -F~ '{ print $22":"$11","$12","$13","$22","$7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44 }' | \
awk -F, '$3 ~ /^639|^9/' | \
awk -F, '$1 ~ /*/' | \
sort >>  $OUTPUTDir"IDP/Temp.txt"


#dialed digit with "#" character

cat $INPUTRaw$DDay1$CHour1$CMin2 | \
awk -F~ '{ print $22":"$11","$12","$13","$22","$7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44 }' | \
awk -F, '$3 ~ /^639|^9/' | \
awk -F, '$1 ~ /#/' | \
sort >>  $OUTPUTDir"IDP/Tempb.txt"

cat $INPUTRaw$PDay1$PHour1$PMin1 | \
awk -F~ '{ print $22":"$11","$12","$13","$22","$7","$26"~"$27","$28","$29","$35","$36","$37","$38","$44 }' | \
awk -F, '$3 ~ /^639|^9/' | \
awk -F, '$1 ~ /#/' | \
sort >>  $OUTPUTDir"IDP/Tempb.txt"

for d in $IDP_CODE ; do

	#dialed digit with "*" character

	cat $OUTPUTDir"IDP/Temp.txt" | \
	grep -av ^$d"0907" | \
	grep -av ^$d"0908" | \
	grep -av ^$d"0909" | \
	grep -av ^$d"0910" | \
	grep -av ^$d"0912" | \
	grep -av ^$d"0918" | \
	grep -av ^$d"0919" | \
	grep -av ^$d"0920" | \
	grep -av ^$d"0921" | \
	grep -av ^$d"0928" | \
	grep -av ^$d"0929" | \
	grep -av ^$d"0930" | \
	grep -av ^$d"0938" | \
	grep -av ^$d"0939" | \
	grep -av ^$d"0948" | \
	grep -av ^$d"0949" | \
	grep -av ^$d"0999" | \
	uniq >> $OUTPUTDir"IDP/re_extract_"$DDay1$CHour1$CMin".txt"

	mv $OUTPUTDir"IDP/re_extract_"$DDay1$CHour1$CMin".txt" $OUTPUTDir"IDP/Temp.txt"

done

#apped all trans with # character
cat $OUTPUTDir"IDP/Tempb.txt" >> $OUTPUTDir"IDP/Temp.txt"
cat $OUTPUTDir"IDP/Temp.txt" | \
awk -F: '{ print $2 }' | \
awk -F~ '{ print $1":"$2 }' | \
uniq > $OUTPUTDir"IDP/re_extract_"$DDay1$CHour1$CMin".txt"

I can already predict that there's a lot of unnecessary duplication of commands here. Indeed, if we knew what it's supposed to be doing, it's likely the whole thing could be done entirely in a single awk script.

A few points that stand out though to start with.

1) $(..) is recommended over `..`

2) sort has a -u option that does the same thing as uniq.

3) Your script doesn't include definitions for most of the variables used. For example, I'm assuming that $INPUTRaw$DDay1$CHour1$CMin2 references a file name. What's in it? Where are the components that make up this string defined? There must be more to this script than what you've shown.

4) Why use the -a option in grep anyway? That's only used for processing binary files, but your script makes it clear that the input is text. Again, what's the intended output of the command?

5) (added) It looks to me like you have one big awk code block duplicated 4 times. That means that the command could easily be redefined as a single function, at the very least. Write once, use many.

grail · 05-02-2011, 09:23 AM

+1 to all that David said.

archtoad6 · 05-04-2011, 07:27 AM

+2 to David.

marq_new,

Welcome to LQ. Glad to see you jump in w/ a Programming question.
Also, thanks for using "Code:" blocks, at all -- they're much easier to read than the alternatives.