Decypting file if see bad file then move to bad dir

dotran · 11-13-2012, 12:12 PM

Please help with this syntax if or case

Here is the good log then decypted file and create trigger file. If see the bad log need to move files to /export/home/mytmp/test1/Bad_Files and send out email then exit not do anything. So far I have no luck yet, cause always have error “gpg: WARNING: using insecure memory!” when decrypting files

Good log:
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: encrypted with 2048-bit RSA key, ID 4C0F8A06, created 2012-02-03
"TEST <dsec-1@test.com>"

Bad log:
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: encrypted with 2048-bit RSA key, ID 4C0F8A06, created 2012-02-03
"TEST <dsec-1@test.com>"

gunzip: Rx_test_20110702001.dat.gz: unexpected end of file
gpg: block_filter 1093f0: read error (size=10513,a->size=10513)
gpg: block_filter 1093a0: read error (size=15835,a->size=15835)
gpg: mdc_packet with invalid encoding
gpg: decryption failed: invalid packet
gpg: block_filter: pending bytes!
gpg: block_filter: pending bytes!

Quote:

KEY=`cat /usr/local/apps/secured/test_passphrase`
for Dotran in Rx_test*dat.gz
do

OUT_FILE=Decrypted_${Dotran}

gunzip -c ${Dotran} | \
gpg --decrypt --no-tty --passphrase ${KEY} | \
gzip -c > ${OUT_FILE}

if [[ $? != 0 ]] ; then
#if [[ $? == 0 ]] ; then
#case $Dotran in
echo "[ FAILED ] -- File had move to to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [${Dotran}] corruption or still transfer." abc@test.com
mv -f Rx_test*dat.gz /export/home/mytmp/test1/Bad_Files
#rm -rf Decrypted_*
exit 0
fi
done

touch TRIG_FILE

exit 0

suicidaleggroll · 11-13-2012, 12:29 PM

I'm guessing it's your gunzip that's crashing? How big are these files and how time-critical is the processing?

You could always throw a "gunzip -t" in the script before the actual gunzip | gpg | gzip command. If the gunzip -t fails, it's a bad file, and you can throw it in the bad directory before doing the processing on it.

Also, why are you moving all of Rx_test*dat.gz to the bad directory when one bad file is encountered? Wouldn't you want to just move $Dotran to the bad directory instead?

dotran · 11-13-2012, 12:41 PM

Thank for reply. Code below work perfect. I just want add more loop if bad file then move to bad dir.

I'm guessing it's your gunzip that's crashing? (No) How big are these files and how time-critical is the processing? (Could be multiple files)

Also, why are you moving all of Rx_test*dat.gz to the bad directory when one bad file is encountered? Wouldn't you want to just move $Dotran to the bad directory instead? Yes...I just typo on that when test the files. And the files really big 3gb or 4gb. Don't want gunzip -t check first and want go head decrypt if see the error then failed. Thanks

Quote:

KEY=`cat /usr/local/apps/secured/test_passphrase`
for Dotran in Rx_test*dat.gz
do

OUT_FILE=Decrypted_${Dotran}

gunzip -c ${Dotran} | \
gpg --decrypt --no-tty --passphrase ${KEY} | \
gzip -c > ${OUT_FILE}

suicidaleggroll · 11-13-2012, 12:50 PM

Quote:

Originally Posted by dotran

I'm guessing it's your gunzip that's crashing? (No)

It looks like it to me, from your bad log file:

Code:

gunzip: Rx_test_20110702001.dat.gz: unexpected end of file

A gunzip -t would catch that problem before you enter into your processing. Something like:

Code:

gunzip -t $Dotran
if [[ $? != 0 ]]; then
   echo "[ FAILED ] -- File had move to to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [${Dotran}] corruption or still transfer." abc@test.com
   mv -f $Dotran /export/home/mytmp/test1/Bad_Files
else
   gunzip -c ${Dotran} | gpg --decrypt --no-tty --passphrase ${KEY} | gzip -c > ${OUT_FILE}
fi

Quote:

Originally Posted by dotran

How big are these files and how time-critical is the processing? (Could be multiple files)

That didn't answer my question. I'm asking if the added processing time caused by the insertion of a gunzip -t before the gunzip | gpg | gzip would adversely affect the operation of the system. If these are huge files and you're already pushing the system to its limits to keep up with some real time deadline, then this could cause a problem. If it's just a few small files then it shouldn't matter.

Checking the exit status of a gunzip -t before running your processing is a much cleaner way to handle this problem than parsing a log file in my opinion.

dotran · 11-13-2012, 01:01 PM

Thank for your reply again. And this code will loop for multiple files if see one good file then decrypt and bad file then move? I will test the code more.

Quote:

gunzip -t $Dotran
if [[ $? != 0 ]]; then
echo "[ FAILED ] -- File had move to to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [${Dotran}] corruption or still transfer." abc@test.com
mv -f $Dotran /export/home/mytmp/test1/Bad_Files
else
gunzip -c ${Dotran} | gpg --decrypt --no-tty --passphrase ${KEY} | gzip -c > ${OUT_FILE}
fi

suicidaleggroll · 11-13-2012, 01:05 PM

The code I posted is intended to be placed inside your for loop, replacing your current gunzip | gpg | gzip and error checking.

Code:

KEY=`cat /usr/local/apps/secured/test_passphrase`
for Dotran in Rx_test*dat.gz
do
   OUT_FILE=Decrypted_"$Dotran"

   gunzip -t "$Dotran"
   if [[ $? != 0 ]]; then
      echo "[ FAILED ] -- File has been moved to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [$Dotran] corruption or still transfer." abc@test.com
      mv -f "$Dotran" /export/home/mytmp/test1/Bad_Files
   else
      gunzip -c "$Dotran" | gpg --decrypt --no-tty --passphrase $KEY | gzip -c > "$OUT_FILE"
   fi
done

touch TRIG_FILE

exit 0

FYI - you don't need {} around your variable names unless there are other characters nearby that could be misinterpreted as part of the variable name. Eg:

Code:

var=1
echo ${var}  # Prints "1"
echo $var    # Also prints "1"
echo ${var}a # Prints "1a"
echo $vara   # Prints nothing, since "vara" is the name of a variable that has not been declared yet

You don't do this in your code, which means the {} are unnecessary (but harmless). You should quote the variable substitutions though, in case there are spaces or other "bad" characters in the file names.

dotran · 11-14-2012, 01:05 PM

Thanks. After a lot the test and I reallly like this code work great, but if we have the wrong password then gunzip -t think is good file and start do decrypting files and create trigger file too. How do we check this syntax password "$KEY" ? Please help or any input on this?

Quote:

KEY=`cat /usr/local/apps/secured/test_passphrase`
for Dotran in Rx_test*dat.gz
do
OUT_FILE=Decrypted_"$Dotran"

gunzip -t "$Dotran"
if [[ $? != 0 ]]; then
echo "[ FAILED ] -- File has been moved to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [$Dotran] corruption or still transfer." abc@test.com
mv -f "$Dotran" /export/home/mytmp/test1/Bad_Files
exit 0
else
gunzip -c "$Dotran" | gpg --decrypt --no-tty --passphrase $KEY | gzip -c > "$OUT_FILE"
fi
done

touch TRIG_FILE

exit 0

suicidaleggroll · 11-14-2012, 01:20 PM

In that case, it may be best to just split up your gunzip | gpg | gzip command into three separate parts. Run them each individually, and check the exit status at each step along the way.

dotran · 11-14-2012, 02:35 PM

Thanks for your input, but don't want to split up your gunzip | gpg | gzip command into three separate parts, cause waste a mount of time to gunzip first (file over 3gb). Is way simple syntac check at the end decypting if see error "bad passphrase" then move to bad directory too? Thanks

gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: encrypted with 2048-bit RSA key, ID 4C0F8A06, created 2012-02-03
"TEST <dsec-1@test.com>"
gpg: public key decryption failed: bad passphrase
gpg: decryption failed: secret key not available

Quote:

else
gunzip -c "$Dotran" | gpg --decrypt --no-tty --passphrase $KEY | gzip -c > "$OUT_FILE"
fi

dotran · 11-15-2012, 01:46 PM

After a lot tested and I see the decypted_* file always empty. I come with another check highlight red and see the script work really great. Any different solution or any put on this. Thank

-rw-rw-r-- 1 ca7prod ftpusers 20 Nov 15 12:43 Rx_test12323.dat.gz

gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: encrypted with 2048-bit RSA key, ID 4C0F8A06, created 2012-02-03
"Rx_test12323.dat.gz <dsec-1@test.com>"
gpg: public key decryption failed: bad passphrase
gpg: decryption failed: secret key not available

Quote:

KEY=`cat /usr/local/apps/secured/test_passphrase`
for Dotran in Rx_test*dat.gz
do
OUT_FILE=Decrypted_"$Dotran"

gunzip -t "$Dotran"
if [[ $? != 0 ]]; then
echo "[ FAILED ] -- File has been moved to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [$Dotran] corruption or still transfer." abc@test.com
mv -f "$Dotran" /export/home/mytmp/test1/Bad_Files
else
gunzip -c "$Dotran" | gpg --decrypt --no-tty --passphrase $KEY | gzip -c > "$OUT_FILE"
#gunzip -c "$Dotran" | gpg --decrypt --no-tty --passphrase '${KEY}' | gzip -c > "$OUT_FILE"
if [ `zcat "$OUT_FILE" | head -n 1 | wc -c ` -eq 0 ]; then
rm -rf "$OUT_FILE" && mv -f "$Dotran" /export/home/mytmp/test1/Bad_Files
echo "[ FAILED ] -- File has been moved to /export/home/mytmp/test1/Bad_Files." | mailx -s "[ FAILED ] -- File{s} [$Dotran] gpg: bad passphrase key or secret key not available." abc@test.com
exit 0
else
echo "[$Dotran decrypting is good file.]"
fi
fi
done

touch TRIG_FILE

exit 0