[SOLVED] Getting incorrect ouput of array

addy54 · 03-10-2016, 01:09 AM

Hello Folks,

How are you all?

I have written below script to get the list of files from ftp.

Code:

#!/bin/bash 
#set -x
#get a list of files and dates from ftp and remove files older than n days

ftpserver="pns95a-4402.corpny.csfb.com"
ftpuser="adoshi9"
ftppass=cs@Cogni2016
putdir="/app/opera/operaqa7/TgtFiles/PAC/test"
ndays=15


#work out cutoff dates.

MM=`date --date="$ndays days ago" +%b`
DD=`date --date="$ndays days ago" +%d`
TT=`date --date="$ndays days ago" +%s`

echo removing files older then $MM $DD from $ftpsite directory $putdir

#get dirlist from ftp

listing=`ftp -i $ftpserver << EOF
user $ftpuser $ftppass
binary
cd $putdir
ls -l
quit
EOF
`
filelist=( $(printf "( $listing\n )" | tail -n +5 |head -n -1 | awk '{print $9'} ) )
echo "${filelist[@]}"
echo "${#filelist[@]}"
actuallist=( $(printf "( $listing\n )" | tail -n +5 |head -n -1 ) )


for (( FNO=0; FNO < ${#filelist[@]}; FNO++ ));do

fdate="${actuallist[`expr $FNO + 5`]} ${actuallist[`expr $FNO + 6`]} ${actuallist[`expr $FNO + 8`]}"
echo "$fdate";
done

Basically I am trying to get the list of files from ftp and compare it with current date timestamp. If it is older than 15 days it should get deleted with this script.

The issue is that in above code ouput of
echo "${#filelist[@]}" is 6 and output of
echo "${actuallist[@]}" is as below.

-rw-rw-rw- 1 addy test 2540 Sep 15 05:47 abc-20150630-20150630.gz
-rw-rw-rw- 1 addy test 126 Sep 15 05:47 abc-20150630-20150630.gz.tmp.ftp.dtl
-rw-rw-rw- 1 addy test 27676 Sep 11 2015 PAC-20150630-20150630.gz
-rw-rw-rw- 1 addy test 126 Sep 14 16:32 PAC-20150630-20150630.gz.tmp.ftp.dtl
-rw-rw-rw- 1 addy test 8890 Sep 15 04:22 AMS-20150630-20150630.gz
-rw-rw-rw- 1 addy test 126 Sep 15 04:22 AMS-20150630-20150630.gz.tmp.ftp.dtl

When script executes for loop it space as field seperater and consider the string as pne argument.

So when value of FN=0 then ouput of echo "$fdate"(which seems correct) is
Sep 15 abc-20150630-20150630.gz

But when value of FN increases in next iteration FN=1 and ouput of echo "$fdate" is
15 05:47 -rw-rw-rw- which is not supposed to be. Here I should get

Sep 15 abc-20150630-20150630.gz.tmp.ftp.dtl

what can be done here to achive above output?

grail · 03-10-2016, 01:56 AM

So you are going to need to be a little clearer (at least I am getting a bit confused).

Firstly why are we comparing the output of counting items in the array and then what the items in the array are?

Code:

echo "${#filelist[@]}" <- - this returns a count

echo "${actuallist[@]}" <- - this return the items in the array

I have tried to perform some of your code locally and it does not perform as expected / shown:

1. printf on my computer does not need the brackets and it is causing the first round bracket to be the data stored in first item of the array

2. Not worrying about above point, the printf / tail / head array produced gives me every part of the listing as an array element, ie. -rw-rw-rw- is in its own element

So the above leads me to wonder if either IFS has been altered or ftp returns data differently.

Also, arrays do not require the expr command:

Code:

${actuallist[`expr $FNO + 5`]}

${actuallist[FNO + 5]}

So I know I didn't answer your problem, but as you can see some of the information is a little confusing

addy54 · 03-10-2016, 02:49 AM

Thanks for your response Grail.

Now answer to your question.

Q:Firstly why are we comparing the output of counting items in the array and then what the items in the array are?

Check the output of below code. May be you get an idea.

Code:

filelist=( $(printf "( $listing\n )" | tail -n +5 |head -n -1 | awk '{print $9'} ) )
echo "${filelist[@]}"
echo "${#filelist[@]}"
echo
echo
actuallist=( $(printf "( $listing\n )" | tail -n +5 |head -n -1 ) )
echo "${actuallist[@]}"
echo "${#actuallist[@]}"

abc-20150630-20150630.gz abc-20150630-20150630.gz.tmp.ftp.dtl PAC-20150630-20150630.gz PAC-20150630-20150630.gz.tmp.ftp.dtl AMS-20150630-20150630.gz AMS-20150630-20150630.gz.tmp.ftp.dtl
6

-rw-rw-rw- 1 operaqa7 opera 2540 Sep 15 05:47 abc-20150630-20150630.gz -rw-rw-rw- 1 operaqa7 opera 126 Sep 15 05:47 abc-20150630-20150630.gz.tmp.ftp.dtl -rw-rw-rw- 1 operaqa7 opera 27676 Sep 11 2015 PAC-20150630-20150630.gz -rw-rw-rw- 1 operaqa7 opera 126 Sep 14 16:32 PAC-20150630-20150630.gz.tmp.ftp.dtl -rw-rw-rw- 1 operaqa7 opera 8890 Sep 15 04:22 AMS-20150630-20150630.gz -rw-rw-rw- 1 operaqa7 opera 126 Sep 15 04:22 AMS-20150630-20150630.gz.tmp.ftp.dtl
54

grail · 03-10-2016, 03:18 AM

Ok, that is what I expected

(btw. still don't need the brackets for printf ... you don't see any of it in your output due to tail and head)

Now you are faced with the general problem of parsing ls output, which is generally dangerous as date formatting and white space in file names can cause all sorts of issues.

To help alleviate some of the issue with the actuallist data, you could try adjusting the IFS value to be just a newline. This should at least then give you the same 6 lines.

Another alternative would be to use stat instead of ls and then only print out the information you want:

Code:

$ ls -l f4
-rw-r--r-- 1 grail users 148 24.01.2016 15:50 f4
$ stat -c "%Y|%n" f4
1453621844|f4

Here I have gone with the time since epoch, but if you look up the alternatives in man you can find which one suits you best.
Also, I have used a pipe as the delimiter to then set IFS and have the array lined up accordingly.

Hope some of that helps.

bonnydeal · 03-10-2016, 04:05 AM

I hope those are not your real login credentials. If they are please change them right now before you do anything else.

Quote:

Originally Posted by addy54

Hello Folks,

How are you all?

I have written below script to get the list of files from ftp.

Code:

#!/bin/bash 
#set -x
#get a list of files and dates from ftp and remove files older than n days

ftpserver="pns95a-4402.corpny.csfb.com"
ftpuser="adoshi9"
ftppass=cs@Cogni2016
putdir="/app/opera/operaqa7/TgtFiles/PAC/test"
ndays=15


#work out cutoff dates.

MM=`date --date="$ndays days ago" +%b`
DD=`date --date="$ndays days ago" +%d`
TT=`date --date="$ndays days ago" +%s`

echo removing files older then $MM $DD from $ftpsite directory $putdir

#get dirlist from ftp

listing=`ftp -i $ftpserver << EOF
user $ftpuser $ftppass
binary
cd $putdir
ls -l
quit
EOF
`
filelist=( $(printf "( $listing\n )" | tail -n +5 |head -n -1 | awk '{print $9'} ) )
echo "${filelist[@]}"
echo "${#filelist[@]}"
actuallist=( $(printf "( $listing\n )" | tail -n +5 |head -n -1 ) )


for (( FNO=0; FNO < ${#filelist[@]}; FNO++ ));do

fdate="${actuallist[`expr $FNO + 5`]} ${actuallist[`expr $FNO + 6`]} ${actuallist[`expr $FNO + 8`]}"
echo "$fdate";
done

Basically I am trying to get the list of files from ftp and compare it with current date timestamp. If it is older than 15 days it should get deleted with this script.

The issue is that in above code ouput of
echo "${#filelist[@]}" is 6 and output of
echo "${actuallist[@]}" is as below.

-rw-rw-rw- 1 addy test 2540 Sep 15 05:47 abc-20150630-20150630.gz
-rw-rw-rw- 1 addy test 126 Sep 15 05:47 abc-20150630-20150630.gz.tmp.ftp.dtl
-rw-rw-rw- 1 addy test 27676 Sep 11 2015 PAC-20150630-20150630.gz
-rw-rw-rw- 1 addy test 126 Sep 14 16:32 PAC-20150630-20150630.gz.tmp.ftp.dtl
-rw-rw-rw- 1 addy test 8890 Sep 15 04:22 AMS-20150630-20150630.gz
-rw-rw-rw- 1 addy test 126 Sep 15 04:22 AMS-20150630-20150630.gz.tmp.ftp.dtl

When script executes for loop it space as field seperater and consider the string as pne argument.

So when value of FN=0 then ouput of echo "$fdate"(which seems correct) is
Sep 15 abc-20150630-20150630.gz

But when value of FN increases in next iteration FN=1 and ouput of echo "$fdate" is
15 05:47 -rw-rw-rw- which is not supposed to be. Here I should get

Sep 15 abc-20150630-20150630.gz.tmp.ftp.dtl

what can be done here to achive above output?