How to loop with different directory

_mz · 01-10-2019, 09:43 PM

I would like to SFTP from A to B and do some file movement.

server A has 4 set of directories:-

set 1

/customermaster/archive
/customermaster/error
/customermaster/log
/customermaster/output

set2

/materialmaster/archive
/materialmaster/error
/materialmaster/log
/materialmaster/output

set 3

/opensalesorder/archive
/opensalesorder/error
/opensalesorder/log
/opensalesorder/output

set 4

/sales_performance/archive
/sales_performance/error
/sales_performance/log
/sales_performance/output

Server B only has folder output of each set. The logic/step is like this:-

From A let say I choose /customermaster/output, SFTP file inside the directory to server B in same directory.
In A, if transfer is successful move the file out to /customermaster/archive
In A, if transfer is fail move the file out to /customermaster/error
Re-transfer file for previous failed transfer

I can only write the scrip up to this level

Code:

#!/bin/bash
...
...
SRCDIR1_SET1=/customermaster/archive
SRCDIR2_SET1=/customermaster/error
SRCDIR3_SET1=/customermaster/log
SRCDIR4_SET1=/customermaster/output
...
...
DSTDIR1=/sales_performance/output
DSTDIR2=/materialmaster/output
DSTDIR3=/opensalesorder/output
DSTDIR4=/customermaster/output

# Transfers the file 
lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput $SRCDIR4_SET1 -O $DSTDIR4; bye" 2>&1 >> $LOG
if [[ "$?" == "0" ]]
then
echo "File transfer successful" |tee -a $LOG
mv $SRCDIR4/* $SRCDIR1_SET1
else
echo "File transfer failed"  |tee -a $LOG
mv $SRCDIR4/* $SRCDIR2_SET1
fi

This is only for 1 set of directory and I got 3 sets of folder some more. I can do for the rest of the folders by repeating the if..fi but it looks like not professional. I am thinking to loop it but the folder structure is not same. With limited knowledge, could someone shed me some lights on this?

Another question is for fail transfer part. How can I re-transfer the file in a scripting way? I mean after step 2/3 is done

l0f4r0 · 01-11-2019, 03:13 AM

Quote:

Originally Posted by _mz

This is only for 1 set of directory and I got 3 sets of folder some more. I can do for the rest of the folders by repeating the if..fi but it looks like not professional. I am thinking to loop it but the folder structure is not same. With limited knowledge, could someone shed me some lights on this?

You could do something like the following:

Code:

myFolders=(customermaster materialmaster opensalesorder sales_performance)
for folder in "${myFolders[@]}"
do
     #...SFTP transfers instructions with variable "$folder"...
done

Stéphane Ascoët · 01-11-2019, 03:35 AM

Quote:

Originally Posted by _mz

Another question is for fail transfer part. How can I re-transfer the file in a scripting way? I mean after step 2/3 is done

Loop in the "failed" directory for sending them again?

l0f4r0 · 01-11-2019, 03:46 AM

Quote:

Originally Posted by _mz

Another question is for fail transfer part. How can I re-transfer the file in a scripting way? I mean after step 2/3 is done

I'm not convinced it's a good idea to retry something that failed without having corrected anything in the meantime...
Except if there was a momentary glitch during your last transfer, the new one is likely to go bad as well.
So maybe IMHO you should just log the actions/results and analyze them by hand before proceeding any further...

_mz · 01-11-2019, 04:15 AM

Quote:

Originally Posted by l0f4r0

You could do something like the following:

Code:

myFolders=(customermaster materialmaster opensalesorder sales_performance)
for folder in "${myFolders[@]"}
do
     #...SFTP transfers instructions with variable "$folder"...
done

Thanks for your help. I have full script here but it is still in progress. Where should I put your one? My weakness is I do not know how to loop it for appropriate folders.

Code:

IPWDSERVER=x.x.x.x
PORT=xx
UserID=xxx
PASSWORD="xxxx"
CURDATE=`date '+%d%m%Y %T'`

outputPath=(
/customermaster/output
/materialmaster/output
/opensalesorder/output
/sales_performance/output
)

archivePath=(
/customermaster/archive
/materialmaster/archive
/opensalesorder/archive
/sales_performance/archive
)

errorPath=(
/customermaster/error
/materialmaster/error
/opensalesorder/error
/sales_performance/error
)

logPath=(
/customermaster/log/log
/materialmaster/log/log
/opensalesorder/log/log
/sales_performance/log/log
)

# This loops the program until all directories listed above is executed
for i in "${outputPath[@]}"
do
   log=${logPath[@]}
   archive=${archivePath[@]}
   error=${errorPath[@]}

   number_of_files=$(ls -A $i | wc -l)

   echo "----START----------------------------" | tee -a $log
   echo "$CURDATE" | tee -a $log
   echo "----Detailed transmission logs are in $log---"

if [ "$number_of_files" == "0" ]; then
   echo "Directory $i is empty" |tee -a $log
else
   echo "Directory $i contains $number_of_files files" |tee -a $log
   ls $i 2>&1 | tee -a $log


# Transfers the file to Web Dispatcher server
lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput $i/* -O $i ; bye" 2>&1 | tee -a $log
if [[ "$?" == "0" ]]
   then
   echo "File transfer successful" |tee -a $log
   mv $i/* $archive |tee -a $log
   echo "All files were moved to archive folder" |tee -a $log
else
   echo "File transfer failed"  |tee -a $log
   mv $i/* $error |tee -a $log
   echo "All files were moved to error folder" |tee -a $log
fi

fi
done

It would be appreciated if you can assist me from here.

_mz · 01-11-2019, 04:19 AM

Quote:

Originally Posted by l0f4r0

I'm not convinced it's a good idea to retry something that failed without having corrected anything in the meantime...
Except if there was a momentary glitch during your last transfer, the new one is likely to go bad as well.
So maybe IMHO you should just log the actions/results and analyze them by hand before proceeding any further...

Ya, you are correct. Thanks for that.

The re-transfer part will be my last portion. Right now, I need to complete the main portion.

I have posted my full script on top of this thread. Would you mind to correct me?

l0f4r0 · 01-11-2019, 04:51 AM

Quote:

Originally Posted by _mz

Thanks for your help. I have full script here but it is still in progress. Where should I put your one? My weakness is I do not know how to loop it for appropriate folders.

Code:

IPWDSERVER=x.x.x.x
PORT=xx
UserID=xxx
PASSWORD="xxxx"
CURDATE=`date '+%d%m%Y %T'`

outputPath=(
/customermaster/output
/materialmaster/output
/opensalesorder/output
/sales_performance/output
)

archivePath=(
/customermaster/archive
/materialmaster/archive
/opensalesorder/archive
/sales_performance/archive
)

errorPath=(
/customermaster/error
/materialmaster/error
/opensalesorder/error
/sales_performance/error
)

logPath=(
/customermaster/log/log
/materialmaster/log/log
/opensalesorder/log/log
/sales_performance/log/log
)

# This loops the program until all directories listed above is executed
for i in "${outputPath[@]}"
do
   log=${logPath[@]}
   archive=${archivePath[@]}
   error=${errorPath[@]}

   number_of_files=$(ls -A $i | wc -l)

   echo "----START----------------------------" | tee -a $log
   echo "$CURDATE" | tee -a $log
   echo "----Detailed transmission logs are in $log---"

if [ "$number_of_files" == "0" ]; then
   echo "Directory $i is empty" |tee -a $log
else
   echo "Directory $i contains $number_of_files files" |tee -a $log
   ls $i 2>&1 | tee -a $log


# Transfers the file to Web Dispatcher server
lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput $i/* -O $i ; bye" 2>&1 | tee -a $log
if [[ "$?" == "0" ]]
   then
   echo "File transfer successful" |tee -a $log
   mv $i/* $archive |tee -a $log
   echo "All files were moved to archive folder" |tee -a $log
else
   echo "File transfer failed"  |tee -a $log
   mv $i/* $error |tee -a $log
   echo "All files were moved to error folder" |tee -a $log
fi

fi
done

It would be appreciated if you can assist me from here.

Actually what I was thinking about was simpler, you don't need so many arrays.
I would do something like the following (untested):

Code:

#!/bin/bash
IPWDSERVER=x.x.x.x
UserID=xxx
PASSWORD=xxxx
CURDATE=$(date '+%d%m%Y %T')
myFolders=(customermaster materialmaster opensalesorder sales_performance)

# This loops the program until all directories listed above is executed
for folder in "${myFolders[@]}"
do
   number_of_files=$(ls -A "$folder" | wc -l)

   echo "----START----------------------------"
   echo "$CURDATE"
   echo "----Detailed transmission logs are in \"$folder\"/log---"

if ((number_of_files == 0))
then
   echo "Directory \"$folder\" is empty"
else
   echo "Directory \"$folder\" contains \"$number_of_files\" files"
   ls "$folder" 2>&1

   # Transfers the file to Web Dispatcher server
   lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput \"$folder\"/* -O \"$folder\" ; bye" 2>&1
   if (($?))
   then
      echo "File transfer failed"
      mv "$folder"/* "$folder"/error
      echo "All files were moved to error folder"
   else
      echo "File transfer successful"
      mv "$folder"/* "$folder"/archive
      echo "All files were moved to archive folder"
   fi
fi
done | tee -a "$folder"/log

Stéphane Ascoët · 01-11-2019, 06:45 AM

If sticking with arrays, naming "$i" something like "$currentoutputpath" would be clearer. And I think you don't need to put "
log=${logPath[@]}
archive=${archivePath[@]}
error=${errorPath[@]}" inside the loop.

michaelk · 01-11-2019, 10:10 AM

Code:

lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput $i/* -O $i ; bye" 2>&1 | tee -a $log

Your syntax is incorrect. The base option -O should be before the source files.

Code:

lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput -O $i  $i/* ; bye" 2>&1 | tee -a $log

BW-userx · 01-11-2019, 11:02 AM

I am curious as to way you're not using absolute paths in your prototype script.

Code:

#!/bin/bash

#set -x

myFolders=(
Desktop Pictures Templates
holdingMusic Public torrents
bin Documents media scripts Videos
Downloads Music SDCard
)

for folder in "${myFolders[@]}"
do
	[[ -d $folder ]] && echo "$folder" || echo "no folder"
	
	[[ -d $(pwd)/$folder ]] && echo "$folder" || echo "no folder $(pwd)/$folder"
	
	[[ -d $HOME/$folder ]] && echo "$folder" || echo "no folder $HOME/$folder"
	
done

using this you can see it is only the last test that does not fail. the script looks in the directory that it is being ran from if no absolute path is given to it to point it elsewhere.

so all of these have to be in a subdir of the script where it is being ran.

Code:

myFolders=(customermaster materialmaster opensalesorder sales_performance)

your destination directories too will fail because it will look in the local directory the script is being ran in and try to put them in the same local directory that it is being ran from.

so should this

Code:

 echo "File transfer failed"
      mv "$folder"/* "$folder"/error
      echo "All files were moved to error folder"
   else
      echo "File transfer successful"
      mv "$folder"/* "$folder"/archive
      echo "All files were moved to archive folder"

not be more something like this?

Code:

#!/bin/bash
IPWDSERVER=x.x.x.x
UserID=xxx
PASSWORD=xxxx
CURDATE=$(date '+%d%m%Y %T')
myFolders=(customermaster materialmaster opensalesorder sales_performance)

ParentDir=$HOME



# This loops the program until all directories listed above is executed
for folder in "${myFolders[@]}"
do
	source=$ParentDir/$folder
	#get number of files inside of the directories
   number_of_files=$(ls -A $source/* | wc -l)

   echo "----START----------------------------"
   echo "$CURDATE"
   echo "----Detailed transmission logs are in \"$source\"/log---"

#check count
if ((number_of_files == 0))
then
   echo "Directory \"$source\" is empty"
else
   echo "Directory "$source contains $number_of_files files"
   
   #no quotes
   ls $source 2>&1

   # Transfers the file to Web Dispatcher server
   lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput \"$folder\"/* -O \"$folder\" ; bye" 2>&1
   
   #if zero no error code returned logic is off??
   #  0 implies true
	# 1 implies false
   if (($?))
   then
      echo "File transfer failed"
      mkdir $source/error
      mv "$source"/* "$source"/error
      echo "All files were moved to error folder"
   else
      echo "File transfer successful"
      # source destination using absolute paths
      
      mv "$folder"/* "$folder"/archive
      echo "All files were moved to archive folder"
   fi
fi
done | tee -a "$folder"/log

_mz · 01-13-2019, 09:15 PM

not be more something like this?

Code:

#!/bin/bash
IPWDSERVER=x.x.x.x
UserID=xxx
PASSWORD=xxxx
CURDATE=$(date '+%d%m%Y %T')
myFolders=(customermaster materialmaster opensalesorder sales_performance)

ParentDir=$HOME



# This loops the program until all directories listed above is executed
for folder in "${myFolders[@]}"
do
	source=$ParentDir/$folder
	#get number of files inside of the directories
   number_of_files=$(ls -A $source/* | wc -l)

   echo "----START----------------------------"
   echo "$CURDATE"
   echo "----Detailed transmission logs are in \"$source\"/log---"

#check count
if ((number_of_files == 0))
then
   echo "Directory \"$source\" is empty"
else
   echo "Directory "$source contains $number_of_files files"
   
   #no quotes
   ls $source 2>&1

   # Transfers the file to Web Dispatcher server
   lftp sftp://$UserID:$PASSWORD@$IPWDSERVER -e "mput \"$folder\"/* -O \"$folder\" ; bye" 2>&1
   
   #if zero no error code returned logic is off??
   #  0 implies true
	# 1 implies false
   if (($?))
   then
      echo "File transfer failed"
      mkdir $source/error
      mv "$source"/* "$source"/error
      echo "All files were moved to error folder"
   else
      echo "File transfer successful"
      # source destination using absolute paths
      
      mv "$folder"/* "$folder"/archive
      echo "All files were moved to archive folder"
   fi
fi
done | tee -a "$folder"/log

This is script is quite close to what I wanted. I modified a bit to suit the requirement. It works accept for logging. It does not write to each of these folders:-

/customermaster/log/log
/materialmaster/log/log
/opensalesorder/log/log
/sales_performance/log/log

Thanks for your assistance.

l0f4r0 · 01-14-2019, 04:32 AM

Quote:

Originally Posted by BW-userx

I am curious as to way you're not using absolute paths in your prototype script.

Code:

#!/bin/bash

#set -x

myFolders=(
Desktop Pictures Templates
holdingMusic Public torrents
bin Documents media scripts Videos
Downloads Music SDCard
)

for folder in "${myFolders[@]}"
do
	[[ -d $folder ]] && echo "$folder" || echo "no folder"
	
	[[ -d $(pwd)/$folder ]] && echo "$folder" || echo "no folder $(pwd)/$folder"
	
	[[ -d $HOME/$folder ]] && echo "$folder" || echo "no folder $HOME/$folder"
	
done

using this you can see it is only the last test that does not fail. the script looks in the directory that it is being ran from if no absolute path is given to it to point it elsewhere.

so all of these have to be in a subdir of the script where it is being ran.

No, I think you are probably mixing up the script directory (ie where the script is located) and the working directory (ie current directory listed with pwd).
Unless instructed otherwise, AFAIK, relative paths are determined from pwd, there is no link with the location of the script.

Let's consider the following example:

Code:

GlobalFolder
|---ParentFolder
    |---SubFolder1
    |---SubFolder2
    |---SubFolder3
    |---myScript

with the following myScript code:

Code:

#!/bin/bash
echo "My working directory is: $PWD"
myFolders=( SubFolder1 SubFolder2 SubFolder3 )
for folder in "${myFolders[@]}"
do
	[[ -d $folder ]] && echo "$folder" || echo "no folder"
	[[ -d $(pwd)/$folder ]] && echo "$folder" || echo "no folder $(pwd)/$folder"
	[[ -d $HOME/$folder ]] && echo "$folder" || echo "no folder $HOME/$folder"
done

Running from ParentFolder, myScript finds SubFolder1 SubFolder2 SubFolder3 thanks to instructions #1 (-d $folder) and #2 (-d $(pwd)/$folder), not with #3 (-d $HOME/$folder):

Code:

$ pwd
[...]/GlobalFolder/ParentFolder
$ ./myScript
My working directory is: [...]/GlobalFolder/ParentFolder
SubFolder1
SubFolder1
no folder /home/user/SubFolder1
SubFolder2
SubFolder2
no folder /home/user/SubFolder2
SubFolder3
SubFolder3
no folder /home/user/SubFolder3

However, running from GlobalFolder, none of the 3 SubFolderX is found anymore:

Code:

$ pwd
[...]/GlobalFolder
$ ParentFolder/myScript
My working directory is: [...]/GlobalFolder
no folder
no folder [...]/GlobalFolder/SubFolder1
no folder /home/user/SubFolder1
no folder
no folder [...]/GlobalFolder/SubFolder2
no folder /home/user/SubFolder2
no folder
no folder [...]/GlobalFolder/SubFolder3
no folder /home/user/SubFolder3

In other words, a script doesn't modify the working directory when it is launched (unless stated otherwise of course). So by default resources are searched from last active/current working directory when script is launched and no matter where this script is located.

Now regarding the use of relative or absolute paths, I think each method has its advantages or drawbacks.

absolute paths are more readable/explicit but you need to rewrite them in your script should you modify your file structure,
relative paths are quicker to write and always work as long as you are very careful with your working directory.

l0f4r0 · 01-14-2019, 04:53 AM

@_mz:

See my previous post to BW-userx.

Sorry, I made a mistake with the log part:

Code:

tee -a "$folder"/log

should be replaced with:

Code:

tee -a "$folder"/log/log

NB: I don't see the added-value of the following instruction:

Code:

ParentDir=$HOME

Indeed you already have variable HOME, why do you (and BW-userx) want to duplicate it?