How to script downloading multiple files from remote server

_mz · 08-24-2016, 02:18 AM

In remote server, there are always a pair of files like:-

in folder /Pair1

Code:

IN_S-WW001_20160418100715.out
IN_S-WW001_20160418100715.txt

in folder /Pair2

Code:

ar.MON.20160226.output.ZZ.out
ar.MON.20160226.output.ZZ.txt

This is just an example of files. There will be a lot of them generated in one day.

File name in a pair is always same except for timestamps. From other server, I need to SFTP to the remote server and check one pair by one pair.

Logic: The script should check if .out file is there, then grab/download the .txt file (from the pair). Once the .txt file is transferred/downloaded successfully, script will go back to remote server and delete the .out file.

This is my test script:-

Code:

lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles|grep .out; bye" >> $LOG
if [ $? -eq 0 ]
then
echo "Files exist, getting *.txt" >> $LOG
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "mget /home/adm/testfiles/*.txt -O /home/adm/test/ ; bye" >> $LOG
else
echo ".out files do not exist" >> $LOG
fi

The problem is that, it will list out all files with extension .out instead of checking pair by pair. I do not know how to classify a pair. I am not good in scripting, so, I have to ask for some help. It is not necessary to be in lftp. It is just that is only I know.

pan64 · 08-24-2016, 02:35 AM

first, I would go for the list

Code:

# something like this
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles" > local_list

next I would try to analyse that list and find out what to download and what should be deleted.
finally execute that.
To process that local_list file I suggest you to use awk/perl/python/whatever, what is your preferred language?

chrism01 · 08-24-2016, 02:38 AM

Try not to use sftp with username & passwd in code - fiddly and asking to be hacked since creds are in the clear.
Instead, use ssh-keys with scp to copy files and ssh to get any other info.

If there are other files in the dirs apart from the pairs, you could use ssh to list .out files, then for each filename look for the matching .txt and scp it down .
In fact (faster/easier), just list the entire dir down (filenames) into a file and process list locally, then just fetch any matching pair's .txt if it's in the list.

I'd personally use Perl (which also has ssh modules), but bash could do it as well, prob with help from awk or sed etc.

HTH

_mz · 08-24-2016, 03:44 AM

Quote:

Originally Posted by pan64

first, I would go for the list

Code:

# something like this
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles" > local_list

next I would try to analyse that list and find out what to download and what should be deleted.
finally execute that.
To process that local_list file I suggest you to use awk/perl/python/whatever, what is your preferred language?

I only know bash but basic only. Perl/python is beyond my knowledge. Any language would be fine as long as I can achieve the objective.

_mz · 08-24-2016, 03:47 AM

Quote:

Originally Posted by chrism01

Try not to use sftp with username & passwd in code - fiddly and asking to be hacked since creds are in the clear.
Instead, use ssh-keys with scp to copy files and ssh to get any other info.

If there are other files in the dirs apart from the pairs, you could use ssh to list .out files, then for each filename look for the matching .txt and scp it down .
In fact (faster/easier), just list the entire dir down (filenames) into a file and process list locally, then just fetch any matching pair's .txt if it's in the list.

I'd personally use Perl (which also has ssh modules), but bash could do it as well, prob with help from awk or sed etc.

HTH

How to do scripting for the matching? How I do I know that file1.out is pair/match with file1.txt, file2.out is pair of file2.txt. This is what I am wondering now. Sorry, I know basic bash scripting only. Perl/Python is beyond my knowledge.

chrism01 · 08-24-2016, 04:02 AM

For a start, list the .out filenames into one filelist .txt filenames into another file, then read through the .out filelist and use eg grep to search for corresponding .txt file in the other filelist.

You may want to read/bookmark these if yopu ar e going to go with bash:
http://tldp.org/LDP/Bash-Beginners-G...tml/index.html
http://www.tldp.org/LDP/abs/html/

If you show us your code as you go we can help you. Use code tags https://www.linuxquestions.org/quest...do=bbcode#code to post it.

One of the most useful cmds is

Code:

set -xv

as the 2nd line. It shows you what the parser is doing.

Shadow_7 · 08-26-2016, 01:08 PM

Create a file with a list of urls.

$ echo site.com/download/bubba1.jpg >> file.list
$ echo site.com/download/bubba2.jpg >> file.list
$ echo site.com/download/bubba3.jpg >> file.list

And.

$ wget -c -i file.list

Stuff I used to do in the days of dialup. Build up my list and hit up the broadband at the truck stop / rest stop / library until the battery on my laptop died.