How to script downloading multiple files from remote server
In remote server, there are always a pair of files like:-
in folder /Pair1 Code:
IN_S-WW001_20160418100715.out Code:
ar.MON.20160226.output.ZZ.out This is just an example of files. There will be a lot of them generated in one day. File name in a pair is always same except for timestamps. From other server, I need to SFTP to the remote server and check one pair by one pair. Logic: The script should check if .out file is there, then grab/download the .txt file (from the pair). Once the .txt file is transferred/downloaded successfully, script will go back to remote server and delete the .out file. This is my test script:- Code:
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles|grep .out; bye" >> $LOG |
first, I would go for the list
Code:
# something like this finally execute that. To process that local_list file I suggest you to use awk/perl/python/whatever, what is your preferred language? |
Try not to use sftp with username & passwd in code - fiddly and asking to be hacked since creds are in the clear.
Instead, use ssh-keys with scp to copy files and ssh to get any other info. If there are other files in the dirs apart from the pairs, you could use ssh to list .out files, then for each filename look for the matching .txt and scp it down . In fact (faster/easier), just list the entire dir down (filenames) into a file and process list locally, then just fetch any matching pair's .txt if it's in the list. I'd personally use Perl (which also has ssh modules), but bash could do it as well, prob with help from awk or sed etc. HTH |
Quote:
|
Quote:
|
For a start, list the .out filenames into one filelist .txt filenames into another file, then read through the .out filelist and use eg grep to search for corresponding .txt file in the other filelist.
You may want to read/bookmark these if yopu ar e going to go with bash: http://tldp.org/LDP/Bash-Beginners-G...tml/index.html http://www.tldp.org/LDP/abs/html/ If you show us your code as you go we can help you. Use code tags https://www.linuxquestions.org/quest...do=bbcode#code to post it. One of the most useful cmds is Code:
set -xv |
Create a file with a list of urls.
$ echo site.com/download/bubba1.jpg >> file.list $ echo site.com/download/bubba2.jpg >> file.list $ echo site.com/download/bubba3.jpg >> file.list And. $ wget -c -i file.list Stuff I used to do in the days of dialup. Build up my list and hit up the broadband at the truck stop / rest stop / library until the battery on my laptop died. |
All times are GMT -5. The time now is 05:15 PM. |