LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-24-2016, 02:18 AM   #1
_mz
Member
 
Registered: Jul 2013
Posts: 33

Rep: Reputation: Disabled
How to script downloading multiple files from remote server


In remote server, there are always a pair of files like:-

in folder /Pair1
Code:
IN_S-WW001_20160418100715.out
IN_S-WW001_20160418100715.txt
in folder /Pair2
Code:
ar.MON.20160226.output.ZZ.out
ar.MON.20160226.output.ZZ.txt

This is just an example of files. There will be a lot of them generated in one day.

File name in a pair is always same except for timestamps. From other server, I need to SFTP to the remote server and check one pair by one pair.

Logic: The script should check if .out file is there, then grab/download the .txt file (from the pair). Once the .txt file is transferred/downloaded successfully, script will go back to remote server and delete the .out file.

This is my test script:-
Code:
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles|grep .out; bye" >> $LOG
if [ $? -eq 0 ]
then
echo "Files exist, getting *.txt" >> $LOG
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "mget /home/adm/testfiles/*.txt -O /home/adm/test/ ; bye" >> $LOG
else
echo ".out files do not exist" >> $LOG
fi
The problem is that, it will list out all files with extension .out instead of checking pair by pair. I do not know how to classify a pair. I am not good in scripting, so, I have to ask for some help. It is not necessary to be in lftp. It is just that is only I know.
 
Old 08-24-2016, 02:35 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,677

Rep: Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855
first, I would go for the list
Code:
# something like this
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles" > local_list
next I would try to analyse that list and find out what to download and what should be deleted.
finally execute that.
To process that local_list file I suggest you to use awk/perl/python/whatever, what is your preferred language?
 
Old 08-24-2016, 02:38 AM   #3
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,406

Rep: Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396
Try not to use sftp with username & passwd in code - fiddly and asking to be hacked since creds are in the clear.
Instead, use ssh-keys with scp to copy files and ssh to get any other info.

If there are other files in the dirs apart from the pairs, you could use ssh to list .out files, then for each filename look for the matching .txt and scp it down .
In fact (faster/easier), just list the entire dir down (filenames) into a file and process list locally, then just fetch any matching pair's .txt if it's in the list.

I'd personally use Perl (which also has ssh modules), but bash could do it as well, prob with help from awk or sed etc.

HTH
 
Old 08-24-2016, 03:44 AM   #4
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
first, I would go for the list
Code:
# something like this
lftp sftp://$UserID:$PASSWORD@10.x.x.x -e "ls /home/adm/testfiles" > local_list
next I would try to analyse that list and find out what to download and what should be deleted.
finally execute that.
To process that local_list file I suggest you to use awk/perl/python/whatever, what is your preferred language?
I only know bash but basic only. Perl/python is beyond my knowledge. Any language would be fine as long as I can achieve the objective.
 
Old 08-24-2016, 03:47 AM   #5
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by chrism01 View Post
Try not to use sftp with username & passwd in code - fiddly and asking to be hacked since creds are in the clear.
Instead, use ssh-keys with scp to copy files and ssh to get any other info.

If there are other files in the dirs apart from the pairs, you could use ssh to list .out files, then for each filename look for the matching .txt and scp it down .
In fact (faster/easier), just list the entire dir down (filenames) into a file and process list locally, then just fetch any matching pair's .txt if it's in the list.

I'd personally use Perl (which also has ssh modules), but bash could do it as well, prob with help from awk or sed etc.

HTH
How to do scripting for the matching? How I do I know that file1.out is pair/match with file1.txt, file2.out is pair of file2.txt. This is what I am wondering now. Sorry, I know basic bash scripting only. Perl/Python is beyond my knowledge.
 
Old 08-24-2016, 04:02 AM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,406

Rep: Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396Reputation: 2396
For a start, list the .out filenames into one filelist .txt filenames into another file, then read through the .out filelist and use eg grep to search for corresponding .txt file in the other filelist.

You may want to read/bookmark these if yopu ar e going to go with bash:
http://tldp.org/LDP/Bash-Beginners-G...tml/index.html
http://www.tldp.org/LDP/abs/html/

If you show us your code as you go we can help you. Use code tags https://www.linuxquestions.org/quest...do=bbcode#code to post it.

One of the most useful cmds is
Code:
set -xv
as the 2nd line. It shows you what the parser is doing.
 
Old 08-26-2016, 01:08 PM   #7
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 3,257
Blog Entries: 1

Rep: Reputation: 681Reputation: 681Reputation: 681Reputation: 681Reputation: 681Reputation: 681
Create a file with a list of urls.

$ echo site.com/download/bubba1.jpg >> file.list
$ echo site.com/download/bubba2.jpg >> file.list
$ echo site.com/download/bubba3.jpg >> file.list

And.

$ wget -c -i file.list

Stuff I used to do in the days of dialup. Build up my list and hit up the broadband at the truck stop / rest stop / library until the battery on my laptop died.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
ssh script to transmit files to remote server ramuind Linux - Server 1 01-04-2015 02:21 AM
Bash script to read .txt files on a remote Windows 2003 R2 server via FTP laurens Linux - Newbie 25 05-15-2009 06:01 AM
Downloading a list of files from a remote server using a list ralcocer Programming 2 02-11-2009 01:02 PM
Remote Downloading of Files debloxie Linux - Newbie 1 02-10-2009 05:40 AM
FTP files to remote server with a shell script Pezzoni Linux - Software 2 06-27-2007 07:01 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:44 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration