LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 05-06-2009, 10:56 AM   #1
laurens
Member
 
Registered: Mar 2009
Posts: 65

Rep: Reputation: 15
Bash script to read .txt files on a remote Windows 2003 R2 server via FTP


As the title suggests, I'm trying to create a bash script to read .txt files on a remote Windows 2003 R2 server via ftp.

EDIT: it doesn't have to be FTP, it can be something else, or a variant of FTP

I have the #remote-server# , #username# , #password# and the folder:
"/Impactstored/Backup" where the txt-file is located.

I'm a real noob in the domain of bash scripting.

The .txt looks like
Code:
regel 1 = IP-adres
regel 2 = FTP-user
regel 3 = FTP-pwd
regel 4 = IP-adres 
regel 5 = FTP-user
etc
What i have is a .sh to read the lines and put them in arrays...
Code:
#!/bin/bash
#Declare array
declare -a ARRAY
#Open file for reading to array
exec 10<ftpservers.txt
let count=0

while read LINE <&10; do

    ARRAY[$count]=$LINE
    ((count++))
done

echo Number of elements: ${#ARRAY[@]}
# echo array's content
echo ${ARRAY[@]}
# close file
exec 10>&-
Thanks in advance !

Last edited by laurens; 05-06-2009 at 11:13 AM.
 
Old 05-06-2009, 11:13 AM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Umm, that's nice. But what is your actual question?
 
Old 05-06-2009, 11:33 AM   #3
laurens
Member
 
Registered: Mar 2009
Posts: 65

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by David the H. View Post
Umm, that's nice. But what is your actual question?
(?) I have a remote server containing a .txt from which I explained the layout above, and I have to read it in and log on to each server (24 in total) to FTP some files to a local TB NAS. But I would like to work in phases; First I have to connect to the remote server #212.xxx.xxx.xxx# to read in the .txt, I could echo it at first for example

The script is executed on a ubuntu machine.
 
Old 05-06-2009, 03:05 PM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Sorry, it's still not exactly clear to me about where your problem is. Are you having trouble getting it to connect, for example? Can it not read the file correctly or something (that's one possible problem I thought of: windows text files use a different line ending code)?

Personally, I know very little about initiating ftp connections, so I can't help you much there. But if you know what commands to use (wget perhaps?), I could possibly help you flesh out the script. Not that I'm an expert scripter, however. There are others much better than me here.

I didn't see anything terribly wrong with the code you posted before, except that I don't know what the "exec" lines are supposed to do.
 
Old 05-06-2009, 07:04 PM   #5
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,004
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Quote:
Originally Posted by laurens View Post
(?) I have a remote server containing a .txt from which I explained the layout above, and I have to read it in and log on to each server (24 in total) to FTP some files to a local TB NAS. But I would like to work in phases; First I have to connect to the remote server #212.xxx.xxx.xxx# to read in the .txt, I could echo it at first for example

The script is executed on a ubuntu machine.
With a word of caution upfront: ftp is inherently insecure,
and using the method below adds to that by sepcifying credentials
in a human readable form in clear-text files.

Create a file ~/.netrc for the user who runs your script
Make its content something like:
Code:
machine windowsbox
login your_user
password his_password
macdef init
get /tmp/myfile
Note: the empty lines at the bottom *ARE* required for the
macro to work.


Now, if you do "ftp windowsbox" from your script it will
go and grab the file /tmp/myfile automatically.




Cheers,
Tink
 
Old 05-07-2009, 08:05 AM   #6
laurens
Member
 
Registered: Mar 2009
Posts: 65

Original Poster
Rep: Reputation: 15
Tinkster, thanks for the input, is that a bash script? Cause it has to be.
I tried
Code:
#!/bin/bash
machine xxx.xxx.xx.be
login MYLOGIN
password PASSWORD
macdef init
get /folder/subfolder/ftpservers.txt
*echo ???*
(I tried to echo the output)
But it keeps asking for a login and pass, although I entered the right ones in the script.
Thx

Last edited by laurens; 05-07-2009 at 08:30 AM.
 
Old 05-07-2009, 04:18 PM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Tinkster said: "Create a file ~/.netrc for the user who runs your script".

That means that it's not a bash script, but text that goes in a separate file. When that .netrc file is in place, then you can run the ftp command and it will automatically look there to get the login information. After you verify that the command is logging in and doing what you want it to, you can include it in your script.

Last edited by David the H.; 05-07-2009 at 04:20 PM.
 
Old 05-07-2009, 04:23 PM   #8
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,004
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Quote:
Originally Posted by laurens View Post
Tinkster, thanks for the input, is that a bash script? Cause it has to be.
I tried
Code:
#!/bin/bash
machine xxx.xxx.xx.be
login MYLOGIN
password PASSWORD
macdef init
get /folder/subfolder/ftpservers.txt
*echo ???*
(I tried to echo the output)
But it keeps asking for a login and pass, although I entered the right ones in the script.
Thx
As David H. pointed out: it's not a script, it's a config
file that the ftp program will honour, and allow you for
"automatic" file transfer from a script ... man netrc



Cheers,
Tink
 
Old 05-08-2009, 07:22 AM   #9
laurens
Member
 
Registered: Mar 2009
Posts: 65

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Tinkster View Post
As David H. pointed out: it's not a script, it's a config
file that the ftp program will honour, and allow you for
"automatic" file transfer from a script ... man netrc



Cheers,
Tink
Thanks Tink... Now time for my next & last question.
I've downloaded a file which containers the IP addresses, usernames a paswords of lots of servers (24). As mentioned before its layout is like this:
Code:
line 1 = IP-adres
line 2 = FTP-user
line 3 = FTP-pwd
line 4 = IP-adres 
line 5 = FTP-user
... etc ...
line 33 = FTP-user
line 34 = FTP-pwd
line 35 = IP-adres
In the same folder there are some files (4 at the moment) named for example 212.xxx.xxx.xxx_folderlist. Where the first part is the IP address. These files contain folders to backup, e.g.
Code:
vabdocshop.xxx.de\files
intranetdocshop.xxx.de\files
fleet.xxx.be\docshop\files
Now the script should do the following: run over all IP addresses and check if there is a 212.xxx.xxx.xxx_folderlist file. If there is it should copy the folders via FTP to a folder, e.g. /backup
That's it!

Thanks a lot in advance

Last edited by laurens; 05-08-2009 at 07:55 AM.
 
Old 05-11-2009, 04:23 AM   #10
laurens
Member
 
Registered: Mar 2009
Posts: 65

Original Poster
Rep: Reputation: 15
You can ask me if something is not clear about my question?
So there is one file with all the IP's, U/N and P/W ; and a few others (in the same folder as ftpservers.txt) with folders that have to be copied. See my previous post. Thx
 
Old 05-11-2009, 05:34 AM   #11
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Quote:
Originally Posted by laurens View Post
You can ask me if something is not clear about my question?
So there is one file with all the IP's, U/N and P/W ; and a few others (in the same folder as ftpservers.txt) with folders that have to be copied. See my previous post. Thx
Well, I understand it now, after the subsequent posts, but it wasn't clear to me at first. Perhaps I just wasn't reading it right, or because I'm not that familiar with ftp, but I think you could've stated the problem a bit more directly.

"First I need to log into the Windows server with ftp and retrieve the file, but I'm not sure how to do it. How can I enter the user name and password from inside the script so that it can read or download it?"

This is the kind of direct question that makes it very clear to the readers what you really need help with. You should never assume that the reader understands what you want to know.

Last edited by David the H.; 05-11-2009 at 05:36 AM.
 
Old 05-11-2009, 07:19 AM   #12
laurens
Member
 
Registered: Mar 2009
Posts: 65

Original Poster
Rep: Reputation: 15
I know, my bad, sorry. It's kind of complicated to describe the whole situation as you've probably found out yet.
What about a bash-code solution for my 'problem'?

That's what I have yet.
Code:
ftp -n -v immserv1.XXX.xxx.be << EOT
ascii
user ftpixxxtstore Venezxxx
prompt
cd BACKUP
get ftpservers.txt
ls
EOT
cat ftpservers.txt
rm ftpservers.txt
This echoes me the list of servers (as found in ftpservers.txt)

Thx
 
Old 05-11-2009, 03:46 PM   #13
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
No problem. At least we're getting somewhere.

It sounds to me like you have it working, right? You should be able to simply drop the above code straight into your script. Are you sure you want to put the username and password in the script though, instead of using the .netrc method we talked about?

But assuming it works as it should, the code for creating the array in your first post should then also work.

Some questions though. Do you just want the lines as-is, or do you need to edit them in any way first? Are there any unwanted lines in the file, for example, or do you need to strip out certain fields from the strings? Is the format exactly the same as the example you gave above? If you need to edit the input in any way then we'll have to do some preprocessing first.

One thing I'm a bit concerned about is that dos/windows uses a different kind of line-feed character than unix. If your file is in dos format, then you may have to convert it to unix format before you can use it properly. There are several tools available that can do the conversion, like tofromdos and flip.

Don't forget to include a line at the end of the script to delete it when you're finished.
 
Old 05-12-2009, 04:00 AM   #14
laurens
Member
 
Registered: Mar 2009
Posts: 65

Original Poster
Rep: Reputation: 15
Well, I already have some of the functionality yet. I have transferred my file ftpservers.txt to my ubuntu server. I don't see why it should be converted whatsoever. It just a TXT file. Important is that each third line starting from the first line (1,4,7,10,13,...) should be processed.
Now.., What I have to have is a script that reads each line of ftpservers.txt and IF (with an IF clause) it finds a line with an IP address (which is every third line starting with line1) which is in the folder, it should go to that file and display it's content. The other lines can be ignored. That's it. For the moment.
Please check the attachment, and it will certainly get clear for you. Please provide me a bash code solution for this. Thx in advance

Last edited by laurens; 05-14-2009 at 08:21 AM.
 
Old 05-13-2009, 12:14 AM   #15
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
As I said, dos text files and unix text files use different codes for line breaks. If the file is in dos format, then many unix tools will not be able to determine where the lines end and will think the file has only one line. Your gui text editor may not show the difference because most of them can automatically detect and handle both formats, but it would probably break your script. So please check this first.

Ok, so you only need the IP addresses from the file. If the other lines will not be used in any way, then we can extract the IP lines and place them in a separate variable.
Code:
iplist=$(sed -n 1~3p ftpservers.txt)
This will take every third line and put resulting list in the iplist variable. It probably won't work right if the file is in dos format though. It might also get your script in trouble if the line numbering of the list ever varies. We may have to use something more fancy if the IP addresses directly.

Now do you really need to put the list in an array? Since you have a list of IP addresses, you could use that directly in a loop to process the list. Unless you have some future need for keeping the data an actual array.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
FTP files to remote server with a shell script Pezzoni Linux - Software 2 06-27-2007 08:01 AM
FTP problem "connection closed by remote host" with RHEL4 and Windows 2003 hooijack Linux - Enterprise 0 03-23-2006 10:29 PM
Read a line in a txt file with bash orgazmo Programming 5 05-03-2005 08:10 AM
Read a line in a txt file with bash orgazmo Linux - Newbie 3 05-03-2005 05:16 AM
Remote desktop SuSE 8.2 -> Windows Server 2003 thacreeper Linux - Networking 2 08-26-2004 03:15 PM


All times are GMT -5. The time now is 12:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration