LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-18-2013, 03:31 PM   #1
mpfeiffer311
LQ Newbie
 
Registered: Oct 2013
Posts: 5

Rep: Reputation: Disabled
Shell Script to FTP Zip File


I wrote the simple shell script below. I automate this using CronJobs nightly. It has been running successfully for weeks, but I decided to test out the backup, and apparently the Zip file after having been transferred, is now corrupt. When I check the Zip file on the original server it is fine. I unzip it and the contents are perfect. It appears that something is happening during the FTP that is causing the file issues. I will reiterate that the file on the other end, the one that has already been transferred using this script, appears to be fine, and seems to be have transferred without issue, except when you try to unzip the file it says that the file is corrupt.



#!/bin/sh

HOST='ftp.mywebsite.com'
USER='myusername'
PASSWD='mypassword'
FILE='myfile.zip'
LDIR='/backups/'
DIR='/website/'

ftp -inv $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
lcd $LDIR
cd $DIR
put $FILE
quit
END_SCRIPT
exit 0
 
Old 10-18-2013, 03:37 PM   #2
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,141

Rep: Reputation: 638Reputation: 638Reputation: 638Reputation: 638Reputation: 638Reputation: 638
i would use scp/sshfs with keys to avoid these limitations with ftp.

this is probably have to do with the ftp protocol being smarter than the user and using an ascii or binary mode transfer when the other is needed (you must be transferring from windows or mvs-mainframe <-> linux ?).

now that i remember the zip command in aix would automatically convert lf's to cr-lf's and there was no way to shut that off (maybe the problem is with the original zip).
i do this in my script to help mitigate that:
Code:
sed 's/$'"/`echo -e \\\r`/" $file > dos.csv
do you have the same problem with bzip2 or gzip ?

Last edited by schneidz; 10-18-2013 at 03:51 PM.
 
Old 10-18-2013, 03:38 PM   #3
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,977

Rep: Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672
Quote:
Originally Posted by mpfeiffer311 View Post
I wrote the simple shell script below. I automate this using CronJobs nightly. It has been running successfully for weeks, but I decided to test out the backup, and apparently the Zip file after having been transferred, is now corrupt. When I check the Zip file on the original server it is fine. I unzip it and the contents are perfect. It appears that something is happening during the FTP that is causing the file issues. I will reiterate that the file on the other end, the one that has already been transferred using this script, appears to be fine, and seems to be have transferred without issue, except when you try to unzip the file it says that the file is corrupt.
Code:
#!/bin/sh

HOST='ftp.mywebsite.com'
USER='myusername'
PASSWD='mypassword'
FILE='myfile.zip'
LDIR='/backups/'
DIR='/website/'

ftp -inv $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
lcd $LDIR
cd $DIR
put $FILE
quit
END_SCRIPT
exit 0
Totally agree with schneidz here. SCP/SFTP would be FAR better, and more secure. Not to mention easier to script for. And it's not surprising it's 'corrupted'...you're not switching transfer mode from ASCII to binary. Try putting "binary" in before your "put" statement.
 
1 members found this post helpful.
Old 10-18-2013, 03:44 PM   #4
mpfeiffer311
LQ Newbie
 
Registered: Oct 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
I hate to be a total newbie, but this was literally the first ever thing I've done like this, and I totally would love to try out your suggestion of setting binary mode, but I don't know how. Can anyone maybe post my code that I shared, but updated with the correct code to set binary?
 
Old 10-18-2013, 03:52 PM   #5
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,977

Rep: Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672
Quote:
Originally Posted by mpfeiffer311 View Post
I hate to be a total newbie, but this was literally the first ever thing I've done like this, and I totally would love to try out your suggestion of setting binary mode, but I don't know how. Can anyone maybe post my code that I shared, but updated with the correct code to set binary?
Well, it's very simple...just add what was said:
Code:
#!/bin/sh

HOST='ftp.mywebsite.com'
USER='myusername'
PASSWD='mypassword'
FILE='myfile.zip'
LDIR='/backups/'
DIR='/website/'

ftp -inv $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
lcd $LDIR
cd $DIR
binary
put $FILE
quit
END_SCRIPT
exit 0
Again, if you're brand new to this, you really should start off using the right tools for the job. FTP is old and insecure, and really should NOT be used at all, if you have alternatives. SSH is pretty much the 'standard' these days...one protocol supports both terminal sessions and file transfer. Doing a key-swap between boxes means you can literally have just ONE line in a shell script, and that line doesn't contain user ID's/passwords in clear text, nor does it send them across the network in clear text either.

Last edited by TB0ne; 10-18-2013 at 03:55 PM.
 
Old 10-18-2013, 04:12 PM   #6
mpfeiffer311
LQ Newbie
 
Registered: Oct 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
Okay. The suggestion for adding binary worked. I very much thank everyone for helping me with this.

Now, can anyone explain how I would do this same thing, except in the way everyone seems to be suggesting? That is, to use SSH? Could possibly someone supply a version of my code that utilizes SSH instead?
 
Old 10-18-2013, 04:19 PM   #7
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,977

Rep: Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672
Quote:
Originally Posted by mpfeiffer311 View Post
Okay. The suggestion for adding binary worked. I very much thank everyone for helping me with this.

Now, can anyone explain how I would do this same thing, except in the way everyone seems to be suggesting? That is, to use SSH? Could possibly someone supply a version of my code that utilizes SSH instead?
This has been covered on here MANY times...check the "Search" function for similar questions. You would generate an SSH key, with the ssh-keygen command, and manually copy that key over to the remote system, and put it into the $HOME/.ssh/authorized_keys2 file. After that, you just type in "ssh <user>@<remote system address or name", and you get right in..no password needed.

To transfer a file, you would use the scp command, "scp /some/file <user>@<remote system>:/some/path". Done. How did you manage to handed this project if you've got no Linux experience? No offense intended, but I'd not give someone a task they didn't have the knowledge to accomplish.
 
Old 10-18-2013, 04:29 PM   #8
mpfeiffer311
LQ Newbie
 
Registered: Oct 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
I completely understand your frustration TB0ne. This was not assigned to me to complete. I am actually a front end web developer... I'm a UI/UX guy, who is trying to automate a backup... this kind of stuff is crazy magic for front end people like me. I am by no means a UNIX person at all. I just thought maybe there was someone on this forum that could help me in my plight.

Again, I say thank you to everyone for the kind help. I know that it must have seemed very simple, but I appreciate it.

As for the explanation of the SSH option, I may have to pursue some additional research in order to accomplish this task. TB0ne's description of what to do seems over the head of a simple front end developer.

I will have to contact RackSpace about how to generate an SSH Key. I don't really have access to the server except through FTP and via their online Cloud Sites tools. Thanks again.
 
Old 10-19-2013, 07:25 AM   #9
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,977

Rep: Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672
Not frustrated at all, I was just wondering how you got into that situation. Sorry If I seemed frustrated.

If you have shell access, Just log in and run ssh-keygen from the command line. Give a quick search here for full details....I'm on my phone now, so looking it up is a bit tougher.
 
Old 10-20-2013, 09:32 AM   #10
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,141

Rep: Reputation: 638Reputation: 638Reputation: 638Reputation: 638Reputation: 638Reputation: 638
i have unix admins at my job (who can log into system accounts) that ask me all the time to do basic things like gunzip a file:
http://www.linuxquestions.org/questi...ow-4175477022/

i think as long as you can wow hr with a few buzz words they would allow you to run a nuclear facility.

Last edited by schneidz; 10-20-2013 at 09:38 AM.
 
Old 10-20-2013, 11:22 AM   #11
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,977

Rep: Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672Reputation: 2672
Quote:
Originally Posted by schneidz View Post
i have unix admins at my job (who can log into system accounts) that ask me all the time to do basic things like gunzip a file:
http://www.linuxquestions.org/questi...ow-4175477022/

i think as long as you can wow hr with a few buzz words they would allow you to run a nuclear facility.
Agreed...that's why <sarcasm>I give certifications SO MUCH weight in the hiring process</sarcasm>
 
Old 10-21-2013, 09:14 AM   #12
mpfeiffer311
LQ Newbie
 
Registered: Oct 2013
Posts: 5

Original Poster
Rep: Reputation: Disabled
I may be in somewhat of a unique situation here though. I am dealing with two very different vendors and have limited access to either of the machines. The Shell script is set up on a RackSpace Cloud Sites server and via their Admin screens I am able to schedule a Shell Script to run. I suppose that I will just need to teach myself how to create aSheel Script to generate the SSH key, which is outside of the main issue here.

The destination of the file transfer is Box.com, which is a cloud storage vendor. The only access I have to this machine is via FTP. I have no access to their actual server at all.

Given this information, would you expect that a person with the appropriate level of knowledge, would be able to accomplish this task without needing to contact either or both vendors to have them actually do the work?

Thanks again everyone. The help is much appreciated.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How can i download a file from remote server using ftp using shell script sanjay87 Linux - Server 10 04-16-2012 07:09 AM
Shell Script for Identifying the file and zip all files, move the files to Target Dir gvenkat Linux - Newbie 3 05-07-2011 11:53 AM
Bash shell script and automating FTP Backups (File Verification) alternaloser Linux - Server 1 10-12-2009 01:18 AM
shell script to transfer file from ftp bbgtilak Linux - General 1 12-27-2006 08:35 AM
How to write a shell script to download a file via FTP? guarriman Linux - General 4 12-21-2004 10:31 AM


All times are GMT -5. The time now is 02:57 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration