Shell Script to FTP Zip File
 

I wrote the simple shell script below. I automate this using CronJobs nightly. It has been running successfully for weeks, but I decided to test out the backup, and apparently the Zip file after having been transferred, is now corrupt. When I check the Zip file on the original server it is fine. I unzip it and the contents are perfect. It appears that something is happening during the FTP that is causing the file issues. I will reiterate that the file on the other end, the one that has already been transferred using this script, appears to be fine, and seems to be have transferred without issue, except when you try to unzip the file it says that the file is corrupt.



#!/bin/sh

HOST='ftp.mywebsite.com'
USER='myusername'
PASSWD='mypassword'
FILE='myfile.zip'
LDIR='/backups/'
DIR='/website/'

ftp -inv $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
lcd $LDIR
cd $DIR
put $FILE
quit
END_SCRIPT
exit 0

i would use scp/sshfs with keys to avoid these limitations with ftp.

this is probably have to do with the ftp protocol being smarter than the user and using an ascii or binary mode transfer when the other is needed (you must be transferring from windows or mvs-mainframe <-> linux ?).

now that i remember the zip command in aix would automatically convert lf's to cr-lf's and there was no way to shut that off (maybe the problem is with the original zip).
i do this in my script to help mitigate that:do you have the same problem with bzip2 or gzip ?

Totally agree with schneidz here. SCP/SFTP would be FAR better, and more secure. Not to mention easier to script for. And it's not surprising it's 'corrupted'...you're not switching transfer mode from ASCII to binary. Try putting "binary" in before your "put" statement.

I hate to be a total newbie, but this was literally the first ever thing I've done like this, and I totally would love to try out your suggestion of setting binary mode, but I don't know how. Can anyone maybe post my code that I shared, but updated with the correct code to set binary?

Well, it's very simple...just add what was said:
Again, if you're brand new to this, you really should start off using the right tools for the job. FTP is old and insecure, and really should NOT be used at all, if you have alternatives. SSH is pretty much the 'standard' these days...one protocol supports both terminal sessions and file transfer. Doing a key-swap between boxes means you can literally have just ONE line in a shell script, and that line doesn't contain user ID's/passwords in clear text, nor does it send them across the network in clear text either.

Okay. The suggestion for adding binary worked. I very much thank everyone for helping me with this.

Now, can anyone explain how I would do this same thing, except in the way everyone seems to be suggesting? That is, to use SSH? Could possibly someone supply a version of my code that utilizes SSH instead?

This has been covered on here MANY times...check the "Search" function for similar questions. You would generate an SSH key, with the ssh-keygen command, and manually copy that key over to the remote system, and put it into the $HOME/.ssh/authorized_keys2 file. After that, you just type in "ssh <user>@<remote system address or name", and you get right in..no password needed.

To transfer a file, you would use the scp command, "scp /some/file <user>@<remote system>:/some/path". Done. How did you manage to handed this project if you've got no Linux experience? No offense intended, but I'd not give someone a task they didn't have the knowledge to accomplish.

I completely understand your frustration TB0ne. This was not assigned to me to complete. I am actually a front end web developer... I'm a UI/UX guy, who is trying to automate a backup... this kind of stuff is crazy magic for front end people like me. I am by no means a UNIX person at all. I just thought maybe there was someone on this forum that could help me in my plight.

Again, I say thank you to everyone for the kind help. I know that it must have seemed very simple, but I appreciate it.

As for the explanation of the SSH option, I may have to pursue some additional research in order to accomplish this task. TB0ne's description of what to do seems over the head of a simple front end developer.

I will have to contact RackSpace about how to generate an SSH Key. I don't really have access to the server except through FTP and via their online Cloud Sites tools. Thanks again.

Not frustrated at all, I was just wondering how you got into that situation. Sorry If I seemed frustrated.

If you have shell access, Just log in and run ssh-keygen from the command line. Give a quick search here for full details....I'm on my phone now, so looking it up is a bit tougher. :)

i have unix admins at my job (who can log into system accounts) that ask me all the time to do basic things like gunzip a file:
http://www.linuxquestions.org/questi...ow-4175477022/

i think as long as you can wow hr with a few buzz words they would allow you to run a nuclear facility.

Agreed...that's why <sarcasm>I give certifications SO MUCH weight in the hiring process</sarcasm> :)

I may be in somewhat of a unique situation here though. I am dealing with two very different vendors and have limited access to either of the machines. The Shell script is set up on a RackSpace Cloud Sites server and via their Admin screens I am able to schedule a Shell Script to run. I suppose that I will just need to teach myself how to create aSheel Script to generate the SSH key, which is outside of the main issue here.

The destination of the file transfer is Box.com, which is a cloud storage vendor. The only access I have to this machine is via FTP. I have no access to their actual server at all.

Given this information, would you expect that a person with the appropriate level of knowledge, would be able to accomplish this task without needing to contact either or both vendors to have them actually do the work?

Thanks again everyone. The help is much appreciated.