shell script required to get the mail after files are uploaded to ftp
 

Hi,

I have shell script which moves the files from linux machine to FTP Windows machine . But some time because of network problem the file are not uploading .Her i need to get the mail weather the files are uploaded or not.And the file size will be less than 1mb.

Here i am trying to get the mail weather the files were uploaded successfully or not .For this i am trying in two different ways

1.) Before uploading i would like to ping the ftp server if it is pinging then the files will be uploaded and the get the mail successfully moved.Other wise if it is not pinging i will get the mail not move .

2.) After moving the file to ftp here i will again get those ftp uploaded folder back to the server and if i find the folder in my server then i want to get the success mail other wise not success mail.

Pleas suggest me which is the best one and pleas help me in my code how to use ping command in the script.Here is my script.



#!/bin/bash
HOST='192.168.7.156'
USER='rishna'
PASSWD='tration'
#FILE='*'
#DT=$(date +%a_%h_%d_%Y)
if [ ]
then
DT=$(date +%Y-%m-%d)
ftp -n $HOST > .ftpout.txt 2>.ftperr.txt <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
#lcd /home/oracle/test
lcd /u04/dbdump/
cd sample
mkdir $DT
cd $DT
prompt
mput *
quit
END_SCRIPT
echo "FTP worked " | mail -s "errors are " sample@chi.com
else
echo "FTP dosent worked" | mail -s "errors are " sample@chi.com
fi

Testing for FTP server
 

If you want to use ping (read below to see why you might not), you can instruct ping how many attempts to make and how long to wait between them. For a local machine I'd suggest a few attempts and a 1s timeout is probably sufficient. Within a script, I usually throw away the output and just check the return code from ping (0 meaning success)--e.g.:

ping -c 3 -W 1 $HOST > /dev/null
if [ $? != 0 ]; then
echo "Ping of $HOST failed; aborting now"
exit 1
fi


Ping is a fundamental test to know if the host is on the network; unfortunately it can't tell whether an FTP server or whether the file(s) transferred successfully. You can use the 'telnet' command to try to connect to the FTP server. I wouldn't recommend using 'telnet' to transfer to the ftp server (it could, but it would be painful for you), it should tell you whether the server is listening. You can use the 'grep' command to check for output that suggests the test was successful, and again just check the return code of the last command run (grep) for decision making e.g:

## Test if the FTP server is available
FTP_PORT=21
echo quit | telnet $HOST $FTP_PORT 2>&1 | grep 'Connected' > /dev/null
if [ $? != 0 ]; then
echo "FTP server on $HOST did not respond; aborting now"
exit 2
fi


However it's certainly possible to pass both of the above tests and still have the transfer fail--e.g. due to a network problem, full filesystem on the server, etc. If you really care, the only deterministic way to monitor success is at the application level--i.e. either watch the output of the 'ftp' command and verify that it succeeds or try to download the files after an upload attempt (which will verify that they made it). The former approach might be possible by parsing your .ftpout.txt file (checking for a "success" response after the mput). The latter could involve downloading to a temporary directory, running the 'diff' command, and then deleting the files in the temporary local directory.

Basically you want to know if the ftp was successful or not, there are many reasons an ftp will fail other than network connectivity, so forget about ping. I've used many methods to ensure an ftp file actual transfers. If your windows server supports it, you might want to consider using sftp. Then look at using the -b switch for batch file. If the batch file fails it will return a failure code. Another option is kermit. Kermit will work with ftp and actually does return failure codes for each particular step. You would only need to install it on the Linux side as only ftp is required on the Windows side. A third option is to code the ftp differently. Generally what I do is to create a work directory and initially send the file there, then I retrieve the file I just sent with a different name. Do a Linux side diff on the original file and the file you pulled back, if there are no differences the transfer was successful and I reconnect and move the file (rename in ftp) to the directory I want it in. One of the problems here might be an ascii transfer mode. You don't explicitly set the mode, so it may switch to ascii with a windows server. That might convert the files from Unix (LF) to Windows (CR/LF) line terminations. A fourth option is to search the ftperr.txt file you are creating in your script. This only works if you know what you are looking for. For instance grep for the word error, if you find it the ftp failed, otherwise it was successful. Note the mkdir command may report an error if the directory already exists. If you use ftp, you may want to look at using a .netrc file instead of embedding the usernames and passwords. I don't like .netrc files, but they are better than embedded passwords. You may also want to send the files one at a time instead of using the mput command.

Hi,

Thanks for your replays.But when i execute the script the file was moving from linux to windows FTP server but in my ftperr.txt and ftpout.txt is like this .

ftperr.txt

KERBEROS_V4 rejected as an authentication type
netout: Connection reset by peer
netout: Connection reset by peer
netout: Connection reset by peer

and my ftpour.txt


'AUTH GSSAPI': command not understood
'AUTH KERBEROS_V4': command not understood
?Invalid command
Local directory now /u04/dbdump
Interactive mode off.


can any one please help to explain what is ment by that outputs because i am new to shell script.As i was an Oracle DBA and i am not having any touch with this script.Her is my shell script file which i runs every day.


#!/bin/bash

HOST='192.168.2.200'
USER='virinchi/krishna'
PASSWD='panetration'
#FILE='*'
#DT=$(date +%a_%h_%d_%Y)
DT=$(date +%Y-%m-%d)
ftp -n $HOST > .ftpout.txt 2>.ftperr.txt <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
#lcd /home/oracle/test
lcd /u04/dbdump/
cd sample
mkdir $DT
cd $DT
prompt
mput *
quit
END_SCRIPT

You should run the ftp manually/interactively to see what the "normal" output is. You can probably just cut & paste the lines from the shell script, but skipping the redirection of STDOUT and STDERR may shed some light on when the messages are coming back to you--and if they can be safely ignored.

My *guess* would be that you might have a firewall (host-based, within the network, or some form of Port Address Translation/NAT) that is preventing the blocking the ftp data connection 'data'. FTP uses a control connection for commands, and then establishes a 2nd network connection for data transfer (including 'dir' or 'ls' output, I think); typically the server initiates the data connection to the client, but it's usually to a semi-random port number (negotiated via the control channel) and simplistic firewall rules & NAT don't like new connections to unexpected ports and block them. Typically, this is addressed by using ftp in a "passive" mode wherein the client initiates the data connection also, so that the firewall/NAT (which "trusts" the client) will permit it. If this is indeed the case, adding a '-p' flag to your ftp command (or using the 'pftp' command instead) may do the trick. The 'netout: Connection reset by peer' lines would occur were "passive" mode necessary and not used (i.e. the server would try to connect to the client for the ftp data connection and fail).




The GSSAPI and KERBEROS_4 messages are related to automatic attempts to authenticate you using a Kerberos ticket (which you may or may not have on your Linux client). These are likely not fatal and just a "normal" attempt by the client to be helpful and try to use Kerberos credentials to avoid you having to type a password.

The '?Invalid command' is most likely a result of the '#lcd /home/oracle/test' line. The "#" would have prevented the line as being interpreted as a shell command (it would have been skipped), but since you have it between '<<END_SCRIPT' and 'END_SCRIPT', then it is sent (just as all the other lines between them) to ftp as a command. You should move/remove this line to avoid the error--though it also should be non-fatal.



So.... all that said, try adding the '-p' flag to your ftp command and removing and moving any lines starting with '#' outside of the series commands you're sending to ftp.

As tidww01 mentioned, ftp is functional, but rather insecure (the passwords are sent as text strings, unencrypted over the network). Hiding the passwords in a config file could make it a little less visible from within the client system--if others have permission to see your script they would be able to see your password).


'sftp' (which uses SSH) is far more secure, and the 'scp' command might simplify your client side a lot (e.g. 'scp -r $MY_FILES $HOST:$REMOTE_DIR' and just check the return code!).

It's been a while since I looked into SSH servers (which is what sftp and scp talk to ) for Windows. CygWin had a [free] suite of tools that included OpenSSH & worked well, though might require a little tweaking to get it running how you like; there may be more "shrink-wrapped' solutions if you look around more. I've not used it, but the link below claims to have a Windows version of OpenSSH (the most prominent SSH server) for free if you want to try it.

http://sshwindows.sourceforge.net/

For sftp & scp, the transfers are secure and you can pretty easily setup "keys" to permit authentication without needing to embed a password in a script.


Technically, your Linux box could mount a share on the Windows server via SAMBA(SMB) if you wanted. That would allow you to just 'cp' the files and use 'diff' or 'ls' to see if they made it. Setting up SMB shares obviously opens up new security concerns also, but if your server is already sharing, the incremental change may be inconsequential.

Hi,

As your replay i replace '-p' in place of '-n' and i get the error in my ftperr.txt file like this

ftp: p: unknown option
Usage: ftp [-v] [-d] [-i] [-n] [-g] [-k realm] [-f] [-x] [-u] [-t] [host]


can u please help me

Hi JPlum,

Can you please help me how to code the scp option in to my script.

Before using the scp command in my file i need to configure SSH login without password(http://linuxproblem.org/art_9.html)

Please help me it is very important for me right now.

Thanks
Poorna

For FTP: I'm suggesting you trigger the passive ftp mode; you are using a different client program that I have (under Ubuntu Linux). If you don't have a command line option for passive transfers, check the man page to see how to invoke passive mode. Some clients implement this as a command executed after the client connects (for example adding 'pass' or 'passive' or 'set passive on' to your list of commands),


ftp -n $HOST > .ftpout.txt 2>.ftperr.txt <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
#lcd /home/oracle/test
lcd /u04/dbdump/
cd sample
mkdir $DT
cd $DT
prompt
pass
mput *
quit
END_SCRIPT



If the above doesn't work, consider getting a different ftp client. There are likely several available; ncftp is one I like and is likely available through yum/yast/apt-get (depending on your distribution). ncftp is handy for interactive use as it'll automatically handle anonymous logins and can save "bookmarks" for sites you use often--neither of which you need here, but it also supports passive mode by running the command 'set passive on' within an ftp session.
----------------------

For scp, the big hurdle is installing an ssh server on the Windows box, creating an account (username & password unless it leverages the Windows accounts & passwords), and making sure that SSHD is always running (installed as a Windows service most likely). The link below has some information and options on doing so; the link I included before seems to have an outdated version of SSHD. With that in place, you would follow the procedure in the last link you sent (http://linuxproblem.org/art_9.html) to setup the keys to permit automated login.mailutils

http://www.windowsnetworking.com/art...rver-2008.html


With all that done, your script gets to bit simpler:


#!/bin/bash
### Parameters ###
HOST='192.168.7.156'
USER='rishna'
SOURCE='/u04/dbdump/'
DEST='sample'
EMAIL='sample@chi.com'
### Create a link to the source directory
DT=$(date +%Y-%m-%d)
cd /tmp
ln -s $SOURCE $DT > $0.$DT.out 2>&1 || (cat $0.$DT.out | mail -s "scp FAILED for $DT" $EMAIL)
### Perform the transfer & mail the result
scp -qr $DT $USER@$HOST:$DEST/ > $0.$DT.out 2>&1
if [ $? != 0 ]; then
cat /tmp/$0.$DT.out | mail -s "scp FAILED for $DT" $EMAIL
else
cat /tmp/$0.$DT.out | mail -s "scp SUCCEEDED for $DT" $EMAIL
fi
### Clean up
rm $DT $0.$DT.out

Hi JPlum,

Thaks for sharing your valuable time for me.

Here we are not having any permission to login to ftp machine It is maintaining my the clients .We just move dataextracts file to the ftp server through crontab.So I can,t make the ssh connections in the Ftp server.Can u please help me is there any command to find the folders inthe ftp server through script and if the folder found then i would like to get the mail foulder was found other wise not found.

Or
Here i would like to get folder back to the linux machine from the ftp server and i will find the particular folder in my machine if it was found then success other wise not success.More over the files must inthe Files must not be removed from the FTP site



In my below script when the crontab fires it will copy the text files in the dbdump directory and in ftp it will create a folder with the present data and copies the files in to that folder.Here i want to get the present date folder to my linux machine please help me in this script


#!/bin/bash
HOST='192.168.2.200'
USER='virinchi/krishna'
PASSWD='panetration'
#FILE='*'
#DT=$(date +%a_%h_%d_%Y)
#if [ ]
#then
DT=$(date +%Y-%m-%d)
ftp -n $HOST > .ftpout.txt 2>.ftperr.txt <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
lcd /u04/dbdump/
cd sample
mkdir $DT
cd $DT
prompt
mput *
cd ..
get $DT
quit
END_SCRIPT

Hi JPlum,

#!/bin/bash

HOST='192.168.2.200'
USER='virinchi/krishna'
PASSWD='panetration'

#FILE='*'

#DT=$(date +%a_%h_%d_%Y)

#if [ ]
#then

DT=$(date +%Y-%m-%d)
ping -c 3 -W 1 $HOST > /dev/null
if [ $? = 0 ]; then
#echo "Ping of $HOST failed; aborting now"
#exit 1
#fi
ftp -n $HOST > .ftpout.txt 2>.ftperr.txt <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
lcd /u04/dbdump/
cd sample
mkdir $DT
cd $DT
prompt
mput *
quit
END_SCRIPT
echo "Ping of $HOST success" #| #mail -s "success" poornaprasads@virinchi.com
else
echo "ping $HOST not success" #| #mail -s "not success" poornaprasads@virinchi.com
fi

As your mentioned i user the ping option here the file is moving but the it cont comming out of the ping option i am aboting it by ctrl+c after that it is showing "Ping of 192.168.2.200 success" why it is not coming out of the ping option when the file was moved to ftp.

Please help me.

and in my ftpout.txt file it is like this

'AUTH GSSAPI': command not understood
'AUTH KERBEROS_V4': command not understood
Local directory now /u04/dbdump
Interactive mode off.

send aborted
waiting for remote to finish abort

I don't think your listening very well. JPlum has given you some very good advice and you just seem to ignore it. Forget the ping command, it's not the problem. In this case your ctrl-c actually killed the ftp command, not the ping. First, run this thing manually. If you can't run it manually, you can't script it. As JPlum noted, your problem is most likely a problem with passive mode. Do you have access to the Windows ftp server? It may not allow passive mode. Make sure it does and check the logs on ftp server to see what is wrong. Also, when you get this working you will need to include the -i switch on the ftp command. Otherwise your mput command will hang waiting for you to acknowledge the transfer. But if you ran it manually, you would see that.

Hi,tidww01 we are not having any access to the ftp server directly. That's why i am trying in another way more over in order to make changes in the ftp with this link
(http://www.windowsnetworking.com/art...rver-2008.html) setting we need to request our client it will not accept by our Seniors . SO i am trying to find it out in some other way .

Please help me how to get the folder back to the linux server to another location.