LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 05-27-2006, 12:15 PM   #1
mcrosby
Member
 
Registered: May 2006
Posts: 39

Rep: Reputation: 15
Syncronizing folders through ftp shell script


Hello I recently made a shell script to connect to a remote server and download the files to my computer and/or put files on the remote server. Does anyone know how i can make the script so it would automatically syncronize the two folders?
 
Old 05-27-2006, 03:18 PM   #2
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
To be honest, I'd probably just use rsync over SSH to do this. It would be easier and more secure than FTP.

Assuming you want to reinvent the wheel, though, you'd need your script to check the mtime of all the files. Anything on the source that was modified after the corresponding file on the destination would be re-uploaded to keep the files in sync. Files existing in the destination but not the source would be deleted. Of course, files that exist on the source but not on the destination would also have to be uploaded. These tasks could be written as a simple directory search. You'd have to make a list of all the files and their mtimes on the source and destination (possibly painful just going over FTP), and then run a script to compare them and make a list of uploads/deletions, and then process it through your script.
 
Old 05-27-2006, 07:38 PM   #3
mcrosby
Member
 
Registered: May 2006
Posts: 39

Original Poster
Rep: Reputation: 15
ok haha so how would i do this over rsync and ssh
 
Old 05-29-2006, 03:52 AM   #4
joseph
Member
 
Registered: Jun 2003
Location: Batam
Distribution: Ubuntu 10 And Linux Mint
Posts: 414

Rep: Reputation: 30
Hi guy

Try this link, it should solved your problem

http://www.jdmz.net/ssh/
 
Old 05-29-2006, 04:51 AM   #5
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
Judging by your reply, I doubt that you'll proceed with FTP and use rsync instead.

Using FTP, an easy solution would be to use wget on both machines. Downside would be that you'll need a ftp daemon running on both machines.
wget can take into account last modification times (-N option), recurse through directory trees (-r, -l),
can continue when it was abruptly halted (-c), etc.
 
Old 05-30-2006, 10:41 AM   #6
mcrosby
Member
 
Registered: May 2006
Posts: 39

Original Poster
Rep: Reputation: 15
instead of syncronizing how would i go about downloading the files and then comparing the files i downloaded from the remote server to those files i have in a folder on my local drive. If the folders are not equal it would then write to a file sayin so..any ideas?
 
Old 05-31-2006, 01:58 AM   #7
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
You could use a command like "diff" (for text files) or "cmp" (for binary & text files) to compare files.
diff reports the result, whereas cmp's result must be checked by the return code ($? parameter in Bash, just after cmp call).

An alternative, often used in distribution of software packages, is the use of checksums. MD5 checksums are most popular here, I'd say. Check out the tool "md5sum".
Personnally, I prefer checksums, certainly in cases where the files you need to compare are on different machines. Sharing (or copying) the small checksum result files between machines is easier and faster than copying around the (sometimes very big) files you need to compare.

Some tools limit their checks to -for instance- file size & last modification dates. But for my applications, neither is considered reliable enough.

Last edited by timmeke; 05-31-2006 at 02:00 AM.
 
Old 05-31-2006, 06:38 AM   #8
mcrosby
Member
 
Registered: May 2006
Posts: 39

Original Poster
Rep: Reputation: 15
I was actually looking into using diffs. The files will all be on the same machine. The files from the remote host will be downloaded to a folder on the local machine and then compared to another folder located on the remote machine. I have my ftp script that connects and downloads the files should the script that compares the folders be in a seperate script and can somebody get me started i am fairly new to scripting.
 
Old 05-31-2006, 07:34 AM   #9
mcrosby
Member
 
Registered: May 2006
Posts: 39

Original Poster
Rep: Reputation: 15
Ok i seem to have the diff script working on its own and it records the differences to a file. How would i go about combining my ftp script and my diff script because if i end the ftp script it closes out the script entirely.
 
Old 05-31-2006, 09:39 AM   #10
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
What shell are you using? Bash? Perl? Tcl?
In Bash, it may not be so easy (ie involves pipes, etc).
In Perl, this is easy:
Code:
open(myFTP, "|ftp some_URL");
print myFTP "your_FTP_Commands";
#Inbetween command printing (multiple "print" statements), you can execute other stuff.
close(myFTP);
In Tcl/Tk, it's similar.

But let us see your script so far and we'll work from there...

Last edited by timmeke; 05-31-2006 at 09:41 AM.
 
Old 05-31-2006, 09:55 AM   #11
mcrosby
Member
 
Registered: May 2006
Posts: 39

Original Poster
Rep: Reputation: 15
i actually ended up getting it all to work but here is my script if you were curious

Quote:
# !/bin/bash
ftp -n -i <<EOF
open ftp.coe.neu.edu
user usr pwd
bin
cd www/backup
lcd ~/Desktop/backup
mget *.*
EOF

cd ~/Desktop
Today="`date +%m%d`"
diff -q backup bin > results-$Today.txt
 
Old 05-31-2006, 01:00 PM   #12
mcrosby
Member
 
Registered: May 2006
Posts: 39

Original Poster
Rep: Reputation: 15
now if I wanted to get this to work using cmp I would have to make a loop of some sort and compare each file individually correct? I would i go about comparing the files as they are downloaded to a file in the directory on the local system
 
Old 06-01-2006, 02:09 AM   #13
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
Quote:
now if I wanted to get this to work using cmp I would have to make a loop of some sort and compare each file individually correct?
Indeed.

Assuming the files have the same name and the directory structure is similar, you could use "find dir -type f" to track down all the files (recursively) and then execute the cmp for each file found.

In other words, something like:
Code:
cd your_download_dir;
files=`find . -type f`;
for file in $files; do
cmp $file your_local_dir/$file
if (( $? != 0 )); then
   echo "cmp says files $file are different"
else
   echo "cmp says files $file are identical"
fi;
done;
Please pay attention to the paths.
Try the "find" command alone for starters to get an idea of what it returns and how you need to modify this for your local directory.

In the example, you'll need to:
-replace the echo's with more useful commands
-replace "your_local_dir" and "your_download_dir" with the paths where the files are stored. Both dirs must have the same structure for the above script to work "as is".
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Auto ftp shell script mcrosby Linux - Newbie 9 06-27-2007 06:09 AM
Automating FTP upload via a shell script? Spitty Programming 5 01-16-2006 10:28 PM
shell script to check ftp communication yuva_mca Linux - General 2 12-01-2005 07:15 AM
Trouble when automating ftp with shell script hari_s_82 Linux - Newbie 2 09-23-2004 02:17 AM
How to Shell Script auto ftp connection? mtn356 Linux - General 4 03-18-2004 07:31 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 07:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration