How to Sync Files
 

Hi!,

I'm maintaining a company website that is hosted in a remote computer. The website accepts file submission(xls, doc, pdf, etc) via an online form. Files submitted are then strored in a directory. Every afternoon all files that are sent have to be downloaded via ftp and or web. Its a very time consuming process. I was thinking if there is a way to automatically synchronize na file from my remote server to my local server?

I'm running slackware linux as my local server and the 3rd party webserver is also running linux.

Thanks in advance!

You could use rsync and set it up as cron job to do every day or whenever.

do i need to have rsync installed and running on my remote web server? i'm asking because i dont have complete control on the web server. presenty, i can only use ftp and the web control panel to download the files.

Yes the rsync service has to be running on the remote machine. If this is not possibe then use wget with the -r switch.

PRECISELY WHAT I'M LOOKING FOR! I'LL READ THE DOCUMENTATION. THANKS!

one thing about that. wget with the -r switch is a "dumb" rsync. It will just grab everything every time you use it. rsync will just transferring things that changed. you can write a wget script checking for dates or something to get it to be more like rsync.

i'm already trying it. just like what you've said, its a little dumb but still very useful. i'm not really into shell scripting, can you help me with this or can you point me to a tutorial?

uup! when i was reading the documentation using -N will download the files that were updated in the server. I'm still trying it out.

You can do it with wget. Here's an example:
This will download everything in the "isolinux", as well as everything in any directory below it, to the directory you are in when you execute it. Here's how it breaks down.

-m - This turns on recursion, timestamping, & infinite recursion depth

-np - This prevents wget from ascending into the directories above the one you want

-nH - This cuts off the slackware.mirrors.tds.net

--cut-dirs=4 - This cuts off the pub/slackware/slackware-10.1/isolinux

Without the last 2 your files would be stored at "slackware.mirrors.tds.net/pub/slackware/slackware-10.1/isolinux" in under the directory you ran it from. See "man wget", as there are loads of options. If you want to run this daily automatically, then use cron, along with the -P option to wget. The -P option tells wget where to store the files, default is "." or current directory. To do that, run "crontab -e" from a command line and add something like this:
That would run the command every day at 6:10 AM and email you the output. You add " 1> /dev/null" without quotes onto the end if you don't want to be emailed the output. Or you can use the -o option to wget to write the output to a log file instead and just email you errors. As you can see, 100s of ways to skin this cat. :) At any rate, the above will only download files that change and do it automatically so you don't have to touch it. If you want to use the HTTP protocol instead, then that's possible as well, but it introduces other issues like the index.html files and robots.txt if they apply. Anyway, play with it a bit to get it like you want it before you automate it.

but how to download only the files that were updated?

What I posted will do just that.