LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Ubuntu (https://www.linuxquestions.org/questions/ubuntu-63/)
-   -   How to automatically detect and backup changes to folder? (https://www.linuxquestions.org/questions/ubuntu-63/how-to-automatically-detect-and-backup-changes-to-folder-707047/)

sentiax 02-24-2009 02:49 AM

How to automatically detect and backup changes to folder?
 
Ok so here's the deal. I've got a main gaming desktop that I use all the time. I've also got an HTPC and a Fileserver. My dilemma is that I like to have my Music, Pictures, Videos, and other important data backed up onto the Fileserver (which is running Xubuntu).

I've gotten FTP, Samba, and everything setup correctly, can access it from outside the home network and stuff so I can stream music to my laptop when I'm on my college campus without having to take up so much hdd space on my laptop, as well as backup my programming and essay homework.

What I would like to do is have the fileserver automatically detect changes in my main desktops 1TB hard drive where all the Music, videos etc are originally from and then copy the new files to itself. Is there any way to have this done without it completely copying the whole folders over. I've found a guide on how to do it with rsync, but it basically copies the entire folder over again, and with 230GB worth of data, that's not worth it every time there's a 20MB change.

Just as an example and for more details in case I've made this confusing:

My terrabyte hard drive on my main Desktop is J:\. The folders I have backed up onto the Fileserver from J:\ are titled Games, Installs, ISO, Pictures, Music, Vids and are named exactly that on the fileserver as well. Let's say I buy a new Album from Itunes and save it to J:\Music\<artist name>\<album name> is there a way to have the fileserver detect that change, and then automatically copy JUST the new data?

Thanks in advance for help. Let me know if I need to give any more info.

zedmelon 02-24-2009 04:54 AM

Quote:

Originally Posted by sentiax (Post 3455351)
I've found a guide on how to do it with rsync, but it basically copies the entire folder over again, and with 230GB worth of data, that's not worth it every time there's a 20MB change.

Are you certain the date/time on both machines is the same? That certainly wouldn't be worth it, which is the point of rsync. In my experience, rsync takes an eternity for only the first run. Once it's made the initial sync, it will only bother with that 20M thereafter.

Timestamps can affect this as well as file size. You can alleviate that with a
cp -p.

jschiwal 02-24-2009 05:28 AM

I don't know what you mean by copy files to itself. You can use find to locate new files. Another option is rsync. Also tar with the -g option for incremental backups. ( See section 5.2 of the info manual for tar for examples. )

sentiax 02-25-2009 12:42 AM

Sorry for the confusion. I actually figured it out using robocopy and a batch file in my gaming machine.
What I meant by copy to itself is:

On my Windows Gaming Machine there is an eSata 1TB hard drive that I keep all the aforementioned folders on Labeled as J:\
What I wanted to happen, is I wanted my Xubuntu Fileserver to automatically copy those folders too itself every night or whenever I wanted on a regular schedule.
My problem was that using a guide I found, it copied and overwrote everything on itself instead of just updating the files or copying new ones that were on J:\

After I posted on here, I actually figured out how to do it the way I wanted with rsync, but it was incredibly slow (we're talking 5KB/s and stuff).
So instead I just created a bath file on the windows machine that ignores existing files on the server and only copies new ones, and does it at around 12MB/s (so basically my servers capacity with a 100mbps nic card).

I've got a gigabit card coming in for the server (which is like an 8 year old dell I bought for 10$) which should boost the transfer speeds up exponentially.
I might try rsync again when it comes in, but I tried rsync even manually and it was still incredibly slow even after the files were already on the server. Let me know if you have any other ideas though. I would prefer it to run on the server itself rather than the Windows machine.


All times are GMT -5. The time now is 01:37 AM.