[HOWTO] rsync with Google Drive
Edit: This HOWTO is now available on the LQ Wiki! Please check it out here: http://wiki.linuxquestions.org/wiki/...h_Google_Drive
Hi, folks. DISCLAIMER: Uploading to a cloud storage system implies that you trust the maintainers of that system and everyone in-between to not mess with/read your data. If this is a concern for you, but you want to upload to Google Drive, anyway, please consider using some of the encryption methods mentioned here (or another encryption method of your choosing): http://how-to.linuxcareer.com/using-...ages-and-files Thanks to mostlyharmless for the tips! It took me a while to find a good, simple, reliable way to backup my stuff to Google drive, using rsync; so I've decided to share my method with the good people at LQ. In short, the answer is to use "gsync" (NOT "grsync", which is different and broken/incomplete). It supports (so far as I can tell) ALL the same options as rsync (glee!), and lets you do it with Google Drive! You can upload to, and download from GD in this way, by picking which to use as SOURCE/DESTINATION folders. First, go here and follow the installation instructions: https://github.com/iwonbigbro/gsync/...ter/README.rst DO NOT NEGLECT THE BIT ABOUT AUTHENTICATING!! If you do, none of the gsync stuff, below, will work! For reference, here's the command I use to backup my stuff between my LOCAL hard drives--from "/mnt/PERSONAL/" to "/mnt/PERSONAL2": Code:
sudo rsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/ /mnt/PERSONAL2 The "gsync" version is this--from "/mnt/PERSONAL/Dane/IMPORTANTSTUFF" to "drive://IMPORTANTSTUFF": Code:
sudo gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/IMPORTANTSTUFF/ drive://IMPORTANTSTUFF Finally, if you want to be able to walk away from it and know how long it actually took when you come back, you can prepend the "time" command to the beginning of the gsync or rsync command, like so: Code:
sudo time gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/IMPORTANTSTUFF/ drive://IMPORTANTSTUFF --Dane |
Very helpful for many people without doubt, perhaps you should also point out that you are relying on data protection/privacy on Google. You might want to consider an optional pipe through encryption.
|
Thanks for your kind words, mostlyharmless. I like your suggestion about encryption. I haven't messed with that very much; what would you suggest?
I've always found gpg to be a bit too...arcane...for quick-and-dirty encryption, and I've lost my private key a couple of times (partly due to not knowing where it's stored); so I'm hesitant to use any kind of encryption whose key/password I can't just remember. I know it's not as secure as a 128-bit string, but I'm sure it's a lot better than nothing. |
Well, there's always a balance between securing data and being able to retrieve it isn't there? I've lost data by misplacing/losing a GPG key too. Maybe something simpler using OpenSSL, as in the examples here:
http://how-to.linuxcareer.com/using-...ages-and-files Even if not, just a disclaimer about it would be good. |
Quote:
|
mostlyharmless: Added disclaimer and link. Thanks for the info! This looks VERY useful.
suicidaleggroll: Corrected info about trailing slashes. Thanks for clarifying! --Dane |
Nice HOWTO.
Problem is that the forums are not really the best place for it, would be nice if you could add it to the LQ Wiki or make a LQ Article from it. |
I haven't done that, before. Where do I go?
|
You can find the LQ Wiki on the right side in the main menu. For work in the Wiki you have to create a new user (you can take your current username), after that just create your article in the appropriate section.
|
Thanks, TobiSGD.
|
|
Thanks for contributing!
|
This is great, thanks!
I am having a small issue. It seems all the directory structure is being created but non of the actual files. I am using all the same options you are using. gsync -t -i -r -p -o -g -v --progress --delete -l -s /vault/ drive:// Any ideas? If I stop it, I get: Interrupted sent 0 bytes received 0 bytes 0.00 bytes/sec |
Found a fix here:
https://github.com/iwonbigbro/gsync/issues/69 Quote: body = {} for k, v in properties.iteritems(): body[k] = _Drive.utf8(v) ...to: body = {} for k, v in properties.iteritems(): if v is not None: body[k] = _Drive.utf8(v) ...solves the problem, per #66. It does, however, produce this when it runs into a file that ends in a ~ character: Monsters/Lvl. 3 - Undead Horrid Housecat~ 0 0% 0.00B/s 0:00:00DEBUG: 'Exception': File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update status, res = req.next_chunk() File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper return wrapped(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk return self._process_response(resp, content) File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response raise HttpError(resp, content, uri=self.uri) |
I posted that fix (based on a post elsewhere). :-)
The program is, however, experiencing a more serious bug: its app authentication key is the same one used for everyone using the program. This is normal for Google-interfacing applications, but there's a data transfer limit on any one key. This means that, once all the gsync users have exceeded that limit, then nobody can use gsync, anymore, until the quota resets. There's an open bug about it, but I don't know if it's being worked on, currently. It's really unfortunate, because I love gsync's functionality. Good luck! --Dane |
All times are GMT -5. The time now is 06:26 AM. |