[HOWTO] rsync with Google Drive

DaneM · 08-10-2014, 01:22 AM

Edit: This HOWTO is now available on the LQ Wiki! Please check it out here: http://wiki.linuxquestions.org/wiki/...h_Google_Drive

Hi, folks.

DISCLAIMER: Uploading to a cloud storage system implies that you trust the maintainers of that system and everyone in-between to not mess with/read your data. If this is a concern for you, but you want to upload to Google Drive, anyway, please consider using some of the encryption methods mentioned here (or another encryption method of your choosing): http://how-to.linuxcareer.com/using-...ages-and-files
Thanks to mostlyharmless for the tips!

It took me a while to find a good, simple, reliable way to backup my stuff to Google drive, using rsync; so I've decided to share my method with the good people at LQ. In short, the answer is to use "gsync" (NOT "grsync", which is different and broken/incomplete). It supports (so far as I can tell) ALL the same options as rsync (glee!), and lets you do it with Google Drive! You can upload to, and download from GD in this way, by picking which to use as SOURCE/DESTINATION folders.

First, go here and follow the installation instructions:
https://github.com/iwonbigbro/gsync/...ter/README.rst

DO NOT NEGLECT THE BIT ABOUT AUTHENTICATING!! If you do, none of the gsync stuff, below, will work!

For reference, here's the command I use to backup my stuff between my LOCAL hard drives--from "/mnt/PERSONAL/" to "/mnt/PERSONAL2":

Code:

sudo rsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/ /mnt/PERSONAL2

You can check what the options do using "man rsync." I don't always use the "-c" option, since it's slow (but more thorough for checking the data). This command will delete files that are missing from the destination, and overwrite duplicates. Use with care! Note the trailing slash on the source folder; this is important! A trailing slash on the source folder means to copy the stuff IN the folder to the destination. No slash means to copy the folder, itself, to the destination. The destination folder ignores trailing slashes. (Thanks to suicidaleggroll for this clarification.)

The "gsync" version is this--from "/mnt/PERSONAL/Dane/IMPORTANTSTUFF" to "drive://IMPORTANTSTUFF":

Code:

sudo gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/IMPORTANTSTUFF/ drive://IMPORTANTSTUFF

Please note that you should probably not upload an entire 1TB+ drive to GD unless you have and want to use up all that storage space on the cloud. Therefore, I've specified the subdirectory of "/mnt/PERSONAL/Dane/IMPORTANTSTUFF" to represent the important files/folders that I absolutely have to have backed-up remotely. You'll need to run a separate command for each folder (including subdirectories) that you want to upload in this fashion; make sure to change both the source and destination in the command when you do. (I haven't yet figured out how to do them all as a batch job, short of writing a script for it.) Also, I use root (sudo) for this and the rsync command because it helps manage permissions properly--but if you're certain that the current user/login owns all the files involved, you don't need it (and probably shoudn't use it, as a general security/safety precaution).

Finally, if you want to be able to walk away from it and know how long it actually took when you come back, you can prepend the "time" command to the beginning of the gsync or rsync command, like so:

Code:

sudo time gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/IMPORTANTSTUFF/ drive://IMPORTANTSTUFF

Enjoy!

--Dane

mostlyharmless · 08-11-2014, 02:51 PM

Very helpful for many people without doubt, perhaps you should also point out that you are relying on data protection/privacy on Google. You might want to consider an optional pipe through encryption.

DaneM · 08-11-2014, 03:15 PM

Thanks for your kind words, mostlyharmless. I like your suggestion about encryption. I haven't messed with that very much; what would you suggest?

I've always found gpg to be a bit too...arcane...for quick-and-dirty encryption, and I've lost my private key a couple of times (partly due to not knowing where it's stored); so I'm hesitant to use any kind of encryption whose key/password I can't just remember. I know it's not as secure as a 128-bit string, but I'm sure it's a lot better than nothing.

mostlyharmless · 08-12-2014, 07:53 AM

Well, there's always a balance between securing data and being able to retrieve it isn't there? I've lost data by misplacing/losing a GPG key too. Maybe something simpler using OpenSSL, as in the examples here:

http://how-to.linuxcareer.com/using-...ages-and-files

Even if not, just a disclaimer about it would be good.

suicidaleggroll · 08-12-2014, 10:27 AM

Quote:

Originally Posted by DaneM

Note the trailing slash on the source folder, but NOT the destination folder; this is important! Please see the man page (or a Linux/Unix web page on rsync) for what the trailing slashes do/don't do.

The trailing slash is only important on the source, it makes no difference on the destination.

DaneM · 08-12-2014, 01:04 PM

mostlyharmless: Added disclaimer and link. Thanks for the info! This looks VERY useful.

suicidaleggroll: Corrected info about trailing slashes. Thanks for clarifying!

--Dane

TobiSGD · 08-12-2014, 01:30 PM

Nice HOWTO.
Problem is that the forums are not really the best place for it, would be nice if you could add it to the LQ Wiki or make a LQ Article from it.

DaneM · 08-12-2014, 02:26 PM

I haven't done that, before. Where do I go?

TobiSGD · 08-13-2014, 05:19 AM

You can find the LQ Wiki on the right side in the main menu. For work in the Wiki you have to create a new user (you can take your current username), after that just create your article in the appropriate section.

DaneM · 08-13-2014, 02:31 PM

Thanks, TobiSGD.

DaneM · 08-13-2014, 02:53 PM

Posted!

http://wiki.linuxquestions.org/wiki/...h_Google_Drive

TobiSGD · 08-13-2014, 03:57 PM

Thanks for contributing!

vonedaddy · 11-20-2014, 08:45 PM

This is great, thanks!

I am having a small issue. It seems all the directory structure is being created but non of the actual files. I am using all the same options you are using.

gsync -t -i -r -p -o -g -v --progress --delete -l -s /vault/ drive://

Any ideas?

If I stop it, I get:

Interrupted
sent 0 bytes received 0 bytes 0.00 bytes/sec

vonedaddy · 11-20-2014, 08:58 PM

Found a fix here:

https://github.com/iwonbigbro/gsync/issues/69

Quote:

body = {}
for k, v in properties.iteritems():
body[k] = _Drive.utf8(v)
...to:

body = {}
for k, v in properties.iteritems():
if v is not None:
body[k] = _Drive.utf8(v)
...solves the problem, per #66.

It does, however, produce this when it runs into a file that ends in a ~ character:

Monsters/Lvl. 3 - Undead Horrid Housecat~
0 0% 0.00B/s 0:00:00DEBUG: 'Exception': File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update
status, res = req.next_chunk()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
return self._process_response(resp, content)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
raise HttpError(resp, content, uri=self.uri)

DaneM · 11-21-2014, 11:12 PM

I posted that fix (based on a post elsewhere). :-)

The program is, however, experiencing a more serious bug: its app authentication key is the same one used for everyone using the program. This is normal for Google-interfacing applications, but there's a data transfer limit on any one key. This means that, once all the gsync users have exceeded that limit, then nobody can use gsync, anymore, until the quota resets. There's an open bug about it, but I don't know if it's being worked on, currently. It's really unfortunate, because I love gsync's functionality.

Good luck!

--Dane