LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-25-2017, 12:17 PM   #1
scottm7
Member
 
Registered: Apr 2017
Posts: 31

Rep: Reputation: 30
Rsyncing to Google Drive Using google-drive-ocamlfuse


I'm using google-drive-ocamlfuse to copy files to Google Drive. I'm using the following Bash Script to do it. It is very slow but I'm sure that's primarily because of the connection. The first run is copying 490 GB. I'm trying to get a good start on it so it will be done in a couple of weeks or less.

I tried using the parameters rlvW. I saw those at: https://github.com/astrada/google-dr...use/wiki/rsync. It seemed really slow so now I'm using azvW.

Does anyone have any advice here or is azvW good enough? I'm new to google-drive-ocamlfuse and I've been using the same rync over and over for my regular drive with azuv.

Thanks!

Quote:
#! /bin/bash
drives=("wdgld6" "wdblk6" "wdred3" "wdblk1" "gdrive")
for i in "${drives[@]}"
do
if [ -d /run/media/user01/$i/backup/ ]; then
echo '--> STARTING '
echo '--> STARTING ' $i $i $i $i $i $i
echo '--> STARTING '
echo $(date)
# VERBOSE = -v
# rsync -azu --exclude 'do-not-sync' --delete
# rsync -azu --delete
folders=("Documents" "Downloads" "Music" "Pictures" "Templates" "Videos")
for f in "${folders[@]}"
do
if [ ! -d /run/media/user01/$i/backup/$f ]; then
mkdir /run/media/user01/$i/backup/$f
fi
echo '--> STARTING '
echo '--> STARTING ' $i $f $f $f $f $f
echo '--> STARTING '
if [ $i = "gdrive" ]; then
rsync -azvW --inplace --size-only --delete /run/media/user01/wdgld6/backup/$f/ /run/media/user01/gdrive/backup/$f/
else
rsync -azuv --exclude-from='/media/user01/Documents/linux/rsync-exclude.txt' --delete-excluded --delete /media/user01/$f/ /run/media/user01/$i/backup/$f/
fi
echo '--> FINISHED '
echo '--> FINISHED ' $i $f $f $f $f $f
echo '--> FINISHED '
done
fi
echo $(date)
echo '--> FINISHED '
echo '--> FINISHED ' $i $i $i $i $i $i
echo '--> FINISHED '
i=$(($i + 1))
done

Last edited by scottm7; 10-25-2017 at 12:44 PM. Reason: Removed drive
 
Old 10-25-2017, 12:36 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 25,654

Rep: Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653
I'd avoid rsync totally using Gdrive and the ocaml-fuse solution. Done that before, and rsync was the cause (unproven, but speculation) of the slower speed. Use rclone instead
https://rclone.org/install/
 
1 members found this post helpful.
Old 10-25-2017, 12:43 PM   #3
scottm7
Member
 
Registered: Apr 2017
Posts: 31

Original Poster
Rep: Reputation: 30
Wow! That seems to be much faster. It's already copying quite a few files.

Should I run this command the second time or something else?

rclone copy /run/media/user01/wdgld6/backup/$f/ /run/media/user01/gdrive/backup/$f/
 
Old 10-25-2017, 12:50 PM   #4
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 25,654

Rep: Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653Reputation: 7653
Quote:
Originally Posted by scottm7 View Post
Wow! That seems to be much faster. It's already copying quite a few files.

Should I run this command the second time or something else?

rclone copy /run/media/user01/wdgld6/backup/$f/ /run/media/user01/gdrive/backup/$f/
Depends on what you're trying to accomplish. Run "rclone --help", and explore the sync vs. copy option.
 
1 members found this post helpful.
Old 10-25-2017, 12:59 PM   #5
scottm7
Member
 
Registered: Apr 2017
Posts: 31

Original Poster
Rep: Reputation: 30
For those of you coming to this thread at a later date, the above suggestion didn't end up working for me but that doesn't mean it won't for you. I'm currently backing by uploading the files to Google Drive using the web browser. At it's current rate it should finish in 2 weeks. I'll then explore how to sync making the data on Google Drive identical to the data from the source.

Last edited by scottm7; 10-26-2017 at 01:36 AM. Reason: edited
 
Old 10-27-2017, 12:34 PM   #6
scottm7
Member
 
Registered: Apr 2017
Posts: 31

Original Poster
Rep: Reputation: 30
For those of you who do a web search for netbrake or trickle for grive or grive-tools and end up here - this is what worked for me:

I adjusted download and upload several times before I got it. Sync has to find out what needs to be transferred and that can take awhile. I probably don't need such a low limit on the download side especially because right now I'm just trying to get the 490 GB up there the first time. I was bottlenecking in the upload Kbps.

trickle -vs -d 450 -u 450 /opt/grive-tools/grive-indicator

On gnome-system-monitor I saw before where it was a flat line for the upload rate but now it looks like a cosine.

I may adjust the download rate later and remove verbose but for now it's working great!

Last edited by scottm7; 10-27-2017 at 12:37 PM. Reason: formatted the command line
 
1 members found this post helpful.
  


Reply

Tags
rsync


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Mount Google Drive On Linux With Google Drive Ocamlfuse Client LXer Syndicated Linux News 0 09-11-2017 01:25 PM
Backup to Google Drive TB0ne LinuxQuestions.org Member Success Stories 2 05-17-2016 10:11 PM
LXer: Google Adds Your Google Drive Files To SERPs LXer Syndicated Linux News 0 10-16-2012 01:40 PM
What does Google Apps moving to google drive mean to us? dougnc Linux - General 11 09-13-2012 01:58 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration