LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 04-26-2009, 06:24 AM   #1
babalot
LQ Newbie
 
Registered: Apr 2009
Posts: 12

Rep: Reputation: 0
rsync bash script w/ progress bar?


I'm writing a script to automate rsync backups which so far is working great. I prefer to send output to a text file instead of watching it fly by but the blank output is slightly boring? I would love to use a progress bar. Not for each file but for overall progress. Something like

syncing /home directory 50% [++++++++ ]

I already know it's way over my head and maybe bash isn't even the right tool for the job. If anyone could drop some tips and maybe point me in the right direction, I would really appreciate it.

Thanks!

Sorry if I posted this in the wrong area, seemed programming related.

Last edited by babalot; 04-26-2009 at 06:26 AM.
 
Old 04-27-2009, 05:36 AM   #2
babalot
LQ Newbie
 
Registered: Apr 2009
Posts: 12

Original Poster
Rep: Reputation: 0
Perhaps the file by file progress is the only way to go. It seems rsync doesn't pre-calculate the backup but only checks files on a per file basis. I guess that would make an overall progress bar rather difficult and probably slow the actual transfer greatly.

Wondering, if the incremental backups are small, would it be possible to run a dry run prior to in order to get total size, then show progression based on that estimate and rsync transfer rate?
 
Old 04-27-2009, 06:21 AM   #3
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,064

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
It seems that this simple question has a complicated solution. That is silly because while rsync is doing the preparations it would be easy to calculate total file size and show overall progress. As a matter of fact this information is available in rsysnc because it gives you statistics at the end of the run.

However, with the current output of rsync this is not so easy.

When I was thinking about this problem yesterday I found not easy solution and I hoped someone else had a better idea. Apparently not.

Your idea of a dry run is useful, however the dry run takes a lot of time, exactly as much as the preparation for the actual copying. That is a lot of time.

The script would be something like this:
- perform a dry run
- catch the list of files in the output
- calculate total file size
- perform the actual run
- catch the output of rsync and calculate the size of every file transferred.
- substract this from the total file size.
- suppress all rsync output and show your progress bar.

In this way it looks like you are using the great capabilities of rsync not just right.

Maybe a suggestion is to let rsync find out the files which have to copied in a dry run, and use another transfer mechanism in your script to actually copy the files.

It almost seems appropriate to go for a source hack in rsync.

jlinkels
 
Old 04-28-2009, 06:45 AM   #4
babalot
LQ Newbie
 
Registered: Apr 2009
Posts: 12

Original Poster
Rep: Reputation: 0
First off, thanks for the reply.

Code:
The script would be something like this:
- perform a dry run
- catch the list of files in the output
- calculate total file size
- perform the actual run
- catch the output of rsync and calculate the size of every file transferred.
- substract this from the total file size.
- suppress all rsync output and show your progress bar.
^ that's what I was thinking except, instead of using rysnc's output to calculate, use the actual directory size.

- perform the dry run
- grab the 'total size' from rsync's output
- convert to KB
- add that total size to the current size of the backup directory = $total
- run rsync in the background
- use something like 'du -s' in a loop to grab the dir's current size compared to the expected $total and use that for the progress bar

I believe that would work but, would it be a decent solution? The extra time required for the dry run is fine by me. For now anyway. This is my first bash script and it's more or less for the learning experience. My script as is does what I need it to. The progress bar is just to give myself a new problem to solve.
 
Old 04-28-2009, 11:09 AM   #5
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,064

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
Quote:
Originally Posted by babalot View Post
- add that total size to the current size of the backup directory = $total
- run rsync in the background
- use something like 'du -s' in a loop to grab the dir's current size compared to the expected $total and use that for the progress bar
If you do it like this, you ignore that rsync doesn't backup all files if it does an update. Usually you will not be backing all your files, but only the new/changed ones. If you always make a fresh copy there is no difference.

If this is mainly a programming exercise it doesn't matter so much whether it is efficient or not. However, if you are copying between mounted drives (not thru SSH to another machine) and you try to use rsync in this way I would say you are trying to exploit rsync in a way contrary to which it was designed for. Or at least contrary to the output it gives. Add to that that rsync with spits out output with ^M characters instead of newlines which makes it even more difficult to create something useful.

If the copy action is within the same directory tree, (mounted drives) it might be easier to retrieve directory listings including full path of both your original and your copy, retrieve access or change date for both, decide what and how much to copy and do that. As a matter of fact you are implementing a part of the functionality of rsync, but you keep track of counting values which are now only internal to rsync.

jlinkels
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script to rsync remote dir with options uopjohnson Linux - General 3 04-23-2010 01:16 AM
Bash Download Manager (using wget) and progress bar rastacre Programming 1 08-29-2008 01:33 PM
Shell Script Progress bar rooch84 Linux - Software 8 09-03-2005 04:55 PM
PROGRESS BAR with shell script murugesan Linux - Software 5 12-13-2004 09:03 AM
bash script HELP - dnotify > rsync GuitsBoy Programming 5 01-28-2004 09:42 AM


All times are GMT -5. The time now is 10:59 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration