LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (http://www.linuxquestions.org/questions/programming-9/)
-   -   rsync bash script w/ progress bar? (http://www.linuxquestions.org/questions/programming-9/rsync-bash-script-w-progress-bar-721771/)

babalot 04-26-2009 06:24 AM

rsync bash script w/ progress bar?
 
I'm writing a script to automate rsync backups which so far is working great. I prefer to send output to a text file instead of watching it fly by but the blank output is slightly boring? I would love to use a progress bar. Not for each file but for overall progress. Something like

syncing /home directory 50% [++++++++ ]

I already know it's way over my head and maybe bash isn't even the right tool for the job. If anyone could drop some tips and maybe point me in the right direction, I would really appreciate it.

Thanks!

Sorry if I posted this in the wrong area, seemed programming related.

babalot 04-27-2009 05:36 AM

Perhaps the file by file progress is the only way to go. It seems rsync doesn't pre-calculate the backup but only checks files on a per file basis. I guess that would make an overall progress bar rather difficult and probably slow the actual transfer greatly.

Wondering, if the incremental backups are small, would it be possible to run a dry run prior to in order to get total size, then show progression based on that estimate and rsync transfer rate?

jlinkels 04-27-2009 06:21 AM

It seems that this simple question has a complicated solution. That is silly because while rsync is doing the preparations it would be easy to calculate total file size and show overall progress. As a matter of fact this information is available in rsysnc because it gives you statistics at the end of the run.

However, with the current output of rsync this is not so easy.

When I was thinking about this problem yesterday I found not easy solution and I hoped someone else had a better idea. Apparently not.

Your idea of a dry run is useful, however the dry run takes a lot of time, exactly as much as the preparation for the actual copying. That is a lot of time.

The script would be something like this:
- perform a dry run
- catch the list of files in the output
- calculate total file size
- perform the actual run
- catch the output of rsync and calculate the size of every file transferred.
- substract this from the total file size.
- suppress all rsync output and show your progress bar.

In this way it looks like you are using the great capabilities of rsync not just right.

Maybe a suggestion is to let rsync find out the files which have to copied in a dry run, and use another transfer mechanism in your script to actually copy the files.

It almost seems appropriate to go for a source hack in rsync.

jlinkels

babalot 04-28-2009 06:45 AM

First off, thanks for the reply.

Code:

The script would be something like this:
- perform a dry run
- catch the list of files in the output
- calculate total file size
- perform the actual run
- catch the output of rsync and calculate the size of every file transferred.
- substract this from the total file size.

- suppress all rsync output and show your progress bar.

^ that's what I was thinking except, instead of using rysnc's output to calculate, use the actual directory size.

- perform the dry run
- grab the 'total size' from rsync's output
- convert to KB
- add that total size to the current size of the backup directory = $total
- run rsync in the background
- use something like 'du -s' in a loop to grab the dir's current size compared to the expected $total and use that for the progress bar

I believe that would work but, would it be a decent solution? The extra time required for the dry run is fine by me. For now anyway. This is my first bash script and it's more or less for the learning experience. My script as is does what I need it to. The progress bar is just to give myself a new problem to solve.

jlinkels 04-28-2009 11:09 AM

Quote:

Originally Posted by babalot (Post 3523433)
- add that total size to the current size of the backup directory = $total
- run rsync in the background
- use something like 'du -s' in a loop to grab the dir's current size compared to the expected $total and use that for the progress bar

If you do it like this, you ignore that rsync doesn't backup all files if it does an update. Usually you will not be backing all your files, but only the new/changed ones. If you always make a fresh copy there is no difference.

If this is mainly a programming exercise it doesn't matter so much whether it is efficient or not. However, if you are copying between mounted drives (not thru SSH to another machine) and you try to use rsync in this way I would say you are trying to exploit rsync in a way contrary to which it was designed for. Or at least contrary to the output it gives. Add to that that rsync with spits out output with ^M characters instead of newlines which makes it even more difficult to create something useful.

If the copy action is within the same directory tree, (mounted drives) it might be easier to retrieve directory listings including full path of both your original and your copy, retrieve access or change date for both, decide what and how much to copy and do that. As a matter of fact you are implementing a part of the functionality of rsync, but you keep track of counting values which are now only internal to rsync.

jlinkels


All times are GMT -5. The time now is 11:30 AM.