LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-02-2016, 06:25 PM   #16
TxLonghorn
Member
 
Registered: Feb 2004
Location: Austin Texas
Distribution: Mandrake 9.2
Posts: 702

Rep: Reputation: 231Reputation: 231Reputation: 231

One of the coolest things about rsync is that you can test run it. Add --dry-run to your command, and it will show you what the result will be without actually making any changes.
If you approve of the results, you can then run it again without --dry-run
Code:
cd ~ && rsync -avz --dry-run  . /path/to/save/to/ --delete --log-file=/path/to/report.rpt
Personally, I use grsync - the GUI program. After it is configured the way you want, it only takes a couple of clicks to do your backup.
 
1 members found this post helpful.
Old 02-02-2016, 08:43 PM   #17
Mikech
Member
 
Registered: Jan 2006
Location: USA
Distribution: Mint
Posts: 90

Original Poster
Rep: Reputation: 24
Thanks TXLonghorn! I wonder why you guys don't need "sudo" and I do? I didn't use the "CD ~ &&" because Mint 17.3 Mate automatically puts you in your home directory as soon as you open the terminal. Ill take a look at grysnc. Maybe that will work. None of the others have after the first aborted run. But the command line seems to be working so far without error and I am up to 200 gigabytes.

OK I ran grsync. It was already on my computer but it said "synchronize files with rsync" Since I had no idea what that meant I left it alone and forgot all about it. But I did a simulate from the GUI and it ran like a charm without errors. Hopefully it will also run the second time! Thanks very much. I may be installing a new hard drive this weekend!!

Last edited by Mikech; 02-02-2016 at 08:55 PM.
 
Old 02-02-2016, 10:08 PM   #18
TxLonghorn
Member
 
Registered: Feb 2004
Location: Austin Texas
Distribution: Mandrake 9.2
Posts: 702

Rep: Reputation: 231Reputation: 231Reputation: 231
Quote:
Originally Posted by Mikech View Post
I wonder why you guys don't need "sudo" and I do?
Often the "sudo" is left off when posting because it is understood. But, also, I believe there are some distros that don't use "sudo" - you have to "su".

By the way, you should not run the command you have on a running system. You should do it from a live DVD or USB. There are system operations which do not need to be rsynced, and can cause problems.
If you want to do the rsync while logged on to the system, exclude the unnecessary processes.
Code:
rsync -avz  --exclude={/proc,/dev,/media,/mnt,/sys,/tmp}  . /path/to/save/to/ --delete --log-file=/path/to/report.rpt
 
1 members found this post helpful.
Old 02-03-2016, 08:52 AM   #19
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Mike:

I offered
Code:
cd ~ && rsync -avz --dry-run  . ...
because new users may take that snippet and try to run it from some place other than their home.

Of course I could have modified the rsync parameter...
 
1 members found this post helpful.
Old 02-03-2016, 10:23 AM   #20
Mikech
Member
 
Registered: Jan 2006
Location: USA
Distribution: Mint
Posts: 90

Original Poster
Rep: Reputation: 24
Thanks to all who have helped me. I especially want to thank Habitual who spent a lot of time helping me understand things.

I have learned a great deal about Linux with this adventure. Using the command line that Habitual and TXLonghorn provided are the only things that worked. I now have 278.4 gigabytes on the Passport external hard drive and 278.4 on the source. While I had some errors, I don't care, because all the files I wanted are there so its working.

Not a single one of the 5 GUI front ends to rsync worked (including grsync). They either did nothing, refused to start, generated errors, or created a new directory inside /media/q and started a backup there instead of to the external hard drive (two of them did that!). I haven't got that many years left so I am not going to waste them trying to figure out why. The command line works good enough and I understand it well enough now to use it easily! Now I see why people use the CLI.

I will continue to study the rsync command though just so that I know what I am doing. Maybe then I can return the favor and help others.

Last edited by Mikech; 02-03-2016 at 10:28 AM.
 
Old 02-03-2016, 11:03 AM   #21
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Glad it worked out for you.
 
Old 02-03-2016, 09:43 PM   #22
polaris96
Member
 
Registered: Jan 2015
Distribution: Slackware, LFS, OpenIndiana, debian wheezy
Posts: 55

Rep: Reputation: Disabled
nice work. I've been meaning to write one of these for awhile. you may prefer this for your cron job...

Code:
#!/bin/bash
##ultra simple backup program

##dependencies:  bash rsync




###20160203 Polaris96  tested ok
##NOTES:
##
##  1.  this script requires root priviledges
##
##  2.  use any text editor (vi, nano, emacs, medit, kate, openoffice writer...)
##      to examine the logfile
##
##  3.  feel free to tweak anything
##
##  4.  use at your own risk
##





#define your primary partition.  this is what we're backing up
#the environment variable $HOME should already hold the path we need
PRIMARYDIR=$HOME

#define the backup partition.  this is for the device node, not the mount
#if it's removable and you plan on detaching it, we'll need a udev rule.
#this assumes you'll leave it plugged in.
BACKUP_PARTITION=$/dev/<your passport partition>

#replace <your passport partition> with the actual designator
#if you aren't sure where it is run this from the command line
# cat /etc/mtab | awk '/MyPassport/{print $1}'
#you must have 'MyPassport' mounted when you run it.  it will give the correct
#device Node.

#now tell bash where to mount the partition.
BACKUP_DIRECTORY=/media/q/MyPassport

#and create a logfile so you can track your backups
#change backups.log to a similar mnemonic title if you want.
BACKUP_LOGFILE=/var/log/backups.log


##with any luck, you won't need to change anything further

TimeStamp() {
    echo $(date +"Y-%m-%d %H:%M:%S:")
}


#if the logfile doesn't exist create it
[ ! -e $BACKUP_LOGFILE ] && touch $BACKUP_LOGFILE


#see if the backup partition is mounted if not mount it
grep -qs $BACKUP_PARTITION /proc/mounts
[ $? -eq 1 ]&&{
    mount $BACKUP_PARTITION $BACKUP_DIRECTORY
#   make sure the mount works
    [ $? -eq 0 ]&&{
        echo "$(TimeStamp)  mounted $BACKUP_PARTITION on $BACKUP_DIRECTORY." >> $BACKUP_LOGFILE
    }||{
        echo "$(TimeStamp)  Failed to mount $BACKUP_PARTITION on $BACKUP_DIRECTORY." >> $BACKUP_LOGFILE
        exit 1
    }
}

#do the backup
#use any rsync recipe you want
#if you don't want compression, btw, you shouldn't be using the -z switch
#--delete is fine but i don't normally use it.  This is how I back my servers
#using the -n switch will allow you to do test runs

rsync -av $PRIMARYDIR/ $BACKUP_DIRECTORY/

#log success or errors
[ $? -eq 0 ] && {
    echo "$(TimeStamp)  $PRIMARYDIR successfully backed up." >> $BACKUP_LOGFILE
    } || {
    echo "$(TimeStamp)  Attempted $PRIMARYDIR backup.  Errors detected." >> $BACKUP_LOGFILE
}

#uncomment the next line to unmount the backup partition
#umount $BACKUP_PARTITION

Last edited by polaris96; 02-03-2016 at 09:46 PM.
 
1 members found this post helpful.
Old 02-04-2016, 10:25 PM   #23
Mikech
Member
 
Registered: Jan 2006
Location: USA
Distribution: Mint
Posts: 90

Original Poster
Rep: Reputation: 24
Words do not adequately express my gratitude polaris86! Thank you for writing the script.

I tried it and made the changes as indicated. Your directions were crystal clear and easy to follow.

However I get the following error:
Code:
mount: special device $/dev/sdd does not exist  OR
mount: special device $/dev/sdd1 does not exist
Its the only line in the log file as well. I saved the file as "Backup" and changed the permissions to executable. I ran it with and without every combination of Bash and Sudo but got the same error every time. Seems to run even if I don't call Bash first.

I ran the rsync line by itself just as a test and it works great.

What I changed:
Code:
BACKUP_PARTITION=$/dev/sdd
Although the cat command returned "/dev/sdd1", I tried it both ways: as SDD and as SDD1 and got the same error. Gparted calls it both SDD and SDD1 depending on where in the dialog box it is displayed. Needless to say I don't think its very user friendly for Gparted to do that without an explanation! But it doesn't seem to matter which one I use. Upon reflection, though I think SDD1 is the one to go with.

I also tried unmounting (but left connected) the Passport hard drive but that just produced a "not found" error. The Passport drive is normally connected (and mounts automatically) unless I need the USB port to charge my IPad (which is rare since it is useless toy).

Also I do want to do the "delete" after Habitual explained what it does. Sometimes I reorganize the "q" user directory and I want that reflected in the backup. As you say I don't want any compressions and the last backup had the "z" in it. So its running in the background right now with the "delete" without any problems. I know its writing to the Passport because the drive is drawing .52 amps from the port where it only draws less than .2 amps when idle.

So I added "delete" to:

Code:
rsync -av $PRIMARYDIR/ $BACKUP_DIRECTORY/ --delete
The backup just finisheed with the usual error which I am guessing is not a big deal:

Code:
sent 174,930,015,812 bytes  received 4,300,818 bytes  34,090,288.73 bytes/sec
total size is 278,347,138,411  speedup is 1.59
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]
I don't know if this matters or not but the USB3 port is a separate card that installed by cable to a USB3 header on the motherboard. The motherboard did not come with native USB3 ports in back, I have two of these ports and use them all the time. The operating system doesn't seem to have any problem with them.

Also when I bought the motherboard, UEFI was a new thing and setting it up on this board with SATA hard drives was a bit flaky. I only got it to work through trial and error. I should do a firmware update. Unfortunately doing motherboard updates with Linux is nontrivial! I could break its tenuous existence.

Last edited by Mikech; 02-04-2016 at 10:51 PM. Reason: update info
 
Old 02-05-2016, 07:24 AM   #24
polaris96
Member
 
Registered: Jan 2015
Distribution: Slackware, LFS, OpenIndiana, debian wheezy
Posts: 55

Rep: Reputation: Disabled
OK, no worries! your backup problem came from this:
Code:
BACKUP_PARTITION=$/dev/sdd
can you see it? it took me a minute. Unfortunately, no matter how experienced a coder you are, little things WILL slip through. Remove the $

Code:
BACKUP_PARTITION=/dev/sdd1
should fix it. Btw the script was catching the error, which is what we need in things like cron jobs that often run at odd times. the string $/* isn't a proper path or a variable so bash didn't know what it was looking for.

You were smart to use the bundled cat | awk command. As mentioned in the comments, it didn't run well when the device was unmounted. The reason is that that code looks at a file called /etc/mtab. This file contains a list of all the currently mounted partitions, their mount points, their filesystem types, and their attributes. try
Code:
~$cat /etc/mtab
and check it out. Very handy file, /etc/mtab. When you unmounted /dev/sdd1, the code couldn't find it in /etc/mtab because it wasn't there - the system erased it when you unmounted the partition. you'd need to look in in /dev, /proc/partitions, or (best) use the blkid command to find an unmounted but physically attached partition.

Remember, we need to tell the system which $PARTITION to mount. Specifying /dev/sdd indicates a whole device. In our case, it isn't specific enough.

your rsync syntax is also wrong.
Code:
rsync -av $PRIMARYDIR/ $BACKUP_DIRECTORY/ --delete
needs to be changed to
Code:
rsync -av --delete $PRIMARYDIR/ $BACKUP_DIRECTORY/
This may (or may not) fix the errors. Rsync's reference a prior run. It might be wise to wipe the destination media (rm -r *) before re-running the script.

None of the hardware issues you mention should be very relevant. One thing I WOULD do, though, is erase /dev/sdd1 and then delete the partition. Recreate and reformat the parition with a brand new FS and then re-run the script with a fully clean drive.

In fact, FIRST I'd fix the path for your backup as mentioned above. Then I'd run it and (hopefully) get a solid hit. THEN I'd erase, reformat and start with a fresh partition.

Finally, feel GOOD about all of this. You have no idea how much you're actually learning through all of this. This kind of thing seems monster only until you start "speaking the language". Pretty soon this all going to make sense



EDIT: And now I must say, "My Bad" I tested the script with different paths (mine) and, after it flew under test, I plugged your paths in. It was me that plugged in the unnecessary $. Sorry about that. Everything I wrote above is right, but you didn't make the error. I did.

Last edited by polaris96; 02-05-2016 at 07:40 AM.
 
1 members found this post helpful.
Old 02-05-2016, 02:38 PM   #25
Mikech
Member
 
Registered: Jan 2006
Location: USA
Distribution: Mint
Posts: 90

Original Poster
Rep: Reputation: 24
That's great advice! I will wipe the drive and start over. Only thing is, and I don't think it matters, I have to format it as NTFS on a windows machine (its NTFS now). The reason is that I want to access my Linux files on the Windows machine. If I let Linux format the drive, Windows wants to "fix" it and someday I am going to accidentally hit the wrong button and Microsoft is gonna "fix" me good!

I did notice the dollar sign but I thought that was a LINUX thing. In windows the $ sign is used as an environmental variable prompt so I assumed it was some sort of requirement in Linux.

I only ran the cat command with the drive mounted. So the /dev/sdd1 was the mounted response.

Thanks for the correction on command line change. The manual showed the source and destination coming first and the options following. But I have never ever gotten any command to work following the "MAN." Most of it is unfathomable (may as well be written in Chinese)` and you have to know some sort of secret coding and formatting convention. When I see examples, their formatting looks nothing like what I see in the "MAN" other than the letter options are the same.

If you want to see errors you should come and watch me code in C and python. I have never written two lines of code that did not have formatting and syntax errors!! A programmer that worked for me a few years ago could write Visual Basic code at the speed of a touch typist and the code ran the first time every time. (Data Wizards in New Mexico in case anyone wants to hire him) Needless to say I was in awe!! He had a Ph.D. in economics from Berkeley and he was worth every penny of the 90 dollars an hour we paid him.

I have tested it again after the changes and it worked great! Its still running so I don't know if I will get errors but It doesn't matter I think I know what is causing that and I will fix it by removing those application files.

Last edited by Mikech; 02-05-2016 at 07:33 PM. Reason: Update
 
Old 02-06-2016, 06:45 AM   #26
polaris96
Member
 
Registered: Jan 2015
Distribution: Slackware, LFS, OpenIndiana, debian wheezy
Posts: 55

Rep: Reputation: Disabled
Excellent.

LetMeTellYa, $NOBODY's code runs on the first try. What you weren't seeing from your former coworker was the hundreds of $GODAMMITs he went through prior to his fingers learning the script.

KEEP READING THE MANPAGES. The tumbler WILL drop and you'll start seeing it. In your defense, rsync is no manpage for the squeamish. I usually print it ( ~$man rsync > rsync.man Then open it in OO and reformat it before printing ) because I find there's too much info for reading on the screen. You can't beat paper for flipping back and forth.

Post if anything goes boink or you need more help.
 
Old 02-06-2016, 06:52 AM   #27
polaris96
Member
 
Registered: Jan 2015
Distribution: Slackware, LFS, OpenIndiana, debian wheezy
Posts: 55

Rep: Reputation: Disabled
Oh, btw, i was looking up some sparse backup script I wrote a while ago and came across -z switch. Turns out it compresses the data for TRANSMISSION across a network. It won't compress the files at the destination, but you still don't need it. It conserves bandwidth on Networks but you're doing an internal backup, so it would be superfluous.
 
1 members found this post helpful.
Old 02-06-2016, 11:50 AM   #28
Mikech
Member
 
Registered: Jan 2006
Location: USA
Distribution: Mint
Posts: 90

Original Poster
Rep: Reputation: 24
Thanks Polaris96.

I was wrong about the manpages. It says to do it exactly the way you did it. I must have seen the information somewhere else and don't remember where. This has been happening a lot. I may be ready for the nursing home.

By the the way, Rick Berg of Data Wizards, the guy I was talking about, would come to my office and add features to the database right there at my desk with me looking over his shoulder. I was not exaggerating. He literally wrote the code as fast as anyone can type and the code ran without errors immediately. I hear what you are saying though, I have never met anyone else who could do that. He is some kind of genius. I would tell him what I needed and he would think for like 2 minutes and then write the code. Amazing!

Thanks again for taking the time to help me! I cannot thank you enough. I reformatted the drive (it took 18 hours on USB3)and I am running your code again. I like the way you wrote it. I only have to change the drive label in one place. If you ever come to New Mexico let me know, I will take you out to dinner (if I am still alive).
 
  


Reply

Tags
backups


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
C program: i am getting errors when compiling this simple program batman4 Programming 8 12-11-2012 08:23 AM
A very simple drawing program to do a very simple job. stf92 Linux - Software 8 06-28-2012 03:28 PM
Newbie trying to write a simple backup script to backup a single folder Nd for school stryker759a Linux - Newbie 2 09-16-2009 08:52 AM
Simple C++ Program: Program Compiles But Won't Run (Segmentation Fault) violagirl23 Programming 3 01-09-2008 12:09 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:44 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration