LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-20-2014, 05:43 AM   #16
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,137
Blog Entries: 1

Rep: Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874Reputation: 874

How long it takes depends on the system. I just made a youtube video doing this process and it's video length is under twenty minutes. The system cloned was a linux distro iso image. The source iso was < 1GB and the uncompressed system was 3.1GB. From pressing record to booted into the distro was under forty minutes. The extra twenty minutes was figuring out a format to record the new distro in which lacked libx264. That was just a basic desktop with low cruft. It doesn't have to take hours, but it could if you don't take steps to ease the burden of backing up. All those movie clips from your camcorder in /home. All those packages downloaded and updated, but the .deb or .rpm files just sitting there. That mostly unused and hefty /usr/share/doc/ that many distros do not segment out as optional when installing packages. Yeah it could take hours, but it doesn't have to. And an unfragmented filesystem has it's perks.
 
1 members found this post helpful.
Old 03-20-2014, 01:42 PM   #17
Gregg Bell
Senior Member
 
Registered: Mar 2014
Location: Illinois
Distribution: Xubuntu
Posts: 2,034

Original Poster
Rep: Reputation: 176Reputation: 176
thanks guys. It's going to take a while to sort through all this. Things I'm currently unfamiliar with (that you mentioned):
"testing any backup w/a restore" "config files" "fragmentation" and lastly "fun."
 
Old 03-20-2014, 02:16 PM   #18
Ryanms3030
Member
 
Registered: Jan 2014
Distribution: Debian, Mint, CentOS, Ubuntu
Posts: 261

Rep: Reputation: Disabled
As much as I like to control my own files and stuff , I tend to back up my really really important files on both my Dropbox and Google Drive. I don't use enough space to have to pay for either and I figure they both have much more redundancy then I can afford to have locally
 
1 members found this post helpful.
Old 03-20-2014, 04:49 PM   #19
Gregg Bell
Senior Member
 
Registered: Mar 2014
Location: Illinois
Distribution: Xubuntu
Posts: 2,034

Original Poster
Rep: Reputation: 176Reputation: 176
Quote:
Originally Posted by Ryanms3030 View Post
As much as I like to control my own files and stuff , I tend to back up my really really important files on both my Dropbox and Google Drive. I don't use enough space to have to pay for either and I figure they both have much more redundancy then I can afford to have locally
Thanks Ryanms, Well, that seems super-simple, which I like, but when I tried to download my "gregory" folder (the one below "home" and containing Documents, Downloads,Pictures etc.) Drive said there was an error and wouldn't upload it. But I was able to upload my Documents folder, which is a start! And I, like you, would never use anywhere near all the cloud storage out there. So if my computer crashes (I'm a writer) and I've got my novels safe in the cloud and on a flash drive, do I really need to do the automated sort of backup that is available (and much more complicated)? The way I see it, if I have my Documents folder and the computer crashes all I have to do is reinstall Ubuntu 13.10, which is a breeze. Is it as easy as that or am I missing something?
 
Old 03-20-2014, 05:04 PM   #20
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 789

Rep: Reputation: Disabled
If you want to backup your configuration, consider http://devstructure.com/blueprint/
 
1 members found this post helpful.
Old 03-20-2014, 05:10 PM   #21
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
Quote:
Originally Posted by AwesomeMachine View Post
rsync takes hours.
Only the first time. I back up my home server nightly to two external USB drives using rsync. All in all it's about 3.3TB that's being backed up, and rsync typically takes about 10 minutes to run (most of that time is just generating the file list and then comparing the current version of each file to the backed up version to check for differences).

OP - using rsync is really very simple. You can back up your entire system to an external drive with one command. The first time it will take forever (any backup will), but after that one is finished, new backups are very fast since they just transfer what has been changed since the last one. If you stick this one line command in a text file (which makes it a script), you can then add it to your cron (like Windows' "Scheduled Tasks") so that it runs automatically on a regular schedule. This is what my backup script looks like:

Code:
#!/bin/bash

if [[ ! -e /home/verify ]]; then
   echo "raid not found, skipping backup"
   exit 1
fi

if [[ ! -e /media/backups/home/verify ]]; then
   echo "primary backup drive not found, skipping"
else
   /usr/bin/rsync -aAXv --progress --delete /* /media/backups/ --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/.gvfs,/var/lib/ntp/proc/*}
fi

echo ""
echo ""
echo "Backup finished $(date)"
The first two "ifs" just check to make sure the local RAID is present and the backup drive is present before it continues, then the rsync command is what does the backup. The above code is just saved to a text file called "backup", which is made executable with "chmod +x backup", and then added to the cron. The crontab entry looks like:
Code:
30 4 * * * /root/bin/backup > /root/backup.out 2>&1
Which runs it at 4:30am every day, logging the output to the backup.out file so I can inspect it later, if I want.

All in all it took about 2 minutes to set up, and now it runs forever without any user intervention, and it didn't require installing or setting up any software (any Linux distro should have rsync installed by default). You shouldn't just blindly implement what I've done here without understanding what it's doing, but it could give you a baseline to work off of.

I am very much against using the anonymous "cloud" for storage or backups. These are servers run by unknown companies in unknown locations who have unknown agendas. Who knows what they're doing with your data or if they'll even be around tomorrow or the next day. There have been countless cases of companies setting up cloud storage for people to either store data on or contribute to, then one day the company decides that everything that has been uploaded is now theirs to do with what they wish, they lock out their users, and then start charging for access. Other companies set up systems like this and then go out of business and the entire network disappears overnight. Other companies one day decide "meh, we're done" and shut down your access with no warning, permanently. Do you really want to base your entire backup system on the whims of some exec who's answering to shareholders and could decide at any moment to lock you out, delete your data, sell your data, or shut down?

Screw that.

You can buy an external hard drive for cheap, and set up a ONE LINE script to do regular backups, and never have to worry about it again (at least until the external hard drive fails some years down the road and you replace it).

Last edited by suicidaleggroll; 03-20-2014 at 05:21 PM.
 
1 members found this post helpful.
Old 03-20-2014, 05:16 PM   #22
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,623

Rep: Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651
the easiest way

buy a tape drive back-up solution
 
1 members found this post helpful.
Old 03-20-2014, 05:22 PM   #23
Gregg Bell
Senior Member
 
Registered: Mar 2014
Location: Illinois
Distribution: Xubuntu
Posts: 2,034

Original Poster
Rep: Reputation: 176Reputation: 176
Quote:
Originally Posted by NotionCommotion View Post
If you want to backup your configuration, consider http://devstructure.com/blueprint/
Thanks Notion. Appreciate it.
 
Old 03-20-2014, 05:26 PM   #24
Ryanms3030
Member
 
Registered: Jan 2014
Distribution: Debian, Mint, CentOS, Ubuntu
Posts: 261

Rep: Reputation: Disabled
Quote:
Originally Posted by Gregg Bell View Post
Thanks Ryanms, Well, that seems super-simple, which I like, but when I tried to download my "gregory" folder (the one below "home" and containing Documents, Downloads,Pictures etc.) Drive said there was an error and wouldn't upload it. But I was able to upload my Documents folder, which is a start! And I, like you, would never use anywhere near all the cloud storage out there. So if my computer crashes (I'm a writer) and I've got my novels safe in the cloud and on a flash drive, do I really need to do the automated sort of backup that is available (and much more complicated)? The way I see it, if I have my Documents folder and the computer crashes all I have to do is reinstall Ubuntu 13.10, which is a breeze. Is it as easy as that or am I missing something?
I also do some writing and that is the bulk of what I back up. That and photos I don't want to lose. My main concern is security of my personal documents on a cloud servers but 1.) I figure Google and Dropbox are a lot better at security than I am since they are multibillion dollar businesses whose entire business involves them being able to secure their and my data 2.)Nobody wants to hack in and steal any of my writing

I like being able to wipe my entire operating system and computer and still be able to access my important stuff and also like to be able to access it from different computers with internet connections
 
1 members found this post helpful.
Old 03-20-2014, 05:28 PM   #25
jamison20000e
Senior Member
 
Registered: Nov 2005
Location: ...uncanny valley... infinity\1975; (randomly born:) Milwaukee, WI, US( + travel,) Earth&Mars (I wish,) END BORDER$!◣◢┌∩┐ Fe26-E,e...
Distribution: any GPL that work on freest-HW; has been KDE, CLI, Novena-SBC but open.. http://goo.gl/NqgqJx &c ;-)
Posts: 4,888
Blog Entries: 2

Rep: Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567Reputation: 1567
Quote:
Originally Posted by Gregg Bell View Post
... my "gregory" folder (the one below "home" and containing Documents, Downloads,Pictures etc.) Drive said there was an error ...
Could always compress the folder see if that goes although I don't think you will have access to the individual files unless the cloud service has the capability to look\take\add in .tar.gz or other .zip like formats? If so could store more for free.

Edit\add: google drive is grate I also email myself important files that sit in folders for years they give lots of storage there too could have lots of email accounts as well...

Last edited by jamison20000e; 03-20-2014 at 05:35 PM. Reason: +more
 
1 members found this post helpful.
Old 03-20-2014, 05:35 PM   #26
Gregg Bell
Senior Member
 
Registered: Mar 2014
Location: Illinois
Distribution: Xubuntu
Posts: 2,034

Original Poster
Rep: Reputation: 176Reputation: 176
Quote:
Originally Posted by suicidaleggroll View Post
Only the first time. I back up my home server nightly to two external USB drives using rsync. All in all it's about 3.3TB that's being backed up, and rsync typically takes about 10 minutes to run (most of that time is just generating the file list and then comparing the current version of each file to the backed up version to check for differences).

OP - using rsync is really very simple. You can back up your entire system to an external drive with one command. The first time it will take forever (any backup will), but after that one is finished, new backups are very fast since they just transfer what has been changed since the last one. If you stick this one line command in a text file (which makes it a script), you can then add it to your cron (like Windows' "Scheduled Tasks") so that it runs automatically on a regular schedule. This is what my backup script looks like:

Code:
#!/bin/bash

if [[ ! -e /home/verify ]]; then
   echo "raid not found, skipping backup"
   exit 1
fi

if [[ ! -e /media/backups/home/verify ]]; then
   echo "primary backup drive not found, skipping"
else
   /usr/bin/rsync -aAXv --progress --delete /* /media/backups/ --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/.gvfs,/var/lib/ntp/proc/*}
fi

echo ""
echo ""
echo "Backup finished $(date)"
The first two "ifs" just check to make sure the local RAID is present and the backup drive is present before it continues, then the rsync command is what does the backup. The above code is just saved to a text file called "backup", which is made executable with "chmod +x backup", and then added to the cron. The crontab entry looks like:
Code:
30 4 * * * /root/bin/backup > /root/backup.out 2>&1
Which runs it at 4:30am every day, logging the output to the backup.out file so I can inspect it later, if I want.

All in all it took about 2 minutes to set up, and now it runs forever without any user intervention, and it didn't require installing or setting up any software (any Linux distro should have rsync installed by default). You shouldn't just blindly implement what I've done here without understanding what it's doing, but it could give you a baseline to work off of.

I am very much against using the anonymous "cloud" for storage or backups. These are servers run by unknown companies in unknown locations who have unknown agendas. Who knows what they're doing with your data or if they'll even be around tomorrow or the next day. There have been countless cases of companies setting up cloud storage for people to either store data on or contribute to, then one day the company decides that everything that has been uploaded is now theirs to do with what they wish, they lock out their users, and then start charging for access. Other companies set up systems like this and then go out of business and the entire network disappears overnight. Other companies one day decide "meh, we're done" and shut down your access with no warning, permanently. Do you really want to base your entire backup system on the whims of some exec who's answering to shareholders and could decide at any moment to lock you out, delete your data, sell your data, or shut down?

Screw that.

You can buy an external hard drive for cheap, and set up a ONE LINE script to do regular backups, and never have to worry about it again (at least until the external hard drive fails some years down the road and you replace it).
Thanks for the calm, considered explanation of how this all works. And the relatively maintenance-free running sounds fantastic. I will of course have to formulate my own backup command and buy an external hard drive. Which might take a while to do, but one of these days. And I agree with you about the vulnerability of cloud storage. I tend to think of my stuff as not being earth shatteringly valuable. But yeah, I wouldn't like any of those scenarios you suggested. I have two computers I regularly use, but I suppose I would only need to backup one of them. (One of them seems to have most of what is important to me on it.) And I see Newegg is always having external hard drives on sale. One thing I'm not entirely convinced of though, is if I'm really mostly concerned about my novels, which would be easily saved on a flash drive, do I really need to back anything else up? Couldn't it just as simple as saving the novels, and then if the computer crashes reinstalling Xubuntu 13.10 and the software I was using?
 
Old 03-20-2014, 05:50 PM   #27
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
Quote:
Originally Posted by Gregg Bell View Post
Thanks for the calm, considered explanation of how this all works. And the relatively maintenance-free running sounds fantastic. I will of course have to formulate my own backup command and buy an external hard drive. Which might take a while to do, but one of these days. And I agree with you about the vulnerability of cloud storage. I tend to think of my stuff as not being earth shatteringly valuable. But yeah, I wouldn't like any of those scenarios you suggested. I have two computers I regularly use, but I suppose I would only need to backup one of them. (One of them seems to have most of what is important to me on it.) And I see Newegg is always having external hard drives on sale. One thing I'm not entirely convinced of though, is if I'm really mostly concerned about my novels, which would be easily saved on a flash drive, do I really need to back anything else up? Couldn't it just as simple as saving the novels, and then if the computer crashes reinstalling Xubuntu 13.10 and the software I was using?
Sure, there's no reason you have to back up everything. In the rsync command, just change the "/*" to "/home/greggbell" or whatever directory you want to back up (and get rid of the unnecessary excludes in the process).

You could also have your second computer back up to the first, and then the first backs up to the external drive. I didn't mention it in my post, but that 3.3TB I'm backing up includes the regular backups from 3 other computers, one Linux, one Windows, and one OSX. So the external hard drive actually holds the backups from four machines, plus two iPhones and an iPad, all of which is being regularly synced to the external drive using that one script I posted earlier (plus similar scripts or settings on the other machines to back them up to the server).

The overall layout is:
portable devices back up to machine A
machines A-C back up to the server (machine D)
machine D backs up to the external drive using the script I posted earlier

So everybody has a backup. Some have two, and the portable devices actually have three.

Last edited by suicidaleggroll; 03-20-2014 at 05:55 PM.
 
1 members found this post helpful.
Old 03-20-2014, 05:51 PM   #28
Gregg Bell
Senior Member
 
Registered: Mar 2014
Location: Illinois
Distribution: Xubuntu
Posts: 2,034

Original Poster
Rep: Reputation: 176Reputation: 176
Quote:
Originally Posted by John VV View Post
the easiest way

buy a tape drive back-up solution
Thanks John. I looked at them at Newegg. They look great but a little pricey. Lowest one I saw around $500.
 
Old 03-20-2014, 05:52 PM   #29
NotionCommotion
Member
 
Registered: Aug 2012
Posts: 789

Rep: Reputation: Disabled
Quote:
Originally Posted by Gregg Bell View Post
Thanks Notion. Appreciate it.
Given my current experience, I don't have much to say, and have to take the opportunity if I ever do
 
1 members found this post helpful.
Old 03-20-2014, 05:59 PM   #30
Gregg Bell
Senior Member
 
Registered: Mar 2014
Location: Illinois
Distribution: Xubuntu
Posts: 2,034

Original Poster
Rep: Reputation: 176Reputation: 176
Quote:
Originally Posted by Ryanms3030 View Post
I also do some writing and that is the bulk of what I back up. That and photos I don't want to lose. My main concern is security of my personal documents on a cloud servers but 1.) I figure Google and Dropbox are a lot better at security than I am since they are multibillion dollar businesses whose entire business involves them being able to secure their and my data 2.)Nobody wants to hack in and steal any of my writing

I like being able to wipe my entire operating system and computer and still be able to access my important stuff and also like to be able to access it from different computers with internet connections
Thanks Ryan, Would you clear up some of your terminology. What (in your second to last post) did you mean by: Drive and Dropbox "both having much more redundancy then I can afford to have locally." And I hear you about the convenience of accessing the cloud storage from anywhere. (I still think keeping stuff on a flash drive or two is a good idea as well.) And what did you mean by "I like being able to wipe my entire operating system and computer"? "Wipe"? Why? How? Thanks.
 
  


Reply

Tags
back-up, home directory


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Can any one suggest a simple OS to run as guest OS balaqemu Linux - Newbie 8 09-22-2011 12:02 PM
Newbie trying to write a simple backup script to backup a single folder Nd for school stryker759a Linux - Newbie 2 09-16-2009 08:52 AM
Backup methodology: what do you suggest? lancherider Linux - General 2 07-01-2006 03:58 PM
Please suggest a simple, Easy-to-use blog vharishankar General 6 05-11-2005 02:45 AM
Real Simple Question neo.sarcastic Linux - Newbie 1 08-21-2003 10:23 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:31 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration