LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-01-2017, 12:43 AM   #1
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,237
Blog Entries: 10

Rep: Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160
Backing Up a Live System?


I am in the process of implementing a backup scheme on my debian oldstable (jessie) server.

I decided to use borgbackup. it's deduplicating.

I want to backup both system and data partitions to a separate backup partition, all on one internal drive.

First time I tried to backup the system, borg balked, but then I adapted the command:
Code:
borg create --stats --progress --exclude-caches --one-file-system -C zlib \
-e /proc -e /tmp -e /run -e /sys -e /media -e /dev root-system.bup::$(date +%Y%m%d%H%M) /
(taken from the docs, mostly)
(-e means exclude)
and now it does not balk.
I am confident I would be able to restore a broken system with this backup data. Am I wrong to be confident about that?

And what are the implications of backing up a live system?
 
Old 09-01-2017, 03:05 PM   #2
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,836

Rep: Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370
"implications of backing up a live system" Well, the confidence that your backup was able to correctly capture the live state and resolve concurrency. The type of data would determine that level I'd think. Unless a product is marketed as a "live state" backup then we can't really assume it will be. Some file systems offer this as a byproduct of how they work I think. Pretty sure I read that even MS has some issues on backing up live state databases. They may have that fixed now.

There was one linux open sourced live state backup but I forget the name. There is at least one commercial live state backup offered.

From the borg site. "check your backup"
 
Old 09-01-2017, 06:14 PM   #3
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,285

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
If I had need for live backups of something like a very active database, I would look into LVM snapshots.

I would not encrypt your backups unless you have a compelling reason to do so (like if you're storing them off-site at a friend's house).

I would not use software that uses custom compression algorithms.

Borg does use encryption, but I do not know if that can be turned off. I wouldn't want it, because then you'd need Borg to decrypt things later. And you'd also be depending on backup software developers (which I assume the Borg community is) ALSO being adept encryption developers, which is probably not the case. My guess is that encryption is more of an afterthought (not trying to insult any developers here, just stating what I assume to be true). I don't know about the compression algorithms used by Borg.

I also would want (and currently use) backup software that does not require itself to be present for restore. I want to be able to manually look at my backups, read them, see the files and directory structure, and assure myself that they are fine without needing any special software to do so. I also want the ability to restore from backup using standard Linux command line tools. Normally I would use the backup software if that seemed appropriate at restore time, but I'd want the option to get things back manually for the worst case scenario.

My choice for backup software was UrBackup. It meets my above goals. At it's heart, I think it is probably just rsync. But with a really nice front end (that you could do without during restore if you absolutely had to). And if UrBackup support goes belly up sometime in the future I am not worried. Because I can access and restore my stuff using standard command line (probably only rsync would be needed) and simple scripts if I have to. I do not know much about Borg. I'm just presenting my personal requirements for your consideration. I do not have any active, live databases that I backup up however. If I did, I would use the LVM snapshots that I mentioned initially. And if I needed encryption, I would investigate using a stand-alone encryption offering that is well known and well vetted as being secure and stable. My feelings on the "do everything" software is that it's a jack of all trades, master of none. That may be an overly conservative viewpoint, but that's the way I am.
 
Old 09-01-2017, 07:19 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,700

Rep: Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548
As above.
I use snapshot (btrfs) all the time - it initiates fsfreeze (what jefro was referring to maybe), as does LVM - so most things will be ok. But in my case I only snap filesystems that are quiesed (data only, not system). Databases still have to be separately flushed and locked IIRC, but downtime is minor.

Short answer, don't trust live backups unless you can do transaction recovery. Logs you may not care about (I don't generally), but if you have auditors, they may have a different opinion.
 
Old 09-02-2017, 02:23 AM   #5
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,237

Original Poster
Blog Entries: 10

Rep: Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160
thanks for the answers.
"databases" are mentioned on one hand, on the other hand nobody's saying that backing up a live system is a big no-no.
only my media streaming server uses a database afaik, but i don't mind if that is lost - at the moment.
i will keep it in mind however.

all this is not for work, so no audits.

and no, i'm not using btrfs or LVM. it's all ext4.

important points about not using encryption and being able to look at the backup.
right now the backup is not encrypted, but it's on an encrypted partition. i might need to change that then?
and i will look at borgbackup's documentation, if it's possible to access the backups without borgbackup.
though it seems to be stable and widespread software.

for lack of storage space, deduplication is very important to me.

Quote:
Originally Posted by jefro View Post
From the borg site.
sorry, i didn't get that.
what is from the borg site? and where?
 
Old 09-03-2017, 08:44 AM   #6
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,237

Original Poster
Blog Entries: 10

Rep: Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160
I take lack of further comments to mean that it'snot wrong to back up a live system the way i do.

i wrote a little article about borg, click on my blog link.

Quote:
Originally Posted by ondoho View Post
sorry, i didn't get that.
what is from the borg site? and where?
ah, it's right there on the borg website.
'borg check' is also a command.
 
Old 09-05-2017, 04:23 PM   #7
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,836

Rep: Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370
You can back up a live state. There are some free tools that people use. There is at least one commercial product. Do the free tools always work??? No. I've never tested the commercial products so I can't say.

I make the statement of confidence. Even the best backup has to be tested to prove. For years windows nt had you backup at night and restore at morning to prove the backup worked.

If you validate that your system is fully working by some means then that is all that matters. Folks use rsync many time. However in most cases a full OS live state backup isn't what is common. Common is data back up and one suspects they can restore the OS by media.

Last edited by jefro; 09-07-2017 at 02:35 PM.
 
Old 09-07-2017, 12:00 PM   #8
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,237

Original Poster
Blog Entries: 10

Rep: Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160
Quote:
Originally Posted by jefro View Post
You can back up a live state. There are some free tools that people use.
which?
honestly haven't found anything that explicitely states it.
borg doesn't deny it at least, but i had to exclude folders as per post #1.

Quote:
Even the best backup has to be tested to prove. For years windows nt had you backup at night and restore at morning to prove the backup worked.
you mean fully trust the program to let it restore everything automatically?
would that even work on a live system?
all in all, i think my laziness again prevails here - just to think about the troubles i'd have to go to to
  1. make a full backup of the system partition (not live this time)
  2. deliberately create some breakage on the live system
  3. see if the backup fixes it
shudder...



Quote:
Folks use rsync many time.
but it's not deduplicating?
Quote:
However in most cases a full OS live state backup isn't what is common. Common is data back up and one suspects they can restore the OS by media.
yes, i gathered that much.
but although my systems are not mission critical, i wanted the full backup because often i found that some important files are just outside the backed up data.
 
Old 09-07-2017, 02:42 PM   #9
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,836

Rep: Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370Reputation: 3370
Not sure how you can create a backup and not have duplicate so I don't get the deduplicate part.

I doubt anyone can make a choice for you. There are maybe 20 or more common ways to create some sort of backup. None of them are much use if you need to recover and find the backup scheme has failed.

If you need to have live state then maybe moving to LVM, Btrfs, or ZFS may assist in this.

Kind of boils down to how to lock a state into the filesystem and then capture it. Linux ext filesystem don't seem to have that ability like NTFS.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Backing up Live Persistent USB - Where is my data? eco_bach Linux - Newbie 3 01-03-2017 05:11 PM
[SOLVED] Making sure I'm backing up QEMU/KVM image correctly (when not live) DJOtaku Linux - Virtualization and Cloud 4 08-20-2015 04:14 PM
Backing up a "live system" using tar ( followed by regular incremental backups) ? uncle-c Linux - Newbie 6 01-22-2010 07:13 PM
Backing up your system? alime Ubuntu 2 05-18-2005 09:06 PM
Backing up the whole system... koyi Linux - General 12 10-07-2003 06:22 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:54 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration