LVM snapshots: How to use?
 

Hi,

I would like to use the snapshot functionality of LVM 2 for a small database server I'm currently planning the setup for. The reason why is that this would allow me to make backups without stopping the database system. As I haven't used LVM before, I would first like to know, what you think of the following two variants, and what the consequences would be, respectively.

1. LVM "striped" snapshot
There are two harddiscs in the systems. The data partitions will be mirrored using soft RAID-1. But there is probably going to be some space left on each disc. Can I use this space for LVM snapshots? This would mean that the snapshot is distributed across two physical harddiscs. Is this possible/useful?

2. LVM snapshot on an external harddisc
Could I use, say, a temporarily mounted external USB harddisc to put snapshots on? And what's the disadvantage compared to a conventional backup? Can I mount the snapshot on a different computer? What can or cannot be done with the snapshot compared to a "conventional" backup?

Any advice or explanation of advantages and disadvantages of both approaches much appreciated!

Thanks a lot in advance,

gargamel

1. Yes if you stripe your snapshot it will be spread extents among the X no of physical disks your stripe number is.

2. No, the very nature of LVM is to "liquidize" your physical storage space and use it like a pool of available space. When using mirroring or striping, then there is some deliberate control over how the physical disks are operated on (if you stripe your extents are spread over X disks in your stripe, if you mirror extents are duplicated among at least 2 disks and presented to you as a logical extent).

Snapshotting in this manner doesnt guarantee you will know where the actual data is stored - merely its in the pool. You can move data in a live environment from one physical disk to another but this is meant for migrating to higher capacity disks or removing data from a disk before it fails.

Frankly I think your solution is not a suitable design choice.

Run a mysql master>slave configuration. This is much more elegent, secure, faster and does exactly what your after doing, and - its failover proof.

http://dev.mysql.com/doc/refman/5.0/...ion-howto.html

Thanks, this is very helpful!

Regarding the database, I wasn't thinking of MySQL, but I'll check if your design proposal can be realized with the database management system of my choice, as your arguments seem to make a lot of sense, to me.

However, the new machine will have to serve more than one purpose. Therefore my question: If I choose the master slave scenario, excluding the database from LVM control, does the scenario make sense, then? Or is it still a bad idea?

Thanks once more,

gargamel

Through 2 posts you've managed to not tell us what database you are using. ;)

Databases are special cases when it comes to backup, and snapshots are not going to really work for databases. For whatever database you are using, you need to look at the specifics for how that database works and the tools that are provided for doing backups. For example, you may have to flush transactions, lock tables, dump the database with an internal utility, etc. Otherwise, you may end up with an inconsistent and corrupted database on your backup.

So you noticed it, eh? ;)
It's going to be PostgreSQL for "conventional" data, XML and geographical data (not sure, if I need that, currently). And db4objects for special applications. And HSQLDB for embedded stuff and web applications. I'll also have a look at some other open source database management systems like Derby, in the near future.

Uhh, you leave me puzzled. Actually, my understanding so far was, that LVM snapshots can be taken at any time, and then a "real" backup can be taken from the snapshot, without interrupting the database, stopping (call it whatever you like) the database...
Wrong?

gargamel

It depends entirely on the situation - it sounds like to me that your after making a "stateful" backup - whereby the the data you backup you can guarantee has not changed whilst you are running the backup.

Most databases come with some kind of backup proposal, the mysql slave solution provides a means of keeping a running copy you can read from (and do backups to) without affecting the mysql services.

Depending on your other needs, check the documentation for them - there might be a effective backup mechanism that you can use that doesnt affect service. Whilst LVM snapshots does work - it might be overkill and too much effort to administrate unless your talking about a very large amount of files that may change randomly.

LVM snapshots (or fssnap snapshots on Solaris, which I use) are not a magic bullet for databases. They do help in that they can limit the window during which you have to lock tables to just long enough to take the snapshot. But you still have to account for the behavior of the database engine, which is more complex than just saving files.

If you want an "official" take on it, here is what the Zmanda folks have to say with respect to the Zmanda Recovery Manager (ZRM) for mySQL when using LVM snapshots:

http://mysqlbackup.zmanda.com/index....ogical_Volumes

Your advice is very helpful, as it shows me that there are still some things I need to understand better in order not to shoot myself in my foot!

Actually, my original idea stems from this article (sorry, German only):
http://de.opensuse.org/SW-RAID_und_LVM_(Grundlagen)

My concept now would be about as follows.
Use LVM for all "partitions", with the exception of /boot, swap and database data partitions.
For server databases, see if they offer some internal non-interrupting backup mechanism, such as pg_dump for PostgreSQL:
http://www.postgresql.org/docs/8.3/i...ckup-dump.html
If the dump is created on a "partition" (or "volume", rather) of which a snapshot is taken later, a complete backup of the snapshot can be taken later without interrupting server operations.

One advantage of snapshots seems to be that they can be mounted r/w, so that I always could easily play back individual files just as if I were using a "conventional" file system. That makes me think that LVM snapshots would allow me to do "temporary" or "intermediate" backups (but not "incremental" backups, of course), and do full backups once or twice a week, only.

However, thinking in that direction, I wonder, what difference an LVM snapshot would make compared to:

#rsync -avp --delete source target

Am I completely off track, now?

Thanks for your helpful advice, again!

gargamel

Snapshotting allows you to recover a partition back to a particular state - or a file back to a particular state for that matter.

The way it works is it just copies a file that has been modified to the snapshotted area, thus your server will be affected by the use of snapshotting (the will do twice as many writes). You can negate this by using a 2 stripe volume for your LVM partitions and a 2 stripe volume for your snapshot.

Thanks again!

Which is exactly what I want...

A 2 stripe volume for the snapshot was what I had in mind. But why would I need a 2 stripe volume for the LVM "partitions"?

Actually, based on what I have read and what you have taught me here so patiently (thx!), I thought of partitioning for two RAID-1 devices, one for /boot, one put under LVM control for the rest (with the possible exception of /, but opinions seem to vary on this), plus a 2 stripe LV for snapshots.

Side note: Seems the authors of docs for open source stuff have learned a lot in terms of marketing. Everything sounds so easy, but then the learning curve turns out to be quite steep... Thanks for pushing me up the hill, guys, your help motivated me to keep on trying to understand this all correctly!


gargamel

...another question, related to the last post:

It seems to be possible to create snapshots outside the volume group the original logical volume (LV) belongs to. But how?

Would I just provide a fully qualified logical volume name in lvcreate, such as:

#lvcreate --size 100m --snapshot --name /dev/vg01_snapshots/snap /dev/vg00/lvol1

(instead of the usual #lvcreate --size 100m --snapshot --name snap /dev/vg00/lvol1).

This would allow to keep snapshots separate from the original logical volumes. But again, if so, how would I do it?

Thanks a lot again for your patience!


BTW, I found a couple of good articles for LVM noobs like me:
http://www.howtoforge.com/linux_lvm
http://www.howtoforge.com/linux_lvm_snapshots_p2


gargamel

To answer your question, the reason why you would need to double your write striping your physical volume along with your snapshotted one is because 1 write will write to the volume and 1 write will go to the snapshot.

This means your snapshot works at half speed and your Log Vol works at half speed. So you would be required to stripe both partitions.

AS for snapshots, I dont think you can keep your snapshots in another volume group.
Just add a physical disk to the existing volume group.

Thanks, again, you really helped me a lot!

Regarding the adding of a physical disc to a volume group: Would this work with an external USB harddrive, too?

Even if this should not be the case, your kind support led me to a very simple, yet flexible setup:

My two harddrives are organized as two RAID-1 devices, one for /boot, the second one under control of LVM for the rest. For the moment being, I'll do my backups using the same tools as ever, but have a look at the more sophisticated backup solutions with LVM support. Regarding snapshots, I'll do some experiments to find out, how useful they actually are, and if they can be created on external drivers. For databases I'll use the tools that come with the database server software.

This should almost as simple to use and administer as a conventional partitioning scheme, with the added benefit of the flexible resizing of logical volumes.

Thanks a lot for your patience, much appreciated!

gargamel