I'm looking for ideas on how to accomplish something equivalent to a typical SAN / VMware setup, where you have a SAN and multiple hosts, that can work off the same LUNs. Basically iSCSI with some kind of cluster aware file system.

What's everyone in the Linux world typically use?

My current setup is using NFS with VMware. I want to use KVM/Qemu eventually, but it requires tons of documentation reading etc... I just wanted something turn key so I can be up and running so I used Vmware, but I do eventually want to switch off it.

Downside to my current setup is the extra overhead of having NFS, instead of block storage. I find it just lacks the performance, and permissions can be a pain to manage, when really I don't care about permissions, I just want files to be accessible, and the way NFS handles security is quite laughable anyway (user/group IDs, really? It does not even care about the password)

My file server is currently more treated as a NAS than a SAN, but I want to treat it as a SAN, and do mostly block storage. For stuff like actual file sharing, that would be done via SMB/NFS on each respective VM.

In fact I'd like to perhaps take it a step further and make it so the VM hosts don't necessarily have LUNs, but instead each VM has it's own LUN(s) that is directly accessed as a hard drive, is that possible with something like KVM/Qemu?

As far as networking goes, ability to hot add/change Vlans like in vmware is also a must. When I tried KVM I think luns required to modify networking related files, which requires restarting that service, which is a big no no if I have production stuff going on as that can cause all sorts of corruptions.


So with that said, what does everyone use in the open source world and how is it setup? I eventually want multiple hosts and ability to move VMs across and what not.

On the storage end, I'm using mdraid. If I go the SAN route, the storage server would be on it's own separate network and the hosts would have their own NICs. Probably want to use teamed networking as well.

Is there not anything? I'm not necessarily looking for a turn key solution (although that would be nice) but just some direction on what set of programs would accomplish what I want so I can start reading. Once I get my hands on some hardware that has VT-D to test with I can start experimenting. Basically I'm looking to know what everyone else's storage/vm infrastructure is like and what is the goto way of doing it.

You can check out Proxmox - its really nice mear-turnkey solution for virtual machines (uses qemu/kvm for the actual vms) with clustering infrastructure and really nice web gui for administration.

Wait, it looks like lightworks is not free. Ok scratch that. Any other options?