Dear friends,

I'm having some thoughts regarding to shared storage for Virtualisation.

I have spend few weeks to test out NAS4FRee high available iSCSI storage on VMware and I just figured out the whole idea is wrong.

If the VMware's iSCSI attached storage dies with any reasons, then
you WILL lose the connection to the storage.

No matter if you have a HAST(https://wiki.freebsd.org/HAST]) iSCSI storage and the second node will take over, you still will lose the running machines on the storage.

I have tested it and about 50% I always lost the whole box and needed manual intervention. So the high available storage didn't worked.

I reconfigured the FreeBSD(NAS4Free) with UFS file system instead of ZFS and I used NFS storage and I got nearly the same speed on the VMware machines then with iSCSI storage.

But NAS4Free does not support UFS file replication, it needs to be ZFS.

So I went back to the good "old" Linux and I need an appropriate and free solution for High Available NFS server.

I had a look Linbit(http://www.linbit.com/en/) DRBD with Pacemaker and Corosync, but all documentations have loads of gaps. (I guess to pay for the support)

I could not find CentOS, but I find RedHat.

Anyhow I found a quiet good docs for Suse HA-NFS server, but it's the same as RedHat, no support and not sure if that's good with out updates.

So tomorrow I'll try the Opensuse as the main system and follow this doc: https://www.suse.com/documentation/s...echguides.html

Is there any concerns regarding to this setup?

Or do you guys have any better idea how to get 100% uptime on an iSCSI or NFS share for VMware?
iSCSI is better/quicker than NFS, but it kills the VMware kernel.

Nowdays this is a must have in any "basic" setup, but as I can see, the industrial solutions for this is quiet expensive and they basically use ported Linux or FreeBSD. Just for example we have few Buffalo NAS servers and most of them ported Debian with ARM based hardware.

Please help me out, any ideas much much appreciated!

I already spend a hell of a lot of time with this and I need to get it sorted in the following month at my workplace.

Thanks in advance.

1. 100% uptime is a myth, you have to deal with the 99.N number of nines.
2. Get a proper enterprise grade storage with redundant controllers and replication, and you will not need to reinvent the wheel.

Vmware states this in one of their papers.

"iSCSI was considered a technology that did not work well over most shared wide-area networks. It has prevalently been approached as a local area network technology. However, this is changing. For synchronous replication writes (in the case of high availability) or remote data writes, iSCSI might not be a good fit. Latency introductions bring greater delays to data transfers and might impact application performance. Asynchronous
replication, which is not dependent upon latency sensitivity, makes iSCSI an ideal solution. For example, VMware vCenter™ Site Recovery Manager™ may build upon iSCSI asynchronous storage replication for simple,
reliable site disaster protection."

www.vmware.com/files/pdf/iSCSI_design_deploy.pdf

I do have a few systems running Suse in HA and they have done well over the years using local raid.

1 members found this post helpful.

Thanks guys for the advice, appreciated.
The proper enterprise solution with replication cost at least 5k.+ which is not really an option unfortunately.

I'll try to give it a go to the SUSE-HA solution, let's see how it goes.

I used to love/deal with Novel Netware(s) and I know their solutions.
Also ages ago I used to install couple of SUSE, but because of the licensing(and wasn't that popular) I used Fedora/CentOS/Debian instead.

Anyhow as I saw the docs, the Pacemaker/Corosync looks to be a promising solution, but I'm afraid I won't have enough time to deal with this. So firstly I guess I'll use a "regular" NAS and then I'll migrate the machines from the basic storage to the HA storage.

Also all docs lacks with few steps, which could be vital in every configuration.
For example the DRBD HA-NFS server with RedHat uses different volume configurations with LVM than the regular RPM based configs.
One of them has the LVM in the DRBD and the other has not. But both of them RPM based distro.
And these are on LinBit(DRBD) site as a full docs! Well clearly just like other Companies they want to sell there own support I guess.
Also I saw exactly the same articles, copied from SUSE site as how to configure the corosync/pacemaker with HA-NFS.
You can recognize the same docs after a while, when you research in this.

The NAS4Free works well with iSCSI, but as I mentioned it can kill the VMware.

The VMware replication and etc solutions are not an option either, because licensing cost a fortune.
For that much we could buy a proper replicated storage, so no point.

So old school solutions: build something good/reliable from cheap/free solutions which usually ends up with a big and terrible disaster.

if you read this carefully, you will see that this has nothing to do with iSCSI. All they are saying is thatthey will support iSCSI arrays that can do async replication. the replication as such doesn't work over iSCSI, that's something that is done by some proprietary protocol or things like FCIP. You can even make it happen with DRBD if you want to get risky, or using ZFS replication.

I think I could not explain myself properly to you guys or I would not get it fully either. Forgive me English is not my first language...

Anyhow, as I said the iSCSI is not a problem neither the replication.
NAS4Free and even FreeNAS works fine with this.

I setup the NAS4Free with ZFS HAST device and the failover worked properly with Windows servers. Also sometimes worked with VMware, but as I said the VMware could die about 50% and could not mount back the attached drives which have the running vmachines. VMware does this automatically as Windows servers. The different is that between the too, VMware can die Windows not that sensitive. On Windows I got back always the attached device with the data on it.

This is the article at VMware: http://kb.vmware.com/selfservice/mic...rnalId=1003681

Also I tested the NAS4FREE NFS failover solution and it is fine, but horrible slow. I got about 4-5Mb speed on the share, because of the ZFS.
Tested it with UFS file system and I got 50-70Mb speed. But obviously no replication with UFS whatsoever.
The NFS mounted share on VMware does not die at all, and does not kill the box.
I even noticed some caching time, when I killed the service the machine worked for 15-20 sec more and then VMware lost it, but the system didn't die. With iSCSI you could even lose the vSphere client, sometimes the server console too.

When people advise that use iSCSI on VMware instead of NFS, then I don't think they know that this could kill all the running machines including the whole box. And all VMware affected by this "issue" as you can see.

This is not about NFS or iSCSI, this is about the fact that VM storage workloads are sensitive to outages, and any HA setup will have a delay while the falover occurs. KVM/QEMU has a feature that willpause the VM on disk error, so the VM doesn't crash, and once storage is restored, you can simply unpause and continue, vmware have been talking about doing the same thing back in vmworld 2012, but I'm not sure it's been implemented.

1 members found this post helpful.

Hm good to know that, thanks!
I have not played with KVM in the last year to be honest, and never used it in production either, just only for testing/study purposes. I managed a bit Citrix XEN servers at the data-center I used to work for, but not much especially not HA.
Anyway I'll give it a go to check it out with NAS4Free/iSCSI combo to see how it works.

Hi Guys,

Some updates regarding to my issue.
Finally we went for Synology storage(https://www.synology.com/en-us/produ...erview/DS1813+) and I must say this is a piece of art.

The HA cluster works fine, the switch over takes about 15-20 seconds all in all.
I've tested it with running VMware ESXi machines installed onto mounted storage and the machines managed to stay up.
Both iSCSI and NFS were fine, seems like the issue just lies with Nas4Free HAST storage.

I'm pleased with this solution, it cost only ~ £700 each node. + disks (WD Red NAS disks)
So we ended up with less than £2k. for the whole solution, both nodes in RAID-10.
Must say it's pretty cheap considering storage prices.

Thanks again for the help!