Run some system monitor when you try test speed tests. I'd be interested in at least cpu, hard drive, and ram tasks.

While I don't feel that most modern systems will run equally fast with ntfs and xfs I'd think they'd be fairly close. If you formatted this drive with each format and the tests are quite different then I guess it is possible that the controller on your system or this disk and it's driver are having some issue. Time to start swapping. Get some old disk and try it. Look for cache settings or even run smart tools for clues.

Put this disk in a different system and re-test.

VMWare itself was causing the I/O spikes. From an article on their website, I added the following config to my virtual machine's .vmx config:
After making these changes the all VMs worked fine again.

So, thank you all for your support.

Thanks for the solution and update.