Thanks, I got vdpau working on VLC under LMDE. It does play a bit more smoothly, but I've discovered that the underlying cause seems to be in the video files. I've played a few on my gaming box (Windows 8.1) and the problems seem to be there still, except somewhat less pronounced, especially in regard to the vsync-like tearing.

Maybe the screen itself does some smoothing, I don't know - at this point I've sort of given up understanding.

I might yet try a Windows install, but I'll do it on another hard disk so if it turns out it doesn't improve things I can just yank it and keep LMDE configured as it now is.

Thanks for the help; although I'm no longer sure it was a Linux problem to begin with, at least I now know about vdpau and how to set it up. This will be useful in the future.

I forgot to mention that you have to set vdpau in 2 places - one, the decoding part is at the codecs section and the other (the presentation) at the video tab..
Also note that hardware decoding is available for h264 and some other less widely used codecs (install and run vdpauinfo). But even if decoding is not available for some reason, the vdpau video output should be still tear free.

For HTPC you do not need a desktop environment, for instance I have a ZOTAC ION with 2 GB of RAM here, I have configured it to autologin and auto-run X, X is configured to run Kodi directly on it. No DE, no WM.

Edit: