Thanks, enorbet and jamison20000e. Turns out it's actually Control -> Configure Dolphin -> General -> Previews and then the checkbox for "video files". At least, it works with Dolphin from the KDE 4.12.5 packages (from Alien Bob) that I'm using.
I just tested Thunar, and video thumbnails for video files (WMVs, MP4s, FLVs) are already working.
I guess there was nothing to set up.
Last edited by dugan; 06-07-2014 at 03:16 AM.