Got it working. Unmuted SPDIF in alsamixer and then it was just some settings in VLC. In preferences, turn on advanced settings, set the output device from Default to ALSA, refresh the list of ALSA devices and select nVidia device 0,7. There were a few nVidia devices, and 7 is the one that seemed to work for me.
Anyway, problem solved! Thanks for all the help!