I think I found a solution.
Well for me VLC audio delay was tolerable but when I tried out Google Play Music service, the audio was stuttering so badly that I couldn't even think of using it. So I dug a little deeper to find a solution.
As I said earlier I'm using Nvidia HDMI audio and ALSA dmix plugin.
My asound.conf file looks like this:
pcm "hw:0,7" # Always use pure hw. dmix will reformat/resample audio.
period_size 2048 # If you get stuttering/or non-working audio, fiddle around with these
rate 48000 # HDMI, I'll assume 48kHz
format S16_LE # Should be default for pretty much any soundcard.
Of course, this is my config and it likely doesn't work as is to your setup but I bolded the options I changed in order to get it work. Previously they were much smaller.
I hope this helps somebody.