So I actually had these exact symptoms using a USB DAC on SolydX, a Debian based distro. It turned out my solution was to comment out "load-module module-suspend-on-idle" in /etc/pulse/default.pa, the default pulseaudio config file for my system.
I had serious troubles finding a solution online, though I kept stumbling onto this thread, so I wanted to give a potential solution to anyone else in a similar boat.
I ended up running pulseaudio in verbose mode while opening and closing pavucontrol with an mp3 file playing in vlc to debug this issue. I noticed messages like this:
"I: [pulseaudio] module-suspend-on-idle.c: Source alsa_input.usb-FiiO_DigiHug_USB_Audio-01-Audio.iec958-stereo idle for too long, suspending ..."
I had already tried all sorts of different settings in pavucontrol to no avail, and noted that my DAC only worked if using a profile that had both input and output (Analog Stereo Duplex for example). With this knowledge, I figured that my DAC needed it's input stream active to give me sound. That combined with the fact that it was reading as idle and getting killed off by pulseaudio, led me to my eventual solution.
It's worth noting that I did not have this issue on my desktop PC using the same DAC and OS. I believe this is likely due to pulseaudio enabling different modules by default based on system hardware differences, but have no clue. It's also worth noting that I disabled my onboard sound by using pavucontrol to set it's profile to "Off". This step may also be necessary for some people, though defaulting the DAC in output devices tab may also have the same effect.
Hope this helps someone
Sorry for any post formatting faux pas, this would be my first here.