Well, since you revived this thread, I will say that I've also solved my issue some time ago. I realized the problem was.... elilo
For very long time I've had my system booting in EFI via rEFInd which called elilo which called the kernel. I won't get into details of why I was doing that.
I removed elilo from the equation and it all worked flawlessly, without any unusual compilation parameter and with nvidia-drm.modeset=1
I assume that elilo was leaving the boot environment in an incorrect graphics configuration compared to the needs of the nvidia driver from a certain version onwards.