@roxie Question: Would your GPUs both be using the same driver? If not, I think you might be able to disable the driver for the one you want to pass through at startup.
Although, I remember when I did a setup like this myself, it did lead to issues, but I believe that was because I was disabling AMD's drivers and there was some weird issue with the proprietary Nvidia drivers that completely broke things, and I needed to use the open source nvidia drivers.