I'm a newb to linux. I recently installed Fedora 24 with a Windows guest with GPU passthrough. I've been using the virt-manager for the time being, and have managed to get everything working about as I'd expect, except for USB.

My Goal: I bought a USB Switch that'd I've hooked to my laptop and my kvm machine. Attached to my usb switch is a 7 port usb 3.0 hub with several items plugged in (mouse, keyboard, etc). I'd like to be able to flip my usb switch, and have my windows VM attach and detach all the devices connect and disconnect accordingly.

It seems though that when my hub is connected to my kvm machine, virt-manager can see individual devices (not my hub as a whole), and I can attach those devices while the VM is off. But it has 2 unexpected downsides:
1. virt-manager refuses to boot the VM if it can't find all the USB devices, so I have to have the usb switch turned on for the kvm machine in order to boot the virtual machine (this is a pain since I want the VM to boot on startup. If I'm manually booting the VM, I typically do that via VNC from my laptop, which requires the mouse and keyboard to be pointed at my laptop at that moment).
2. Once the VM is booted, if I swap the USB switch to point at my laptop, and then switch it back to the VM, my usb devices no longer work for my VM.

Is my goal achievable? If not, what's my next best thing? I care about latency, as I'm trying to use my windows vm with gpu passthrough for gaming.

If I'm imagining your situation right, the OS doesn't "see" the switch.
And then you mention a USB hub. The OS doesn't see the hub either, only the devices on the end.

Also I'm not sure why you don't have the keyboard and mouse connected directly to the host. KVM manages the keyboard very well like that. If you really want some kind of dedicated serial comms (what a keyboard is waay down the stack), get another keyboard and connect it to a port that you pass through to the VM.

Gaming is not an optimal use for VMs if you are so sensitive about input speeds. I know that doesn't really answer your question.

Input speed isn't really my goal. Convenience is. I use the same pair of monitors for my laptop, and I'd like to be able to easily switch between my laptop and my windows VM by pressing my usb switch without having to restart my computer.

Then, I'm not understanding.

You have a Linux laptop that runs a windows VM only?

You have a Linux tower that runs a windows VM and a laptop and you want to effortlessly switch between the two?

I have a mac laptop. I also have a linux desktop that has a windows VM running inside it. I have them setup next to each other, and would like to dynamically switch my keyboard and mouse between the 2. I have a USB switch setup to allow for this. Currently though, the VM insists that the USB devices be attached to the linux tower when the windows VM starts. In addition to that, if I swap them to the mac, and then swap back, they no longer belong to the windows VM, and won't work until I restart.

Thanks for the clear explanation. Somehow, I wasn't completely sure exactly what you were trying to do in the first post.

This sounds like a situation where synergy is preferable to physically switching the devices, although that isn't an option of the Mac laptop and Linux desktop aren't on the same network (i.e. Mac is on a work network while Linux is on a guest network). Still, you mention gaming so I'm guessing this is a home setup with both machines on your personal LAN/WiFi network. In that case, synergy is a great solution.

Basically, I would:

Install synergy on both machines.

Plug in USB devices to desktop.

Use synergy client on Mac; use synergy server on Linux.

Whatever other devices you wish to share, such as an external drive, share via nfs or sshfs or some other method.

I can walk you through the specifics of setting up synergy if you're willing to accept a huge data dump. But there are a lot of tutorials on how to set it up out there, that have nice friendly screenshots and everything.

I bought the usb switch to try and avoid network solutions. I'd also like to avoid network based solutions to avoid input lag.

Is there no way to pass usb ports / an entire usb controller to the windows VM? I'd even be fine if the windows VM owned ALL the USB devices by default. none of the other VMs need USB devices.

My experience with synergy is that unless the network is being saturated with something (which will make EVERYTHING laggy), and synergyc/synergys are fresh, then there is no perceptible lag. And I find that the speed of simply moving the mouse left or right to go from screen to screen is faster and smoother than applying some KVM switch.

Now, with gaming even the slightest lag can be significant (at least in one's own mind), but your situation suggests putting the synergy server on the desktop computer anyway.

If your gaming is on the Mac laptop, though, then it could make more sense to put the synergy server on the Mac laptop.

To me, putting the server on the desktop is a more elegant solution, though, since it's one less the to plug/unplug on the laptop when it goes travelling.