I haven't used Xen yet on Fedora, although I plan to try soon. According to what I have read, Any slowdown in OS performance should be negligible when using Xen. It is supposed to be as fast as (or faster than) VMware. That is why all of the big tech players like IBM, Intel, AMD, and even Microsoft are creating virtualization-based technology with Xen at the forefront. Xen acts as an interpreter to both the Host and the guest OS, essentially resting between the OS and the hardware. As a result, the OS kernel needs to be reconfigured to work under Xen (which is why you won't see Windows running directly under Xen - you can't modify the source; although rumor has it that Microsoft already created an experimental version of XP that works under Xen). Thus it would stand to reason that if you optimize a Xen-configured kernel you would not need to reinstall Xen - however I am not 100% sure though.
If you haven't already, check out these links: