A shot in the dark: perhaps some framebuffer module is conflicting with Xen?
1) Maybe if you check the output of "lsmod" when using the stock kernel and compare it to the same output when using the new Xen kernel (using a different kernel when using Xen?) you'll see a module that is loaded only when using Xen? => blacklist it.
2) Perhaps by checking the output of "lsmod" (connecting to it using ssh from another PC) when booting with the Xen-Kernel you can select the modules that are not really needed and then blacklist them all to see if it gets better?
Unluckily I just know that with the Gentoo distribution the blacklisting of kernel modules is done by specifying their names in "/etc/modprobe.d/blacklist.conf" and have no clue if Ubuntu uses the same file or some other one