The desktop installs of Debian, Ubuntu, Fedora, etc., for ARM, put the ARM OS over a stock Android Linux kernel. The kernels for these devices often have proprietary hardware drivers, and running a replacement kernel (built from scratch versus hacked together from a stock kernel) would break a lot of these devices' hardware support (like cameras, GPS, and graphics chipsets).
So, you might as well just leave the stock OS (or modified stock OS) intact. You stand a better chance of "bricking" your device if you try replacing the OS than getting the ARM versions of the distros running well on it (as an OS replacement, I mean). That said, what I've heard of was running ARM ports of these OSs using chroot,... not a VNC to another machine.
... for an example.