I'd try a few out and be pragmatic about which you choose. i recently setup my first linux desktop and after trying centos, fedora, and opensuse - none of which would run my video card and tablet drivers without too much stuffing around - and finally settled on ubuntu which ran it without a hitch.
it's fairly easy to dial back all the eye candy in any distro, just turn off all the desktop compositing. apparently xfce is one of the more lightweight desktops in terms of resources, but having said that on a vaguely modern machine with all the compositing eye candy turned off, gnome/kde will hardly bog your system down. unless your daily computing tasks max out your hardware i doubt xfce will make much difference.