Done it recently on 8.04.
The quick and easy way is to install meta-package xubuntu-desktop but that pulls in lots of applications so was going to download 354 MB of packages on my system. I was looking to slim down so didn't want that.
I ended up looking at the packages xubuntu-desktop was going to install and picked a minimal list just to get the essentials: xfdesktop4, xfce4-session, thunar and mousepad. I still haven't tried mousepad in place of gedit. I added xfce4-datetime-plugin to get date and time displayed simultaneously without having to mouse-over.
During the cut-over, I used gdm's session chooser to choose Xfce or Gnome at logon time, as required.
Setting up panels was easier than in Gnome but putting Launchers on them was more manual.
Setting up dynamically changing desktop wallpapers ("backdrops" in Xfce-speak) was easier than in Gnome, not requiring any extra package, just a noddy shell script, configured to launch at session startup. See my blog
Configuring context menu items in Thunar (Xfce's default file manager) was easier than in Gnome's nautilus, again not requiring any extra package.
"Launch Gnome services on startup" was configured by default. I de-configured it and nothing broke.
There was a problem with VirtualBox windows not displaying nicely which I worked around by installing Xfce 4.6.1 from https://launchpad.net/~jerome-guelfucci/+archive/ppa.
(thanks to LQ's Sasquatch for telling about that). 4.6.1 is a little more polished than the current ubuntu Xfce package but does have a "silly" problem with taskbar buttons (see http://bugzilla.xfce.org/show_bug.cgi?id=4840
and, of course you don't have the "comfort" of being in the main stream of official ubuntu packages (for what it is worth).
When it was all working nicely, I changed from gdm to xdm for the logon screen. Very happy with the changes -- so much quicker, lighter and somehow "smart".
I've got lots of notes about the changeover -- too much to post here -- but could turn them into a .pdf and attach if it would help (how do you know, when you haven't seen the shorthand notes I wrote for myself?!)