Xdm reads ~/.xsession and executes the Xsession exectable (startxfce4) that you've set in that file. Wdm also reads ~/.xsession if you choose the default session from the wdm sessions list.
Supposing you've installed your locally compiled xfce to /usr/local/xfce/ (and you have startxfce4 in /usr/local/xfce/bin/), you need to make the contents of ~/.xsession to look like this:
After that, xdm should start your local version of xfce. Also, wdm should start your local version of xfce if you choose the default session from the wdm sessions list.
If you're sure that the path in ~/.xsession is correct and xdm still doesn't start your local xfce, you probably need to make ~/.xsession executable:
After that it should work.
For wdm you can also make a new .desktop file in /usr/share/xsessions/ (take a look at the other Xsession files under that directory to see what it should look like) and edit this new .desktop file to point to your local startxfce4. This should add a new entry to the wdm sessions list.