What Niki said. On the other hand, removing XFCE can only free a few MB on a hard drive. Keeping it won't slow your machine in any way and will allow you to use a few XFCE apps that you could find handy, as all can be used under Fluxbox (which I am using too, by the way).
More generally bear in mind, especially if you install applications built from SlackBuilds available @
http://slackbuilds.org that a full installation is always assumed, so you could find cases where you miss some library to compile or run the application, as it is supposed to be already installed.
Let's give an example: a friend of mine who's blind didn't initially install the X series of packages as he can't use a desktop or windows manager, but recently, trying to compile "catdoc" he came across a missing "libXft.so.2", contained in the libXft package from the X series.