Check out
Ratpoison for a minimalist window manager. Out of the box you'll wonder what's good for - you have to configure it yourself - so download some
configurations from otther ratpoison users. Google on "ratpoisonrc".
It doesn't require any libraries other than those you've already got. It uses the small X utilities which so many of us didn't know that we had on our system. It's so small that you can compile it in minutes.
So keep all the X stock programs and throw out all the window managers (kde etc.). and you really will have the minimal X install - yet one far more powerful than the glamorous big window managers.