Using X in your situation is hard, but possible. Just don't touch KDE or GNOME! Look for other window managers (Window Maker, Blackbox, AfterStep...). And your system configuration should be a very good one. Don't install anything you aren't sure you'll need. It will be also good to edit startup scripts.