I have the very same problem before but not anymore. Kernel 2.4.19 (athlon, latest), KDE 3.1 (apt updated), XFree 4.3.0 (stock), sax2 (apt updated), 3Ddiag (apt updated), NVIDIA_* (newest from nVidia)
I solve the problem mainly by using apt-get ;-) Here some that I can recall:
Rpm -Uvh XFree86-everything exept: -libs -man -server and -xf86tools
(I did the rest by following the README instruction -> this give me troubles) :-(
Fire up apt-get update then apt-get upgrade (but not to choose yes -> do it manually with apt-get install packagename). This gave me a clue of some conflicting packages (XFree86 obsoletes these packages ... delete them all)
Run nv_check.sh (this gave me a clue of problems with my nVidia driver)
-> solve those problem as suggested (well... not exactly) ;-)
Remove previously installed video card driver
Reinstall with newest driver (a pair) -> Stomping one of the XFree86 file :-(
Run 3Ddiag -> This give some clues to fix 3D problems
Run x86config -> This give some newest options available
Run sax2 -> Need OSD to correct further
Edit /etc/X11/XF86Config -> Further tweaking...
Select other kernel (k_athlon) -> Oops not too stable with production 2.4.20 -> back to the latest 2.4.19
Reconfigure KDE3 -> make it nice
Have a lot of fun!