When you recompile, in the first option after prompt for development options is architecture. In there, (if in menuconfig) highlight i386/i486/i586... and press enter, then arrow down to Athlon/Duron
X11: I am afraid I will be comparing apples to oranges sort of, but when I recompiled my kernel, I optimized for architecture, but also took out module support, and built everything I needed into the kernel. This was on my mobile box, so all I needed was filesystem support, keyboard, and sound. I also disabled startup otptions, and went from around 1 min 20 sec boot to around 25 sec. I can't really calc performance other than time it takes to recompile the kernel. And since there was much less to compile, it obviously went faster.
The only other time I have recompiled my desktop's kernel, I saw a minimal gain in performance. Not quite what I expected, but there was some. Switching to a lighter X, and doing more in Console was what really proved performace gains.