What exactly do you want to speed up or optimize ? I'll assume overall system responsiveness and speed.
What window manager are you using ? If it's KDE or GNOME ... might wanna try some others.
Because gcc is truly a black box when it comes to optimizations, recompiling most things won't help much performance-wise. In certain circumstances tho, you can compile say a 3D graphics engine and then the game that runs on it from source and see a moderate improvement in performance.
Mostly you'll want to compile a kernel from source, especially taking note of processor options, I/O scheduler and a few other things. See here (also has boot speed optimizations):
I also hope you know which CFLAGS to use. I usually stick with the same ones used for the kernel. So when you compile the kernel with the right processor selected, you'll notice it will spit out what CFLAGS it's using. I use those for anything I want to optimize, it seems to work best. Also, I would strongly recommend NOT even using the '-O3' option. It doesn't help, and it will most likely break the program you're compiling. '-O2' works just fine. Some suggest '-Os' ... you could try that, but I don't trust it.