Yeah Gentoo could be, but it's main idea is to compile everything from source when installing - which means long compiling times when installing anything. If you plan to upgrade your desktop environment, prepare for a really long compilation.
I could basically vote for Slackware, but you can get pretty much any distro slimmed down to a speed daemon with small size; with Slackware it's just relatively easy. It depends on how low you want to go; with LFS you could get very small sizes, but you would have to "do it yourself" completely
Note that all RedHat distributions are "biggish"; the only way to make them small is to leave everything off during the installation and then pick up only those single packages you need (which draws in their dependencies too), and/or after setup remove unneeded stuff.
DistroWatch => small distributions. Either one of them or LFS.