Check out the 'CLFS' project under 'LFS' (www.linuxfromscratch.org
The problem with just about ANY distribution is that your system will be bloated with package managers, GNU libc, and numerous other things. If you build from scratch you can have an awful lot of tools etc (and only the tools that are absolutely essential), all crammed into under 15MB (about 6MB compressed). It's just that it takes a lot of time to sort the software out ... one at a time. Using a distro you can push something out the door in under a week - if you don't mind the bloat wasting space on your CF card.
The other advantage of building yourself is that you have a greater choice of target processor. If, for example, you want to run on an ARM machine, then building from a distro limits your choices (Debian has packages for ARM, and Gentoo probably does too).