Well, I take it that you will be keeping to the Filesystem Hierarchy Standard
for your build.
Having built my fair share of systems from the kernel up, I can recommend two places to consider starting from:
- Linux From Scratch (a.k.a. LFS)
- Gentoo Linux
Both of them require that you configure and build your own kernel, so you can make it as lean-and-mean as you want. Both are building the entire overall system from source code... The real key difference is that Gentoo will provide an existing stable environment for you to build your new kernel in and provides a very stable, robust package manager for you to pull in the packages you will need. LFS, on the other hand, makes you build your environment and toolchain before you even start pulling the kernel sources. It's really good for getting an idea of what the original devs had to go through to get us to where we are today.
So, you have options to go check and see which one you want to start with.
HTH. Let us know.