Though I haven't attempted something like this myself, I believe you would begin with CLFS and then use what you can from HLFS.
This thread seems to confirm that, and while it doesn't offer detailed instructions, it should hopefully provide you with some valuable hints.