Can someone explain to me the logic of LFS's new build system?
 

I am ploughing through LFS 10.0-rc1 and making very heavy weather of it. I feel I really don't understand what I am doing and that's a feeling I don't like.

The old system was logical and easy to understand. You built a set of tools and they went in the $LFS/tools directory. That was at the head of your path, so each tool came into use as soon as it was built. In effect you peeled away gradually from the host system until you were independent enough to go into chroot.

Under the new system, $LFS/tools is still at the head of the path but only the primary compilation tools go there. Chapter 6.1 says of the other tools:
Why can they not be used? Is it simply because they are not on the command path, or are they not on the path because they cannot be used for some other reason. And why is everything cross-compiled for a fictitious host? I can see why that must be done for gcc and binutils to make a clean break with the host environment, but it never used to be done for the ancilliary tools.

I'd be grateful if someone could explain to me why things are now done this way.

I don't think that much is changed.
The old chapter 6 would chroot into a tools-environment as built in chapter 5.
In the new version of the book, you would do that in chapter 7 now.
The way the book is divided now, is confusing. Part III contains chapter 7 and part IV contains chapter 8 but to me chapter 7 and 8 together remind me of the old chapter 6.

I have not done a build yet, I liked the way the tools could be used to build newer LFS-versions. A LFS-8.2 ch5 tarball can be used to build LFS-9.0 for example.

When you do a build, you'll see the difference. All the tools are now installed in their final positions and not used at all until after you have chrooted. The stage 2 (native) builds of gcc and binutils come right at the end of the new chapter 6 (previously the end of chapter 5), so it is the cross-compiler versions that actually build all the tools. It would be nice to know what advantage this provides. There must surely be a reason for such a great upheaval.

Took one look at v10 it just looked a complete mess, I will skip until the next version hopefully it will be a bit better arranged.

This has been officially released now but you can't reach it yet from the Read Online page of the LFS site. You can find it at http://www.linuxfromscratch.org/lfs/view/10.0.

I've now got to the beginning of the chroot section. As far as I can see, it's fairly familiar from here on except that you no longer need to create symbolic links for things like bash and perl because they actually are in the places they should be. The new way also does away with all those edits you used to have to do before building Pass 1 and Pass 2 gcc, and the reconfiguring of gcc and binutils after installing the final version of glibc. Whether that's enough to justify such an upheaval is another matter.

Like a lot of people who use LFS all the time I script all of the tools and basic system building, I can see this breaking a lot of scripts, oh well devs just love to change things :)

I do it all by hand. Using scripts seems to me like cheating. But I do want to understand why it's done this way.

Completed the final gcc tests this morning. From now on, it should be no different from the old way.

Curiously I had 16 unexpected failures in libstdc++ which I have not had on this machine before. They all cluster around two directories: 27_io/filesystem/iterators and 27_io/filesystem/operations, mostly the latter. It seems that this part of gcc-10 doesn't much like my system!

@hendrickxm. They've basically split the old tools chapter into three:
5) Basic compilation tools, cross-compiled: binutils, gcc, glib, stdlibc++.
6) Secondary build tools (e.g. bash, coreutils) that must be built mostly using host tools, also cross-compiled. This section ends with the Stage 2 recompilation of gcc and binutils.
7) Tools that can be built in chroot without using the host at all, natively compiled, e.g. perl, python, testing software.

I suppose Chap 7 is grouped with Chap 8 (building the rest of the system) because it's all done in chroot from this point onward. It seems to be separated out as a chapter only because these programs historically went into the toolset. But then pkg-config and libtool are tools too. If I was editing this, I'd combine chaps 7 & 8, rather than putting chap 7 with 5 and 6.

My experiences up to and including chapter 7 are positive.
The build was done on a LFS-7.3 chroot with an updated toochain.
For newcomers it can be more of a hassle and I can see them having issues where they would not have those building LFS the old way.
The backing up is a nice addition but how many will forget to re-chroot afterwards.
I wonder if you could do build for an older LFS using the new build system.
I am also curious to find out if I can manage to build using an even smaller chroot. Right now it's a full LFS-7.3 I use but I expect you can skip what will be build in the new chapter 7.

I still haven't built the kernel. But I have already noticed an unusual number of check errors. As well as the ones in gcc, I had 5 in coreutils and 4 in vim. I have never had check errors in vim before. I doubt if that has anything to do with the build method; more likely they are using less well established software versions than they usually do.

As I mentioned in another post, I was not able to do the util-linux tests with the tester user as the book sets it up. I'd be interested to know if you run into the same problem.

Usually I don't do the tests. Mostly it is clear something is wrong when you start compiling things after final GCC, at least that is how it always was using the old build system.
But now the packages needed for testing are not build in the temporary tools but in the final system, so I will build them this time and test gcc, coreutils and vim (I prefer nano so normally I skip vim).

EDIT: Tests for gcc were normal and libstdc++ had no unexpected failures.
I did get an error from attr getfattr.test failed.

I wonder why I got so many. That's never happened to me on any previous build. I usually get just the 6 expected ones.

Anyway the thing boots, so the new method definitely works.

In the buildlogs linked in the LFS-book, there were 7 unexpected fails for libstdc++ o one of the builds, the one I checked.
Some positive news about this new way of building is that the final glibc installation no longer needs the toolchain adjustments.

There used to be 6. I wasn't counting those; mine were additional.
You don't need to do all that tiresome editing before building the 1st and 2nd pass gcc either. But is that enough of an improvement to justify turning the whole book upside down?

I must say I used to enjoy gradually peeling myself away from the host system as more and more of the new tools came into use. Now you are basically using the host until you chroot over and start using the new tools for the first time.

Saved by the past again
 

The Toolchain Technical Notes got more complex, and the text underwent serious reorganization, but I guess the most important change is the thing with the toolchain packages. I think I will like that. In a way, LFS has circled back to its past by installing most of the toolchain packages in the $LFS directory tree. The very first version of LFS did exactly that.

A brief history of LFS

I haven't built 10.0 and probably won't because I just rebuilt my whole system with an upgraded package set from a development book. I don't won't to do all that work right now. But I might try modifying my existing scripts to build my current system with the strategy of version 10.0 just to see what happens. That won't be much work.