The process of making a Linux distribution from scratch (concerning libraries, etc.)
 

I haven't been learning for too long about this. I have C in a Nutshell but I'm learning Bash right now. I've gone through a lot of LFS, but didn't complete it. I'm still learning about symbolic links, hard links, libraries, and this sort of thing. I'm wondering about quite a few things at this point.
How the hell can anyone make a distribution from the ground up? Even if they start with another distribution as a base, how do they know (I realize one person doesn't do all the work) what to do, what order to do it in, how to synchronize what they're doing with what everyone else is doing, etc.
How do you know which software to compile and install first? etc.

Also, I'm entirely confused by this:
how can everyone (even large teams) get everything working together? Knowing that this works with that and that doesn't work with this unless it's patched but that's what we're going with, etc. I can't understand how these things can be done without an almost unbelievable amount of effort unless a lot more is done by ./configure and make than I know about at this point.
And I don't understand how libraries work at all. I mean, I'm going to learn a lot more and I've been reading a lot... but how can all the distributions just download and install 50 updates at a time and not have to restart the computer most of the time!? There are a hundred things happening in the background constantly. How does everything not just fall apart as soon as you update one package, let alone 50 at a time!? I realize this is stuff that you learn by reading, but I've been reading for quite a while now and I do not understand even the basics about why any of this works or how to know what works and what doesn't unless every single little piece of every distribution is gone over with a 50,000-page reference manual that's constantly updated. And all of this can't be as complex and hard to do as it seems, or there couldn't possibly be a hundred distributions that work and are updated constantly.
Linux From Scratch is a good example. You have to download all the very specific versions of the software and patch it with special little patches, etc. How the hell can anyone keep track of all of this? Going by what I know about things, it's almost hard to believe that it's even done.

It's actually quite simple. The key to a working distribution is package management! You can do your own (some LFS-article suggested symlink-based package management, for my LFS-based PVR, I took the idea and wrote a VERY tiny package manager in perl) -- or you just pick up an existing package manager, that would be either dpkg or rpm.

A good, feature-complete, package manager will provide at least installation and removal of packages, dependency tracking and patch management. dpkg nowadays can delegate patch management do either dpatch or quilt, the latter seems to have the better features.

Whatever you do, proper packages management is the key to success. If your package management supports maintainers in creating stable packages, it's quite easy to "distribute" the work needed for a distribution to hundreds or thousands of individual package maintainers.

edit: Let me add the following thought: As there are two major package managers out there already, you would probably go with one of them: convincing maintainers to support a different package format could be quite hard :)

How does everyone keep track of all the patches that they need to keep track of? Is it not as often as I'm thinking that patching, etc., have to be done? I'm thinking especially of LFS, because every version of LFS has a big list of patches for a very specific combination of software versions and it's hard for me to imagine that it's not an extremely complex process.

prdeltoid, i don't know the specific development process of the LFS books, but i assume it's somehow the same as in binary distributions: every maintainer of a package looks for needed patches etc...

Of course, it's quite impossible for a SINGLE person to get it right.

If you really want to start a new distribution, i'd suggest to focus on the GNU "toolchain". That's the base for all the other parts, every maintaier of a package must use it.

Well, I'm not really planning on starting my own... thanks for your help, though.

Ah ok, I think it's an interesting subject, because I once started following the original LFS book, and even as a single person, following exactly the book, it turns out to be a beast in the long run. That's how I found out package management really IS the crucial element. I even started to use my own extremely tiny package manager (written in perl) for /usr/local on debian :)

I really think "management" is the key to your question. A single person can never maintain a distribution, and good collaboration requires good package management.

Once you have glibc, kernel headers, and gcc you can compile everything.

look at lfs - it's a textbook example of how to do it. Back in the early 80s a guy called Richard Stallman started projects for the entire unix system called gnu. At that stage, it was all extremely difficult to get together, but they simplified it greatly. The system is called linux because they were trying for a light kernel, linus wrote a heavier one. It was like putting the battery in a car ready to start - the thing took off from there. It's really a linux kernel on top of a gnu operating system.

I think I'll try LFS again. I was just reading some on linkers/loaders, and symbolic and hard links. I think the rest of it is understandable... I don't think it's worth spending too much more time learning before I try it again.
Thanks for all the help, pointing out how important package management is helps a lot also. I really wasn't thinking that it was that central at all to the process. It just surprises me that even with a large team things come together so well. :D
EDIT: I think I'll learn some more grep/sed/awk first, though. Just a few tutorials.

I think LFS doesn't stress package management that much because LFS' intended audience is a SINGLE user trying to compile his own system :) But for me, even in this scenario i found the benefits of "proper" package management quite valuable. In a scenario where the release of a distribution is the goal and a lot of maintainers work together, it IS really crucial :)

btw, grap/awk/sed is really a good idea. I would add quilt to the list, because this tool simplifies patch management a lot.

How does a company like Boeing keep track of all the parts of a 777? Some of the same principles apply.....

pixellany, that's a great example -- especially to all those condemning management nowadays. GOOD management is an important success factor for all projects where many individuals are involved :) In a linux distribution, the "management entity" typically consists of a package -- what is needed to build it, what information artifacts are required for trouble-free installation and removal of the binary distribution, etc.

I've been through the LFS book.........and it booted first time, it even pings. Two months of Slackware was enough training to make it through LFS. It's mostly tar, make, make install, cat, cp, rm, mv. A little experience with those commands will get you through the book.
A second trip, or a patient first one, will help you a lot in understanding. The book is really good at explaining what and why things are happening.

If you want package management in LFS check out checkinstall. It's something like

checkinstall make install
after you compile, and that installs and logs the packages.

Thanks for the advice. I'm not going to be using LFS now though... the LiveCD I downloaded only had a 64-bit kernel on it and so I got about 5 chapters into the book before I used "uname -m" and saw that I'd been using a 64-bit kernel, so I restarted and pressed F1 for the list of kernels and that is all there was. I even made sure by checking the /boot directory. I'm fed up, now I'm back with Slackware and Debian again.

A stock slackware makes a fine LFS host system.