Building software from source

In the early days of Linux, a lot of people found themselves having to build programs from source. Nowadays many users have never done this or wanted to do it. And that can be seen as a great step forwards because it means that Linux is now an operating system for ordinary people and not just for hackers and geeks. And given all the bad things that one can say about Windows -- the instability, the malware, the constant "phoning home", the unfair EULA conditions, the invasions of privacy, the enforced updates and the constant need to upgrade your hardware -- the need for a user-friendly alternative is patently obvious.

But suppose you find yourself in the position of really needing to use a program which is not available from your distro's repository. You have been using Linux long enough to know that Google (or DuckDuckGo) is your friend, so you do a search and quickly find the website belonging to the project. You go to the section signposted Download/Linux and immediately you find you have a problem.

There may well be a couple of installable packages for Linux (typically a .deb and a .rpm) but these are quite likely to be incompatible with your distro. For example a .deb file can be unpacked inside any Debian-based distro, such as Ubuntu or Mint, but is unlikely to be installable as it will probably have been built against different libraries from the ones you have. That is so even if you are using Debian itself, as the package on the site will probably have been built for an older version. And if you do manage to install it, something else may break as a result.

The other file that you will probably see in the Linux section of the website will have a suffix like .tar.gz or .tar.xz. This is a source "tarball" and it is the safest way to get the software installed on your system. Source code is the program as the programmer wrote it: a collection of plain text files containing compilable code in C or C++, together with headers, accessory graphics, documentation, and build scripts. Because it is all plain text, it should be compatible with any version of Linux.

So you download it, use tar to decompress and unpack it, and quickly move into the folder that you have created. You expect it to look like an unzipped Windows package with a file called something like Setup or Install or Run that you simply have to launch to get the other files into their final place. But you will not find them. If there is an INSTALL file, it will be a text file containing a set of rather obscure instructions that you are somehow supposed to be able to understand intuitively. And if you go on YouTube for enlightenment, it will probably leave you more confused than ever.

In fact building software from source is not difficult once you understand what you are actually doing. Linux has a standard build system (autotools) which ensures that most software builds are done the same way. The problem is getting an explanation of what is going on that is appropriate to your current knowledge. Here are the basic steps that the build scripts need to carry out:

• Configuration. This is carried out by a shell script called configure. The script checks your system to ensure that you have all the necessary programs and headers to carry out the build and all the libraries and other dependencies that the program will need to run. If it finds that something is missing, it will stop and tell you. If everything is in order, it will create a set of "Makefiles" that will guide the actual build.
• The build itself. This is carried out by the make program, guided by the Makefiles that the configure script created in the toplevel- and sub-directories. The build takes place in several stages:
  • Compilation is carried out by the gcc compiler, which translates the code into a much larger set of very simple statements corresponding to operations that a computer's cpu can actually carry out. These statements are written in a so-called assembly language.
  • Assembly. The assembly language version is rewritten in binary machine code by the GNU assembler. This is a relatively easy process as assembler and machine code have a one-to-one relationship.
  • Linking. The separate machine code files are linked together into one executable file. At the same time, pointers are inserted so that the executable can link dynamically to all the libraries that it will be using when it runs.
• Installation. This is done using the make command again, this time with "install" as argument. Following Makefile instructions, it copies the executable to its final location (by default /usr/local/bin) and all the ancillary files to their proper locations in the /usr/local tree.

NOTE 1
The above steps are pretty universal but some packages use different software to carry them out. Two programs that are sometimes used as alternatives to a custom configure script are meson and cmake. Meson builds typically use ninja instead of make. Cmake builds may use either make or ninja. But the overall build process is still structured in exactly the same way.

Now that you basically know what you are trying to do, you will find the documentation inside the package much easier to use. Start by looking for files called README and INSTALL. If both are present, read them both! These files should tell you what libraries and other dependencies you will need for a successful build and the specific commands that you will need to give for each stage of the build. In 99% of cases, the command sequence is simply:
1) ./configure
2) make
3) sudo make install

For a meson/ninja build, it will usually be:
1) meson
2) ninja
3) sudo ninja install

NOTE 2
1) Note the "./" in front of "configure". This tells your system to run the executable configure file in the current directory rather than looking for a command with that name.
2) The installation stage requires root privileges, hence the need to use sudo.

Tools for the job
Building software requires a specific set of tools. Some of these, such as gcc and make, have been mentioned above. There are quite a few others that are invoked by makefiles. Fortunately most distros package these tools together so that they can be installed in a single operation. For example, in Debian-based distros the package is called build-essential. Install this and you are good to go.

You will also need the header files for any libraries that the package links to. The libc headers come with your toolset, but headers for other libraries may need to be installed by the package manager. This is one reason why you need to read the INSTALL file carefully. It will tell you if particular library headers are needed for this build; installing them in advance is better than scratching your head wondering why the configure script crashed. It is actually the headers that the configuration script looks for to check that necessary libraries are present, because it is the headers that are more likely to be missing.

Most distros package libraries in two parts: there is the runtime library containing the binary code that programs need to link to, and the "development package" which contains the headers and the documentation. In Debian distros, the development package for libfoo will be called libfoo-dev. In Fedora and its relatives, it will be libfoo-devel. Installing the headers will usually bring the runtime library along too as a dependency if you haven't already got it.

The commonest cause of crashes during a build is failure by the configure script to find something that it needs. But don't just look at the final error message, which is usually completely uninformative. Let your eye travel up the output and you will soon see the name of the missing item. The solution is usually quite simple: break off the build and install it. Then use "make clean" to reset your build and start again with "./configure".

Very occasionally you may run into a problem with version mismatching. The build requires a particular minimum version of some piece of software and the one on your system is older than that. Usually it is the configure script that reports the problem. A crash during linking, accompanied by an error message from ld can also be caused by this. The linker is looking for a function to link to which is different or lacking in your installed version of the library. Simply installing a more up-to-date library isn't usually a good solution as it may break your system elsewhere. This is where it makes sense to ask the Forum for help.