Experience: CPIO-based Slackware packages
 

Hello,

I develop and maintain Spack, a tool (one more ;)) dedicated to the automation of the Slackware packaging process.

In the last release of this one, I've decided to put in code some ideas I had in a corner for a long while, elaborating an alternative way to make and manage Slackware packages.

The main interest was to remove tar-1.13 from the process. This was achieved by replacing it with cpio, which has the advantage, among others, to be a maintained and widely deployed tool. I've also tried to optimize as much as it was possible the attached package manager and to think another interface for it.

Maybe the result can interest some curious and/or adventurous people here. You don't need to use Spack to test it, just install the spkcpio and spkman tools from the sources, like this:

Then you replace the makepkg's line with this one in your SlackBuilds:

After, install the result with:

Or (as an equivalent for upgradepkg --install-new):

Plain upgrade is peformed with:

And removing with:

For the two last operations, you can of course use the standard pkgtools, as once installed there is no difference with a good old txz (bug except).

For safer tests, there is also a --root=/alternative/root option, which do the same as the one in the pkgtools, and a --dry-run one.

Note that the current release (2.4) has a little known bug, which will be fixed in the next one: the upgrade process wipe out all the links of the old package instead of just an old/new packages diff (the final result is the same, it's just not optimal).

Good test for those who will try it, don't hesitate to give some feedback. :)

Hi,

As I was notified about this zero-reply thread (don't worry, I won't flood the forum about my stuff ;)), just a little "up" to say the previously described bug has been fixed. Moreover, spkcpio is from now on able to generate a CPIO package directly from a Slackware one.

Uh, there's a couple of special reasons for using specifically tar-1.13.
1. tar-1.13 has a special way of listing the archive contents and of storing the contents in the archive(this is important because of the way the database files are generated. Using other tar versions to install packages created with ta-1.13 or using tar-1.13 to install packages created using other tar versions may not work correctly)
2. tar-1.13 had two non-POSIX-compliant behaviours which were very desirable -at least to PAT:
a. preservation of existing linked directories -even when the archive contains a 'real' directory the same as the linked directory on your system. This means that if you have /usr/share/doc as a link to /usr/doc, a package which contains the directory /usr/share/doc, tar-1.13 will redirect the files to /usr/doc without overwriting the link /usr/share/doc
b. meta-data for existing directories is preserved instead of being reset.

Later versions of tar do not preserve these behaviours -not even as options.

cpio has another disadvantage. In cpio archives which contain directories which do not already exist on the system, these will be create with perms of 0700 (or is it 0777??) instead of 0755. Either way, those are not sane settings for normal dirs. You'd need script lines in each doinst.sh which would properly reset the perms. Or maybe resetting the perms during the build of the package....

I'll gladly stand corrected on any of these points if anyone knows more about it than I think I do...

Slackware without pkgtools is like pizza without cheese.
Nevertheless, I wish you luck.

About point 2a, which is the one mentioned in the tar SlackBuild script, I found this interesting message from 2006:

http://www.mail-archive.com/bug-tar@.../msg00645.html

The question now is, why is U needed?

Say, that's an interesting find. There is this quote:
'tar didn't have such a bug AFAIK'
maybe the responder *didn't* know about the bug or feature. I did lots of tests between tar-1.13 and newer versions and was unable to consistently(or maybe ever) get them to not overwrite linked dirs with real dirs.
After the release of tar-1.1.3, there were no releases for several years and when tar-1.14 came it had huge amounts of changes in there by many different people. Making it more POSIX-compliant was a stated goal once releases started again. Version 1.14 (and 1.15 IIRC) had transitional syntax for non-posix leftovers or other weird syntax. The -l option used with tar-1.13 is no longer used -instead it's '--one-file-system'

In the end, it doesn't matter whether it was bug or whatever -the behaviour of not overwriting linked *dirs* was seen as valuable by PatV. The -U option still unlinked *files* before writing new ones, which is/was good. But, (I assume) Pat didn't want any 'dirty' packages with /usr/share/doc directory overwriting the link which slackware uses to /usr/doc -makepkg includes a routine which warns of their presence when creating packages.

The other problem was/is about getting a proper listing of the archive when writing the database file so that removepkg works properly. Unpacking tar-1.13-created archives with later versions of tar(or maybe it's the other way around) can cause the first two characters of the files' path to be deleted from the listing. Creating archives with later tar versions will cause the files to be listed like this:
./some/path
whereas tar-1.13 would list them like this:
some/path
Using (in makepkg) 'tar * commands' instead of 'tar . commands' with later versions of tar fixes the listing problem. removepkg expects to find:
./
as the first entry, but installpkg will create the listing if not present. The real problem is mixing the version of tar used to create the packages with other versions used to extract the package. If you assume the user has tar>1.15, then the mentioned behaviours and syntax should now be consistent -just hope nobody is using tar-1.14 or tar-1.15.

I invite you to spend a few days checking it all out -I'd love to know I'm not the only one who has done it...

So, -l, -U and the difference between what 'tar -v ???' shows and what the real archive entries are. tar-1.13 stores all the entries with the leading './'. like this:
./
./some/path
but lists them like this:
./
some/path

If you use tar-1.13 to list an archive created with a later version you may get this:
./
ome/path

Hello,

Maybe but, as you say in your further post, it's a problem only if you use tar-1.13 to create the package then an other tar version to install it, or the contrary. I use cpio for both creation and installation, so there is no problem here (removepkg use lists it sorts by itself to remove links, files then directories, so the order of what is written in log files does not matter).

The creation of the tree is managed by a dedicated script, generated during the creation of the package. When you install the package, it is first executed and initializes the directories with the right permissions and ownerships, using mkdir (-p), chown and chmod. So, the CPIO format does the same as tar-1.13 here, in a much more standard way. :)

Maybe I wrongly understand what you mean by meta-data, but installpkg does not seem to preserve it:
You're right, CPIO doesn't know anything about directories. It just tries to do its best if you say it to initialize them, but it's not the stuff it manages (well, maybe not its best, follow the umask instead of set 0700 wouldn't hurt IMHO). But as I manage the directories apart, it's not a problem for me. On the other hand, CPIO has some interesting features like the ability to store non-regular files and to internally check file integrity in the CRC format.

I think you wanted to say "Slackware without pkgtools", didn't you ? :)

Well, you however will need pkgtools, because spkman does not manage the official Slackware package format. That's why under Slackware it's primarily an experience (I realize I probably made a bad translation, "experiment" would be the right term). But maybe it can also interest people who works on Slackware derivates or uses things like LFS (such a trivial package manager suits this kind of distro, IMHO). Moreover, pkgtools are not just the package manager, they are also a part of setup (there are pieces of dialog inside), so it would be hard to wipe them out without rethinking the installer.

Thanks for your good wishes. :)

That was one embarrasing typo. Thanks for pointing it out.