LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices

Reply
 
Search this Thread
Old 02-10-2011, 05:11 PM   #31
GazL
Senior Member
 
Registered: May 2008
Posts: 3,312

Rep: Reputation: Disabled

Quote:
Originally Posted by gnashley View Post
guanx, cpio is not a good option as it arbitrarily creates new directories chmod 755 if they don't already exist.
Are you sure about that? Unless I'm misunderstanding you that's not my experience. Any missing leading directories will be created with permission based on the umask currently in effect (assuming you specify '-d' to have it create them at all rather than just failing).

Also, if there are any system directories that you absolutely never want changed (such as /usr and /usr/bin etc.) you can exclude them from the archive at creating time.
Here's an example
Code:
root@slack:/tmp/package-cpioexample# find . -exec stat -c "%A %n" {} \;
drwxr-xr-x .
drwxr-xr-x ./var
drwx------ ./var/private_dir
drwxr-xr-x ./lib
drwxr-xr-x ./usr
drwxr-xr-x ./usr/bin
-rwxr-xr-x ./usr/bin/runme
root@slack:/tmp/package-cpioexample# find . -path "." -o -path "./usr" -o -path "./usr/bin" -o -path "./lib" -o -path "./var" -o -print | cpio -ov >/tmp/package.cpio
./var/private_dir
./usr/bin/runme
1 block
root@slack:/tmp/package-cpioexample# cpio -iv --list < /tmp/package.cpio 
drwx------   2 root     root            0 Feb 10 22:38 var/private_dir
-rwxr-xr-x   1 root     root           45 Feb 10 22:37 usr/bin/runme
1 block
As you can see, the shared system directories and the "." directory are not included in the cpio archive at all, but the executable and private directory under /var are in there ready to be extracted. In theory one could set it up such that the only package that contains the system directories would be aaa_base and you have a really safe way of doing packages.

Unless, I'm overlooking something, which is quite possible as it's a big topic with far too many facets to think about in one brain.. - please tell me if I am.

Last edited by GazL; 02-10-2011 at 05:23 PM.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 02-14-2011, 07:28 AM   #32
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Quote:
Originally Posted by gnashley View Post
guanx, cpio is not a good option as it arbitrarily creates new directories chmod 755 if they don't already exist.
I think you are referring to this old bug. However, if you look at the Changelog for GNU cpio you will see this was fixed long ago (version 2.9).

Quote:
Originally Posted by gnashley View Post
The real problem is that links shouldn't be overwritten by a real directory included in the archive(non-posix behaviour 'fixed' in later versions of tar), and that existing directory metadata shouldn't be overwritten. I think the latter may be possible with a combination of later tar options. It would mean *not* using 'U' (unlink first).
The meta data seems fine and symlinks are not overwritten in modern tar as long as you leave out '-U', at least in my own quick tests. Though I'll grant you this is a complex issue so I may have missed something.

I have no problem with sticking with tar-1.13 for Slackware. I have come around to the idea that others pointed out and Patrick confirmed he agrees with, i.e. "It works so why change?". Slackware generally just works because of this no nonsense approach of not changing just for change's sake.

However, I am still interested in this issue because I want to be able to use pkgtools on other distros for these reasons:

1) Repacking and/or quick testing of Slackware packages when not running Slackware
2) I can use pkgtools as a supplementary packager on non-Slackware distros

That second point probably deserves some clarification. For stuff I compile from source I want an easy way to keep track of packages, however making an rpm or deb (or even a Arch PKGBUILD) adds a lot more overhead than using Slackware's makepkg. Also not every source package provides a good (or sometimes any) 'make uninstall'. Yes I know about checkinstall but it has several draw backs. Off the top of my head, a limited selection of package formats (read, no Arch support) and it installs stuff to / first and then makes the package. I want to look at the contents first and perhaps adjust things before installation. I won't comment on your src2pkg as I only have limited familiarity with it. I hear great things but I don't think I need that level of automation personally. Though maybe I am wrong! I would also still have the issue that there will always be some package format you don't support and I use a wide range of distros in my job.

Furthermore using two package managers does not cause issues (like file ownership clashes) as long as you leave the native package manager to look after the main '/usr' structure and leave your secondary manager (pkgtools in my case) to look after your locally created packages in '/usr/local'.

Now the 'correct' way to port pkgtools for use on another distro would be to make as few changes as possible and hence include a copy of tar-1.13. However this means extra build dependencies and a development environment installed on the distro you are porting to. One of the the nice things about pkgtools is that they are just shell scripts using standard utilities. Hence if I can simulate the correct behaviour I would like to do some minor patches to pkgtools so that I can use the native version of tar included with the distros I work with.

I am fairly comfortable with the transform for package creation but I have a vague worry that I may not have considered or noticed everything with regards to extraction.

I should also add I think this discussion is also helpful in case there every comes a day when Patrick does want/need to move from tar-1.13. If we already know the best way to do things with modern tar, he can just lift the examples from this thread.

Last edited by ruario; 02-14-2011 at 07:30 AM. Reason: removed duplicated words
 
Old 02-14-2011, 08:05 AM   #33
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Another option for anyone else considering something along these lines would be to patch only makepkg to use a transform for package creation (since this seems pretty safe) and using spkg for install, upgrade and remove. You don't really need explodepkg since you can just unpack with 'tar xf' and pkgtool itself is just an extra front end. Of course it still requires that you compile spkg, so it may not seem like much of an improvement over compiling tar-1.13 (which has less dependencies anyway). However on the plus side spkg is very quick and should be well tested on the install/upgrade side, since SalixOS uses it. And no it isn't using tar-1.13 on the back end, the author wrote his own tar unpacking routine.
 
Old 02-14-2011, 09:05 AM   #34
wadsworth
Member
 
Registered: Aug 2007
Distribution: Slackware64 13.37
Posts: 214

Rep: Reputation: 64
The "Spack" project has some interesting tools;

spkadd — install or upgrade the new experimental CPIO package format

spkrm — remove the package (of both TAR and CPIO formats)

spkcpio — generate a package in the new experimental CPIO format


Why reinvent the wheel when someone else has already reinvented it?

http://requiescant.tuxfamily.org/pag...105F.html#cpio

Last edited by wadsworth; 02-14-2011 at 09:17 AM.
 
Old 02-14-2011, 12:18 PM   #35
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,746

Rep: Reputation: 458Reputation: 458Reputation: 458Reputation: 458Reputation: 458
I have some experience with spkg -the biggest benefit is when using it to remove (or upgrade) packages.
As far as using a newer version of tar, you are on the right track. I did lots of tests, but it was back when tar latest was about 0.16 or 0.17. As you say, using the '-U' option appears to cause later tar to overwrite links with real dirs. I can't remember exactly what happened about directory meta-data -the oenwerships of a directory in the package shouldn't overwrite the ownership nor perms of an existing dir. The other option used by installpkg when unpacking packages is the '-l' -in tar-1.13 that means '--one-file-system', but in ltare versions it means something else completely.
One other thing to look at is when makepkg writes the tar file -it uses:
'tar name .'. For later versions you should probably use:
'tar name *'

Look closely at the expected listing in the /var/log/packages/* files. The contents should read:
./
relative/path/to
relative/path/to/contents
In later versions of tar, 'tar name .' will get you archives which list content like this (notice no './' topdir and leading ./ for others):
./relative/path/to
./relative/path/to/contents

The upshot is yes, you'd need to do a little tweaking in both the package creation in makepkg and when listing/unpacking them in installpkg.

However, the archives(packages) won't be cross-compatible -at least not with more effort with 'transform' or something. Old-style packages won't be easily handled by you new installer, and new-style packages won't be compatible with old pkgtools.

Let us know what you find out, or get working...
 
Old 02-14-2011, 03:00 PM   #36
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Quote:
Originally Posted by gnashley View Post
However, the archives(packages) won't be cross-compatible -at least not with more effort with 'transform' or something. Old-style packages won't be easily handled by you new installer, and new-style packages won't be compatible with old pkgtools.
I've made packages with only (modern) tar and a simple transform in the past and they install and uninstall just fine with the original pkgtools.

Last edited by ruario; 02-14-2011 at 03:00 PM. Reason: added comment about uninstall
 
Old 02-14-2011, 03:57 PM   #37
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Here is a quickly patched pkgtools for use on non-Slackware based distros for the purposes of quick editing and repacking Slackware packages, or for use managing self built Slack-style packages in /usr/local.

I made the following changes:

1. The scripts are installed into /usr/local/sbin.
2. The man files for scripts are installed into /usr/local/share/man.
3. I removed the checks for tar-1.13 and just use the native tar, without waiting 5 seconds.
4. I altered 'installpkg' to not use '-l' '-U' on extraction.
5. I renamed 'makepkg' and all references to it, to become 's-makepkg'.
6. I altered 's-makepkg' (makepkg) to use a transform on creation and display transformed names.
7. I changed absolute paths such as '/sbin/installpkg' in 'upgradepkg'.
8. I swapped the order of the '-type f' and '-maxdepth 1' find options in removepkg.

I did "2" because this seems like the appropriate location on non-Slackware distros for self-made packages. I did "5" because of a potential name clash on Arch and distros that use Arch pkgtools, which provide their own makepkg utility. I did "8" because on some distros (e.g. Arch), find prints a warning if you have them the other way around.

Here is archive containing the original pkgtools and a very simple script to repackage them along the lines listed above:

pkgtools-13.1-src-3.tar.gz

On a non-Slackware based distro (there is no point on Slackware or derivatives since you could just use the original utilities) you could install them as follows:

Code:
$ tar xf pkgtools-13.1-src-3.tar.gz
$ cd pkgtools-13.1-src
$ ./createpkg
$ su
# pkg/usr/local/sbin/installpkg pkgtools-13.1-noarch-3ro.tgz
Everything would then be installed as:

Code:
PACKAGE NAME:     pkgtools-13.1-noarch-3ro
COMPRESSED PACKAGE SIZE:     32K
UNCOMPRESSED PACKAGE SIZE:     110K
PACKAGE LOCATION: pkgtools-13.1-noarch-3ro.tgz
PACKAGE DESCRIPTION:
pkgtools: pkgtools (The Slackware package maintenance system)
pkgtools:
pkgtools: This package contains utilities for handling Slackware packages.
pkgtools: Included are the command line utilities 'installpkg', 'removepkg',
pkgtools: 's-makepkg', 'explodepkg', and 'upgradepkg' that install, remove,
pkgtools: build, examine, and upgrade software packages.  Also included are 
pkgtools: 'pkgtool', a menu based program for installing packages, removing
pkgtools: packages, or viewing the packages that are installed on the system,
pkgtools: documentation (man pages), and a few other system admin scripts.
pkgtools:
pkgtools:
FILE LIST:
./
usr/
usr/local/
usr/local/share/
usr/local/share/man/
usr/local/share/man/man8/
usr/local/share/man/man8/explodepkg.8.gz
usr/local/share/man/man8/installpkg.8.gz
usr/local/share/man/man8/s-makepkg.8.gz
usr/local/share/man/man8/pkgtool.8.gz
usr/local/share/man/man8/removepkg.8.gz
usr/local/share/man/man8/upgradepkg.8.gz
usr/local/sbin/
usr/local/sbin/installpkg
usr/local/sbin/explodepkg
usr/local/sbin/removepkg
usr/local/sbin/pkgtool
usr/local/sbin/upgradepkg
usr/local/sbin/s-makepkg
install/
install/slack-desc
I could then install and use pkgtools to track my source installs. For example if I wanted to install the Karmen window manager on a distro that does not include it (like Ubuntu). I could simply issue the following:

Code:
$ wget http://downloads.sourceforge.net/karmen/karmen-0.15.tar.gz
$ tar xf karmen-0.15.tar.gz
$ cd karmen-0.15
$ ./configure && make && make DESTDIR=`pwd`/staging install
$ cd staging
$ sudo s-makepkg -c y ../karmen-0.15-1686-1.tgz
$ sudo installpkg ../karmen-0.15-1686-1.tgz
Note: Yes this is quick and dirty but my intention here is only to show that it is fairly easy to use pkgtools on non-Slackware based distros to track and install source packages installed into /usr/local, comparative to making a .deb (or .rpm for that matter).

And if I later decide to remove karmen:
Code:
$ sudo removepkg karmen
or perhaps the pkgtools themselves:
Code:
$ sudo removepkg pkgtools
Of course I still don't know for 100% that my changes to installpkg (see "4" above) are sufficient but I guess after playing with this for a few weeks I probably will.

P.S. I realise (but should perhaps point out for the benefit of anyone who doesn't) that it wouldn't be wise to try and make brand new packages on say Ubuntu and then try to use them on Slackware as I am likely to end up depending on some lib that either doesn't exist on Slackware or is of a different version. However I don't think it will be a problem to repackage or edit precompiled (binary) packages since I have already tested the transform trick when making Opera packages as I mentioned previously.

Last edited by ruario; 02-16-2011 at 05:19 AM. Reason: made a slight tweak to source package and linked to new version
 
Old 02-14-2011, 04:06 PM   #38
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Quote:
Originally Posted by wadsworth View Post
The "Spack" project has some interesting tools;

spkadd install or upgrade the new experimental CPIO package format

spkrm remove the package (of both TAR and CPIO formats)

spkcpio generate a package in the new experimental CPIO format


Why reinvent the wheel when someone else has already reinvented it?

http://requiescant.tuxfamily.org/pag...105F.html#cpio
If I wanted to stray that far from the original utils I'd go with spkg and get the speed benefit.
 
Old 02-15-2011, 12:20 PM   #39
NonNonBa
Member
 
Registered: Aug 2010
Distribution: Slackware
Posts: 61

Rep: Reputation: 21
Quote:
Originally Posted by wadsworth
Why reinvent the wheel when someone else has already reinvented it?
You hit a bit rough... Spack uses its own package layout, provides a quite different UI, and doesn't require anything in Slackware to work; the only thing it shares with the Slackware's pkgtools is the generated package log file.

Quote:
Originally Posted by ruario
If I wanted to stray that far from the original utils I'd go with spkg and get the speed benefit.
Are you sure? deal with the package DB doesn't consume so much time than that, the "strangling node" is the decompression of the package, where you can't go faster than gzip/xz go. I didn't test, but I would say that your gain are not significative (countable in seconds) unless you do a huge install at once (what you never do in fact, as the Slackware installer use the pkgtools) or you make an upgrade, as upgradepkg installs each new package twice...
 
Old 02-15-2011, 01:49 PM   #40
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,746

Rep: Reputation: 458Reputation: 458Reputation: 458Reputation: 458Reputation: 458
As I mentioned, the real speec gain with spkg is when removing or upgrading packages.
 
Old 02-16-2011, 03:34 AM   #41
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Quote:
Originally Posted by ruario View Post
Can't you just use a transform?
Code:
tar --transform "s,^\./\(.\),\1," --show-transformed-names -cvf- . | xz -c > ../example-1.0-i486-1.txz
./
usr/
usr/man/
usr/man/man1/
usr/man/man1/example.1.gz
usr/bin/
usr/bin/example
usr/doc/
usr/doc/example-1.0/
usr/doc/example-1.0/COPYING
usr/doc/example-1.0/ChangeLog
usr/doc/example-1.0/README
install/
install/slack-desc
Or here is an alternative hack to get the file names in tar-1.13 style:

Code:
tar --no-recursion -cvf- . `find *` | xz -c > ../example-1.0-i486-1.txz
./
install/
install/slack-desc
usr/
usr/man/
usr/man/man1/
usr/man/man1/example.1.gz
usr/bin/
usr/bin/example
usr/doc/
usr/doc/example-1.0/
usr/doc/example-1.0/COPYING
usr/doc/example-1.0/ChangeLog
usr/doc/example-1.0/README
However it is not really as good because for a big package you might hit a problem with the argument list being too long. On the other hand '--no-recursion' is an option that has been around for over 10 years, whilst '--transform' has only been around since GNU tar 1.20 from 2008-04-14. Also it is worth noting that even Slackware 12.2 didn't have tar-1.20+, it only had 1.16 (and 1.13 obviously).

Last edited by ruario; 02-20-2011 at 07:23 AM. Reason: added more detail about the history of --no-recursion
 
Old 02-19-2011, 10:42 AM   #42
NonNonBa
Member
 
Registered: Aug 2010
Distribution: Slackware
Posts: 61

Rep: Reputation: 21
Quote:
Originally Posted by gnashley
As I mentioned, the real speec gain with spkg is when removing or upgrading packages.
Even when removing? I didn't think all the rm calls consumed so much time... well, why not after all, but removing is however something you rarely massively do.

Don't misinterpret what I say, though. I don't mean spkg is worthless, just that speed is IMHO not a good criteria to choose, as it's not something you can really enjoy in daily operations. The stability (C makes things not depend on command outputs) and the quality of the UI are more meaningful, I think.
 
Old 02-19-2011, 11:04 AM   #43
Darth Vader
Member
 
Registered: May 2008
Location: Romania
Distribution: DARKSTAR Linux 2008.1
Posts: 593

Rep: Reputation: 108Reputation: 108
Working on my pet C++ project called pkgbuild, I written an C++ API on top of libarchive, able to create and extract tarballs using the Slackware specifications. Yeah, pkgbuild is capable to create several tarballs, from a single directory tree, because it internal handle the tarballs.

So, I know very well the secrets of Slackware tarball-ing. So, I can say, the great secret of "Why TAR-1.13?" is only the directory symlink management and the store method of filenames.

If people really need, I can write an "pkgextract", able to automatically handle all compression methods suported by libarchive, and an "pkgcreate", able to create TAR/TGZ/TBZ/TLZ/TXZ tarballs using only the switches from command line.
 
Old 02-19-2011, 02:29 PM   #44
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,746

Rep: Reputation: 458Reputation: 458Reputation: 458Reputation: 458Reputation: 458
NonNonBa, it's not the 'rm' calls themselves that take so long -it's the chceking to see if the file is claimed by any other package before removing it that takes so much 'grep file /var/log/packages/*' for every file a package contains makes a lot of work...

Thanks very much to ruario for spending so much time on all this and providing some alternatives. And, Darth Vader I'd love to see something built on libarchive which handles this stuff gracefully -but please make it in C, not C++ -we don't want an installer that needs libstdc++...
 
Old 02-19-2011, 04:26 PM   #45
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 1,797

Rep: Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803Reputation: 803
Quote:
Originally Posted by gnashley View Post
Thanks very much to ruario for spending so much time on all this and providing some alternatives.
He he ... I think you give me too much credit. You seem to have done the bulk of testing and also worked out that '-U' was no longer needed. So thaks to you!

As a side note, to amuse myself and also to further prove "There is more than one way to skin a cat", I managed to think of yet another way to get the files stored "tar-1.13 style". Funnily enough it was wadsworth's mention of NonNonBa's spack and its use of cpio.

Before I give the example though just some quick thoughts for wadsworth and NonNonBa. I installed and looked at the spack tools very breifly. Firstly I don't really see the point of replacing the tar format with cpio and secondly I don't personally like the packages that the spack tools make. In my eyes the internal structure is overly complex, which means that making small edits to them after creation suddenly becomes a major hassle unless you entirely commit yourself to using the spack tools. I really like Slackware packages because I can just un tar them, tweak them, and then tar them up again. I don't need anything unusual preinstalled. In fact I don't even need to run Slackware as my host distro when I make the edits since (as I demonstrated earlier) you can just use any modern version of tar as long as you pick the right options (or even cpio as I will demonstrate below). To be clear however, that is not to say what NonNonBa has done isn't clever. It is just a statement that spack isn't for me 'personally'. If someone else finds it useful then great!

Anyway, back to what I was saying with regards to cpio. It occured to me that since cpio can optionally output in UStar format instead of one of the native cpio formats, you could actually use have it make the tar package.

In a way the following example kind of combines both the earlier tricks I used with (modern) GNU tar. I use the regex that I had previously used in the transform example and get the file/directory list via the find command, like the second example.

So, how about this?

Code:
find ./ | sed "s,^\./\(.\),\1," | cpio --quiet -ovHustar | xz -c > ../example-1.0-i486-1.txz
./
usr/
usr/man/
usr/man/man1/
usr/man/man1/example.1.gz
usr/bin/
usr/bin/example
usr/doc/
usr/doc/example-1.0/
usr/doc/example-1.0/COPYING
usr/doc/example-1.0/ChangeLog
usr/doc/example-1.0/README
install/
install/slack-desc
As I said earlier, this is more for fun but you never know it might be useful, should PatV ever switch to using a transform and then the GNU tar maintainer does something odd like change the way transforms work (in some non compatible way). Whilst that seems exceedingly unlikely, it is always nice to have a backup method.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
BackUp & Restore with TAR (.tar / .tar.gz / .tar.bz2 / tar.Z) asgarcymed Linux - General 5 12-31-2006 02:53 AM
Slack 10.2 pkgtools in 10.0 linuxhippy Slackware 3 01-17-2006 09:54 AM
pkgtools Nightmare HELP!!!!!!!!!! waterox Slackware 4 09-12-2005 12:46 PM
pkgtools problem??? brbrow Slackware 2 09-10-2003 01:58 PM


All times are GMT -5. The time now is 07:01 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration