Why pkgtools still using tar-1.13

ruario · 02-21-2011, 03:24 PM

Quote:

Originally Posted by NonNonBa

Do you know an official Slackware package specification?

Well many aspects of Slackware packages are documented all over the web on numerous sites, like having the slack-desc file for package description (and how this file should be formatted), using files like doinst.sh for post install, and so on and so fourth. But you are right, other stuff like the exact formatting of the tar files is not so well documented. However, isn't that why we are all here discussing it? At least that is why I am here.

Don't get me wrong I am not adverse to using makepkg. I use it often, however (as you now know about me) I like to keep my options open. My primary motivation is I have a script (which I linked to previously) that will convert Opera packages into Slackware format. I wrote it without calling makepkg so that it could be used on any system I happen to be using at the time (Slackware or not). This might also be handy for others who run multiple distros who might want to convert on one distro and install on another. Another nice thing is that because I don't call makepkg I can set the file ownership without running as root. Others who write package creation/conversion utilities (usually ones several orders of magnitude more complex than mine) no doubt have similar needs.

The way I see it is, if you create a tar package that looks so much like the ones that makepkg produce that the other Pkgtools treat them exactly the same way, then you have obviously done this correctly. So 'the spec' is the Pkgtools themselves. I believe you used the same spec to work out the formatting of the package log files for your spack utils right? :P

Anyway, I think I can make packages correctly as it seems to me that the files I make are treated exactly the same way by Pkgtools. However, it is always nice to be certain, which is why I asked gnashley if he can tell the difference since I know he has looked at this as well for src2pkg. Until he (or someone else here) can prove me wrong, I'll continue to believe I understand the 'specification'.

Darth Vader · 02-21-2011, 03:32 PM

You known everything?

Q: Why Slackware move the symlinks creation on the install/doinst.sh?
A: Because TAR-1.13 have an well known old bug which create problems with the stored symlinks. Technically, these symlinks are extracted as something like "zero length files".

ruario · 02-21-2011, 03:47 PM

I didn't say I knew everything and actually, no I didn't know that was the reason ... but now I do!

volkerdi · 02-21-2011, 03:52 PM

Quote:

Originally Posted by Darth Vader

You known everything?

Q: Why Slackware move the symlinks creation on the install/doinst.sh?
A: Because TAR-1.13 have an well known old bug which create problems with the stored symlinks. Technically, these symlinks are extracted as something like "zero length files".

I'm not sure where you heard that, but that's not the way I remember it. I'm not aware of any serious bugs in tar-1.13 that should affect most uses for tar, and none that really impact the pkgtools. If I recall correctly, the original reason that symlink creation was moved to the doinst.sh was to avoid problems from installing packages that were already installed when those packages contain symlinks to directories. If you untar a symlink to a directory and it already exists it drops a dangling symlink into the pointed-to directory. There were some other quirks besides. Now installpkg will remove any existing symlinks that sit at a location that's about to be extracted to, so it might not be as big an issue. But that brings me to my next point, which is that it's a good idea to change the package specification as infrequently as possible. So we still use tar-1.13, and the symlinks are in doinst.sh, and we haven't broken pkgtools recently.

ruario · 02-21-2011, 04:01 PM

@volkerdi: Thanks for the insight. And don't get me wrong, I appreciate that is one of the reasons why things don't break so often on Slackware. Indeed, that is part of the reason I choose Slackware in the first place and why I have come around to the idea of your sticking with tar-1.13 for Slackware's official tools. It does seem like the safest choice.

But I also think it is nice to discuss all of this here, for those of us who want to understand how things work, have slightly more exotic needs or are willing to take some extra risk occasionally.

xflow7 · 09-13-2016, 08:26 PM

Forgive me for opening this old thread, but I've just encountered an issue that appears to arise from the use of tar-1.13 and is kind of a big problem.

I attempted to install the SpiderOakONE backup client and couldn't figure out why it wasn't working. I finally discovered that one filename was truncated on extraction.

spideroak_version_matcher.so was extracted as spideroak_version_matcher.s

I can see that the filename is correctly encoded using tar-1.29

Code:

drw1@nasr:~/Documents/Software/Evaluation$ tar-1.29 tvzf SpiderOakONE-6.1.5-x86_64-1.tgz --wildcards *spideroak_version*
-rw-r--r-- root/root     14312 2016-07-07 08:58 opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.so
-rw-r--r-- root/root       523 2016-07-07 08:57 opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.pyc

However, tar-1.13 only sees it as spideroak_version_matcher.s

Code:

drw1@nasr:~/Documents/Software/Evaluation$ tar-1.13 tvzf SpiderOakONE-6.1.5-x86_64-1.tgz opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.so
tar-1.13: opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.so: Not found in archive
tar-1.13: Error exit delayed from previous errors

drw1@nasr:~/Documents/Software/Evaluation$ tar-1.13 tvzf SpiderOakONE-6.1.5-x86_64-1.tgz opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.s
-rw-r--r-- root/root     14312 2016-07-07 08:58 opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.s

I assume this is related to the length of the filename, although as you can see, this isn't even the longest filename and spideroak_version_matcher.pyc extracts correctly, so who knows.

Has anyone encountered other issues with filename truncation with tar-1.13-based pkgtools?

Dave

ruario · 09-14-2016, 02:46 AM

I'd love to know exactly what archive format is being used for that package (.e.g gnu, oldgnu, v7, ustar or posix), this might explain the file length limitation.

Could you run this command and report back the result:

Code:

bsdtar tvvf SpiderOakONE-6.1.5-x86_64-1.tgz | tail -1

Also run the following and look at the last few lines it should say what format tar is using by default when creating archives:

Code:

tar-1.13 --help

Tar 1.13 should be using "gnu" by default (which is unlimited in terms of file length), unless you have recompiled tar-1.13 yourself and changed this. Also please confirm that you created SpiderOakONE-6.1.5-x86_64-1.tgz with Pkgtools (makepkg) and hence tar-1.13 and not via some other means.

xflow7 · 09-14-2016, 06:50 AM

Hi Ruari,

Thanks for the reply.

The archive does appear to be using GNU format:

Code:

drw1@nasr:~/Documents/Software/Evaluation$ bsdtar tvvf SpiderOakONE-6.1.5-x86_64-1.tgz | tail -1
Archive Format: GNU tar format,  Compression: gzip

tar-1.13 doesn't seem to report format explicitly:

Code:

drw1@nasr:~/Documents/Software/Evaluation$ tar-1.13 --help | tail
  nil, existing   numbered if numbered backups exist, simple otherwise
  never, simple   always make simple backups

GNU tar cannot read nor produce `--posix' archives.  If POSIXLY_CORRECT
is set in the environment, GNU extensions are disallowed with `--posix'.
Support for POSIX is only partially implemented, don't count on it yet.
ARCHIVE may be FILE, HOST:FILE or USER@HOST:FILE; and FILE may be a file
or a device.  *This* `tar' defaults to `-f- -b20'.

Report bugs to <tar-bugs@gnu.org>.

But if I create a little test archive using 1.13, bsdtar reports back that it is GNU format.

In any case, I didn't create the package myself; SpiderOak lists packages for Slackware in their Download area. And, indeed, installpkg reports that it is not created with makepkg, so it's likely they are tarred with a newer version of tar.

In fact, it appears to be a pretty nuanced thing. If I tar . with 1.29 and list with 1.13, all is well:

Code:

bash-4.3# tar-1.29 cf - . | tar-1.13 t | grep spideroak
./opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.pyc
./opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.so

But if I tar * with 1.29 and list with 1.13, the filename gets truncated:

Code:

bash-4.3# tar-1.29 cf - * | tar-1.13 t | grep spideroak
opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.pyc
opt/SpiderOakONE/lib/SpideroakVersionMatcher-1.1-py2.7-linux-x86_64.egg/spideroak_version_matcher.s

Using 1.13 -> 1.13 or 1.29 -> 1.29 works as expected, of course.

So, notwithstanding that the format of the tar archive is notionally the same (GNU), there seems to be some difference between 1.29 and 1.13.

If I untar their package (using tar-1.29) and run makepkg, all is well when I install.

Anyway, looks as though we can chalk this up to not using makepkg, but I wonder how often this happens given that Slackware packages use standard tar extensions, so non-Slacker third-parties wouldn't necessarily suspect that there was a tar-version-dependency.

Edit: I'll also alert the folks at SpiderOak that they ought to be building Slackware packages using pkgtools, and if they can't do that it's probably better to have Slackers use rpm2tgz or something.

Dave

gnashley · 09-14-2016, 12:36 PM

tar-1.13 archives are listed internally like this:
./
./some/dir
./somefile
But, when listing with the '-t' option, it strips the first two characters from the path, except for the initial './' entry. Later versions of tar store the paths internally the same, but do not strip the chars when listing -and the initial dir is also listed differently or missing.