Recording a file-containing package into an eXtended ATTRibute.

Lockywolf · 02-17-2021, 01:05 AM

So, having a look at the makepkg script, the proposal in this thread can basically be implemented like this:

At line 414 of /sbin/makepkg, add

Code:

find ./ -exec setfattr --name=trusted.slackware_v1.package_name "--value=${TAR_NAME}" {} +

And that's it.

The "trusted" namespace is better fitted for the system parameters. I guess, letting users see the name of the package is a security vulnerability.
And it also works on tmpfs, whcih has user namespace xattrs disabled.

This is also very fast. I tested it on a huge WebRTC repo (25Gb on f2fs), and it only took 70 seconds.
For most slackware packages it shouldn't be noticeable at all.
For vfat, I guess, || true can be added to this line, to make it gracefully degrade.

Lockywolf · 05-22-2021, 11:55 PM

With pkgtools-15.0-noarch-41, the installpkg can be modified to obtain the $SUBJ behaviour by adding

Code:

  ( # line 670
      cd $ROOT/
      grep -v '^install' "$TMP/$shortname"  > "$TMP/$shortname"_noinst
      xargs --arg-file="$TMP/$shortname"_noinst --delim='\n' setfattr --name=user.slackware_v1.installpkg.package_name --value="$shortname"
      rm -f "$TMP/$shortname"_noinst
  )

to line 670 of /sbin/installpkg

and makepkg can be modified by adding

Code:

find ./ -type f -exec setfattr --name=trusted.slackware_v1.makepkg.package_name "--value=${TAR_NAME}" {} + # line 414

to line 414 of /sbin/makepkg

The "user" namespace is used at the installpkg time, because I wanted to be less invasive and be friendlier to users. Replace it with "trusted" for (apparently) more security.

makepkg has "trusted" anyway, because my /tmp is on tmpfs, and it does not support the "user" namespace.

Richard Cranium · 05-23-2021, 12:05 AM

Sounds more like something for which one would use a database. https://sqlite.org/index.html perhaps (comes with Slackware!).

Lockywolf · 05-23-2021, 07:02 AM

Quote:

Originally Posted by Richard Cranium

Sounds more like something for which one would use a database. https://sqlite.org/index.html perhaps (comes with Slackware!).

I think that a separate entity (an extra file) would be quite un-slacky. Compare having to maintain database consistency to running a single additional metadata-adding command. The metadata is also automatically invalidated (lost) if a version of pkgtools without xattr-tagging support is used, which would not be the case with any external database.

That is, if you use any other version of pkgtools without the setfattr line, it would just overwrite the file, and xattrs would be lost automatically, no false data preserved.

Richard Cranium · 05-24-2021, 02:20 AM

Quote:

Originally Posted by Lockywolf

I think that a separate entity (an extra file) would be quite un-slacky. Compare having to maintain database consistency to running a single additional metadata-adding command. The metadata is also automatically invalidated (lost) if a version of pkgtools without xattr-tagging support is used, which would not be the case with any external database.

That is, if you use any other version of pkgtools without the setfattr line, it would just overwrite the file, and xattrs would be lost automatically, no false data preserved.

The locate command uses such a database; a cron job could update any such pkgtool related database as well. I'll also add that overwriting the xattrs would create false data to those who expected that information to be present, since it would imply that the file in question was installed by some other means.

OTOH, my grep commands within /var/lib/pkgtools/packages have sub-second real times.

On a virtual machine, I get...

Code:

cranium@currentbuild:/var/lib/pkgtools/packages$ time grep updatedb *
dcron-4.5-x86_64-11:dcron: with cron, such as the nightly indexing with updatedb.
mlocate-0.26-x86_64-4:mlocate: mlocate (locate/updatedb implementation)
mlocate-0.26-x86_64-4:mlocate: mlocate is a locate/updatedb implementation. It keeps a database of
mlocate-0.26-x86_64-4:mlocate: stands for "merging": updatedb reuses the existing database to avoid
mlocate-0.26-x86_64-4:mlocate: rereading most of the file system, which makes updatedb faster and
mlocate-0.26-x86_64-4:etc/updatedb.conf.new
mlocate-0.26-x86_64-4:usr/bin/updatedb
mlocate-0.26-x86_64-4:usr/libexec/mlocate-run-updatedb
mlocate-0.26-x86_64-4:usr/man/man5/updatedb.conf.5.gz
mlocate-0.26-x86_64-4:usr/man/man8/updatedb.8.gz
vim-8.2.2876-x86_64-1:usr/share/vim/vim82/ftplugin/updatedb.vim
vim-8.2.2876-x86_64-1:usr/share/vim/vim82/syntax/updatedb.vim

real    0m0.037s
user    0m0.021s
sys     0m0.015s

On my Slackware64 14.2 machine, the first run took almost 2 seconds against the /var/log/packages directory (0m1.895s real time) to respond (due to iowait). The second run took advantage of the cache and returned in 0m0.022s real time.

Lockywolf · 05-24-2021, 02:27 AM

Quote:

I'll also add that overwriting the xattrs would create false data to those who expected that information to be present, since it would imply that the file in question was installed by some other means.

Valid point. That's why the 'trusted' namespace. But the same argument works for a separate sqlite database.

Quote:

a cron job could update any such pkgtool related database as well

A separate database would be, essentially, a cached version of /var/log/packages.
Fine if you just want to accelerate grep. But the point of the xattr version is not just speed, but also resolving ambiguity for the cases when some packages provide overlapping files.

Lockywolf · 05-10-2023, 10:34 PM

I have been recently asked if my patches still worked, and whether it would be possible to add a time-stamp, to see when a file was installed. I am rewriting the patch here again, for clarity.

1. Makepkg patch:

on like 414, remove the 'find . ..." line, and add

Code:

find ./ -type f -exec setfattr --name=trusted.slackware_v1.makepkg.package_name "--value=${TAR_NAME}" {} +
L_DATE="$(TZ=UTC date --iso=seconds)"
find ./ -type f -exec setfattr --name=trusted.slackware_v1.makepkg.package_date "--value=${L_DATE}" {} +
# Create the package:
xattrs $MTIME -T - -cvf - | $COMPRESSOR > ${TARGET_NAME}/${TAR_NAME}.${EXTENSION}"
find ./ | LC_COLLATE=C sort | sed '2,$s,^\./,,' | tar --no-recursion $ACLS --xattrs $MTIME -T - -cvf - | $COMPRESSOR > ${TARGET_NAME}/${TAR_NAME}.${EXTENSION}

I am adding --xattrs unconditionally here, because it would not make sense to write package metadate only to ignore it at compression time.

The "trusted.*" namespace needs a special switch to getfattr to read it, and is visible only to root (or processes having some CAP_) so I am using

Code:

sudo getfattr -m '.*' -d  "$filename"

to read them.

2. Installpkg patch:

After line 669, add

Code:

  fi
    ( # line 670
      cd $ROOT/
      grep -v '^install' "$TMP/$shortname"  > "$TMP/$shortname"_noinst
      xargs --arg-file="$TMP/$shortname"_noinst --delim='\n' setfattr --name=user.slackware_v1.installpkg.package_name --value="$shortname"
      L_DATE="$(TZ=UTC date --iso=seconds)"
      xargs --arg-file="$TMP/$shortname"_noinst --delim='\n' setfattr --name=user.slackware_v1.installpkg.package_date --value="$L_DATE"
      rm -f "$TMP/$shortname"_noinst
  )

This gives all the installed files the same date, which should be good enough usually, and is much faster than calling "date" for each file.