Recording a file-containing package into an eXtended ATTRibute.
SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Recording a file-containing package into an eXtended ATTRibute.
Hello, everyone.
This topic is not actually a question but if someone is feeling adventurous, beta-testers welcome.
I I got tired of grepping /var/lib/pkgtools/packages/* , and wanted to implement the O(1) method of finding "what package this file belongs to".
So I added the following into /sbin/installpkg at line 659 (after the untar-if):
Code:
while read -r
do
#pkg_files+=("$REPLY")
if [[ -f "$ROOT/$REPLY" ]]
then
setfattr -n user.beta.slackware.pkgtools.package \
-v "$shortname" \
-h \
"$ROOT/$REPLY"
fi
done < <(cat "$TMP/$shortname")
I also changed the shebang of the file from "/bin/sh" to "/bin/bash", for the "<(..)" to work. It's possible to rewrite this more portably, but I don't think many people actually have sh that is not bash. If anybody really wants it, ping me.
It's kinda slower for numFiles-bound packages. On kernel-source it's:
Code:
No xattr: Package kernel-source-5.4.46-noarch-1.txz installed.
real 0m48.604s
user 0m44.882s
sys 0m17.139s
xattr: Package kernel-source-5.4.46-noarch-1.txz installed.
real 4m37.083s
user 3m21.302s
sys 1m32.191s
For me the difference of 5 minutes on the package that has probably the largest number of files doesn't matter too much. ( I guess if I had a patch for tar to append an xattr manually, it would have no performance penalty. )
I guess this could also be offloaded to "makepkg", which would only incur this penalty once on the packager's machine.
If anybody finds this useful, I can make this patch for makepkg too, and even suggest including in the mainline.
Tests on filesystems that don't support xattrs are wecome. It's likely that the script will just complain, and nothing more, I have no FAT machines to test.
You can then check the result with getfattr -d filename.
I am not sure if all Linux filesystems supports XATTRs and anyway there are many files which are shipped by multiple Slackware packages.
So, I do not think this is a good idea.
Certainly not all, although I guess all supported by the Slackware installer, except FAT.
I should have potentially called the attribute "last_providing_package", because indeed, there are files provided by multiple packages, but any way "a" file is actually coming from "a particular" package. And even when removepkg removes a package, and does _not_ remove a file due to package intersections, it doesn't _revert_ a file, so I think that keeping a memory that "this file is recorded as belonging to package B that you have in your system, but hey, it actually comes from some other package A that you uninstalled" is valuable.
Quote:
do any other Linux package managers do this?
Eh, sorry, it's been such a long time since I used other distros deep enough. I don't think so.
I I got tired of grepping /var/lib/pkgtools/packages/* , and wanted to implement the O(1) method of finding "what package this file belongs to".
Just wondering? Perhaps I don't correctly understand what you want to do. You can search for files with slackpkg to find out what packages they belong to.
From the slacpkg man page
Code:
file-search
You can search the official Slackware packages for any file in the Slackware distribution. Do you need a strange li‐
brary? Use file-search to find it.
# slackpkg file-search filename
All packages with matching "filename" will be shown, thus you can see whether the packages are installed or not; if
not, you can download and install them with other slackpkg actions.
Here is a search with slackpkg+ installed:
Code:
# slackpkg file-search irqbypass
Looking for irqbypass in package list. Please wait... DONE
The list below shows the packages that contains "irqbypass" file.
[ Status ] [ Repository ] [ Package ]
installed slackware64 kernel-modules-5.4.46-x86_64-1
installed slackware64 kernel-source-5.4.46-noarch-1
upgrade slackware64 kernel-modules-5.4.45-x86_64-1 --> kernel-modules-5.4.46-x86_64-1
upgrade slackware64 kernel-source-5.4.45-noarch-1 --> kernel-source-5.4.46-noarch-1
You can search specific packages using "slackpkg search package".
Here is the same search with slackpkg+ turned off:
Code:
# SLACKPKGPLUS=off slackpkg file-search irqbypass
Looking for irqbypass in package list. Please wait... DONE
The list below shows the packages that contains "irqbypass" file.
[ upgrade ] - kernel-modules-5.4.45-x86_64-1
kernel-modules-5.4.46-x86_64-1 --> kernel-modules-5.4.46-x86_64-1
[ upgrade ] - kernel-source-5.4.45-noarch-1
kernel-source-5.4.46-noarch-1 --> kernel-source-5.4.46-noarch-1
You can search specific packages using "slackpkg search package".
Just wondering? Perhaps I don't correctly understand what you want <...>
You can search specific packages using "slackpkg search package".[/code]
Indeed you can, but slackpkg has no other way to find the package than to look in the database, which basically means to check every file for a line containing the path being searched.
That is the O(n) search where n is the number of packages, and actually O(n*m), where m is the average number of files in a package. The m is not super important, because it's usually fast, but n means n disk accesses.
This's basically caching.
Quote:
Here is the same search with slackpkg+ turned off:
_A_ file comes from _A_ concrete package. Which one does it _actually_ come from, if two packages provide it?
Maybe I just don't understand the xattr benefit, but is the penalty when installing packages (almost 6x slower with your example) really worth what can simply be done with a bash function that can be added per user (.bashrc or .bash_profile) or systemwide (/etc/profile.d/ entry)?
1. The extra time is an academic matter, to be tackled academically. Something is serializing the setfattr operation in the OP. It should be journaled, if the underlying FS supports it. @LockyWolf, what FS is your root?
2. If the FS is FAT, eh, forget it. Or at least, throw the error messages into the bit-bucket.
3. The shellfunc I used a few years ago was simply
Code:
whichpkg () { grep -l "$1" /var/log/packages/* }
It was crude, but it got the job done.
4. @TheRealGrogan, copying or archiving the root partition is a rare matter for most users (actually, admins). Let's say it does happen, that you transfer root to another device's partition. It's up to you, as admin, to (a) make sure the FS supports fattr's, and (b) use the proper parameters to tar (--acls --selinux --xattrs) or rsync (-A -X) to make sure the attributes get copied.
5. There used to be a few editors that weren't file-attribute-aware. Using them to modify files under /etc would cause the files to lose their attributes. (I don't remember what they were, although the line-oriented "ed" comes to mind. Not like anyone uses that interactively.)
Now, I'm going to design some tests, to find out why "setfattr" is adding so much time on individual files.
Last edited by gus3; 06-19-2020 at 07:22 PM.
Reason: should be "grep -l", not "grep -L"
Indeed you can, but slackpkg has no other way to find the package than to look in the database, which basically means to check every file for a line containing the path being searched.
That is the O(n) search where n is the number of packages, and actually O(n*m), where m is the average number of files in a package. The m is not super important, because it's usually fast, but n means n disk accesses.
This's basically caching.
That search for a specific file in kernel-source was fast.
real 0m2.173s
user 0m1.079s
sys 0m1.175s
With slackpkg+ installed you can be more specific in a search:
Code:
# time slackpkg file-search /virt/lib/Makefile
Looking for virt/lib/Makefile in package list. Please wait... DONE
The list below shows the packages that contains "virt/lib/Makefile" file.
[ Status ] [ Repository ] [ Package ]
installed slackware64 kernel-source-5.4.46-noarch-1
upgrade slackware64 kernel-source-5.4.45-noarch-1 --> kernel-source-5.4.46-noarch-1
real 0m1.359s
user 0m1.008s
sys 0m1.046s
Quote:
Originally Posted by Lockywolf
_A_ file comes from _A_ concrete package. Which one does it _actually_ come from, if two packages provide it?
Both packages.
I am still failing to see any advantage in using xattrib. How do you handle files that are located in many packages.
The extra time is an academic matter, to be tackled academically. Something is serializing the setfattr operation in the OP. It should be journaled, if the underlying FS supports it. @LockyWolf, what FS is your root?
F2FS is my root FS.
I am not actually surprised by the 6 times slowdown, it's absolutely expected, since creating a process in Linux is an expensive operation, and each setfattr creates a separate process. I don't think it is filesystem-related.
Again, the much better behaviour would be to only have this slowdown once, on the packager's machine.
Quote:
That search for a specific file in kernel-source was fast.
TWO seconds is "fast"? Are you joking or what?
Code:
root@delllaptop:/usr/src/linux/virt/lib# time slackpkg file-search /virt/lib/Makefile
Looking for virt/lib/Makefile in package list. Please wait... DONE
The list below shows the packages that contains "virt/lib/Makefile" file.
[ Status ] [ Repository ] [ Package ]
installed slackware64 kernel-source-5.4.46-noarch-1
You can search specific packages using "slackpkg search package".
real 0m1.737s
user 0m0.681s
sys 0m0.278s
root@delllaptop:/usr/src/linux/virt/lib# time getfattr -d Makefile
# file: Makefile
user.beta.slackware.pkgtools.package="kernel-source-5.4.46-noarch-1"
real 0m0.004s
user 0m0.002s
sys 0m0.002s
root@delllaptop:/usr/src/linux/virt/lib#
Quote:
Both packages.
What you are saying, contradicts not just the laws of physics, but the laws of mathematics. Set theory specifically.
Quote:
How do you handle files that are located in many packages.
Do you understand that these are not the same file by any standard? In fact, the _only_ thing that they have in common is the basename.
No xattr: Package kernel-source-5.4.46-noarch-1.txz installed.
real 0m48.604s
user 0m44.882s
sys 0m17.139s
xattr: Package kernel-source-5.4.46-noarch-1.txz installed.
real 4m37.083s
user 3m21.302s
sys 1m32.191s
Quote:
Originally Posted by Lockywolf
Code:
root@delllaptop:/usr/src/linux/virt/lib# time slackpkg file-search /virt/lib/Makefile
Looking for virt/lib/Makefile in package list. Please wait... DONE
The list below shows the packages that contains "virt/lib/Makefile" file.
[ Status ] [ Repository ] [ Package ]
installed slackware64 kernel-source-5.4.46-noarch-1
You can search specific packages using "slackpkg search package".
real 0m1.737s
user 0m0.681s
sys 0m0.278s
root@delllaptop:/usr/src/linux/virt/lib# time getfattr -d Makefile
# file: Makefile
user.beta.slackware.pkgtools.package="kernel-source-5.4.46-noarch-1"
real 0m0.004s
user 0m0.002s
sys 0m0.002s
root@delllaptop:/usr/src/linux/virt/lib#
Okay that is fast.
Quote:
Originally Posted by Lockywolf
_A_ file comes from _A_ concrete package. Which one does it _actually_ come from, if two packages provide it?
Quote:
Originally Posted by chrisretusn
Both packages.
Quote:
Originally Posted by Lockywolf
What you are saying, contradicts not just the laws of physics, but the laws of mathematics. Set theory specifically.
virt/lib/Makefile is identical in both kernel-source-5.4.45-noarch-1 and kernel-source-5.4.46-noarch-1
Quote:
Do you understand that these are not the same file by any standard? In fact, the _only_ thing that they have in common is the basename.
Okay then, I'm clueless.
What it looks like to me is you want to include xattrib data to every file in every package that identifies what packages it comes from. Guess I fail to understand the beneficial gain from doing so.
Last edited by chrisretusn; 06-19-2020 at 08:29 AM.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
Recording a file-containing package into an eXtended ATTRibute.
