LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (https://www.linuxquestions.org/questions/slackware-14/)
-   -   How can removepkg be triggered to remove stuff placed by a doinst.sh? (https://www.linuxquestions.org/questions/slackware-14/how-can-removepkg-be-triggered-to-remove-stuff-placed-by-a-doinst-sh-4175672073/)

brobr 03-27-2020 09:09 AM

How can removepkg be triggered to remove stuff placed by a doinst.sh?
 
I ran in the same problem as described in this link
execute-script-after-removepkg-in-slackware.

That is, say after installing a jupyter nbextension one has to run extra commands to get links etc. made that connect the installed program to the nbextension machinery. The extra commands can be run from the doinst.sh.

However, after removepkg of the installed package the stuff (links, folders etc) put in place by those extra commands from doinst.sh need to be removed as well. (The same holds for for path-extension scripts put in /etc/profile.d, that have to be manually removed after a package).

Is there a way (in the Slackbuild or doinst.sh) to get these extra bits added to the description of the installed pkg so that they can be cleaned out by removepkg?

tadgy 03-27-2020 12:30 PM

Quote:

Originally Posted by brobr (Post 6104956)
However, after removepkg of the installed package the stuff (links, folders etc) put in place by those extra commands from doinst.sh need to be removed as well. (The same holds for for path-extension scripts put in /etc/profile.d, that have to be manually removed after a package).

Is there a way (in the Slackbuild or doinst.sh) to get these extra bits added to the description of the installed pkg so that they can be cleaned out by removepkg?

I have often suffered from the same problem - I don't like turds (my name for untracked files) on the filesystem, and often wished that removepkg would clean up the stuff doinst.sh renames from .new etc.

What packages (and removepkg itself) need is a 'douninst.sh' script that's executed after package removal.
It wouldn't be too hard to code it in to removepkg, but as to whether Pat would accept such a patch....

volkerdi 03-27-2020 02:19 PM

Quote:

Originally Posted by tadgy (Post 6105014)
I have often suffered from the same problem - I don't like turds (my name for untracked files) on the filesystem, and often wished that removepkg would clean up the stuff doinst.sh renames from .new etc.

Might as well not have *.new files then - you'd end up with all your config files replaced whenever you used upgradepkg.

ehartman 03-27-2020 02:23 PM

Quote:

Originally Posted by brobr (Post 6104956)
those extra commands from doinst.sh need to be removed as well.

Links, strictly created with the syntax, in doinst.sh
Quote:

( cd usr/sbin ; rm -rf in.proftpd )
( cd usr/sbin ; ln -sf proftpd in.proftpd )
(note the double cd commands and the enclosing within parentheseses)
will be removed by removepkg.
Anything moved into place (or not) by a call to the config function will not as removepkg can never be sure if that file is the default one out of the install package or a modified one (config() only moves it over when the file wasn't there yet, so at a first installpkg, normally never after an updatepkg). When config() leaves a .new file, that one will be removed, though.
In principle removepkg will never remove files and/or directories that may have been modified by the administrator.

tadgy 03-27-2020 03:46 PM

Quote:

Originally Posted by volkerdi (Post 6105057)
Might as well not have *.new files then - you'd end up with all your config files replaced whenever you used upgradepkg.

I said it was a wish, not something that could be easily implemented :)

tadgy 03-27-2020 03:48 PM

Quote:

Originally Posted by tadgy (Post 6105014)
What packages (and removepkg itself) need is a 'douninst.sh' script that's executed after package removal.
It wouldn't be too hard to code it in to removepkg, but as to whether Pat would accept such a patch....

Is this something you'd like to see in Slackware, Pat?

If so, I'm sure some of us could come up with some workable patches... :)

volkerdi 03-27-2020 04:09 PM

Quote:

Originally Posted by tadgy (Post 6105087)
Is this something you'd like to see in Slackware, Pat?

If so, I'm sure some of us could come up with some workable patches... :)

To be honest, not particularly. This is hardly the first time the idea has been suggested, and a few leftover config files don't really seem to cause much harm in my humble opinion.

brobr 03-27-2020 05:03 PM

Quote:

In principle removepkg will never remove files and/or directories that may have been modified by the administrator.
Yes this makes perfect sense and the fact that .new files are kept being created/found after upgrading packages is great as these signal that things could have been changed. I am impressed with the way for example /etc/group and /etc/shadow always accommodate novelties apart from having .new files going along with these that flag this. Most of these, though, concern packages that need to be ok to make everything work. And with 'ls *.new' they are easily found.

But what about the less essential stuff, say that comes with third party packages that are installed with build.scripts like those from SBo (These things can accumulate quite quickly, getting beyond a few leftover config files)? In most cases when one removes these (not upgrade) one is also no longer interested in old config files (if so, one would make a back-up). Bits of these packages can thus be kept lying around if these are not part of the package description, like files in /etc/profile.d/ that end up there via a doinst.sh. These often do not have a nice 'new' flag attached to them that makes them quickly traceable...

Well, questions may not be new, nor so are some wishes. But novel solutions to old looking questions could be interesting. How would a mechanism look like that 'document' what is installed outwith the package. This 'document' (or 'douninst.sh') could then be the basis - when removing a package - for informing (the most riskless scenario) that some bits are still left behind (that then can easily be dealt with 'by hand') or (more risky) could lead to their removal as part of removepkg. As described by e.hartmann, for links, such a mechanism is in place; but could this be done with/for other stuff?

bifferos 03-27-2020 05:36 PM

Quote:

Originally Posted by tadgy (Post 6105087)
Is this something you'd like to see in Slackware, Pat?

I'd like to see this, for the simple reason that it's fiddly to bolt on. You either wrap the pkg tools, or hack them, neither are particularly nice. It doesn't mean Slackware core distro has to use the feature (and I'd honestly prefer that it didn't).

'Repackaging' Slackbuilds could benefit because it could be a third-party provided script that does the install and it may provide an uninstall option. Install may involve a kernel module compile which you don't want to bake into your package. It's part of what makes Slackware so flexible that the packaging system doesn't care what happens in doinst.sh and you can bend it any way you like but in missing a douninst.sh there feels like a bit of a gap.

Didier Spaier 03-31-2020 06:37 AM

Quote:

Tue Mar 31 04:00:43 UTC 2020 a/pkgtools-15.0-noarch-31.txz: Rebuilt. removepkg: support an uninstall script. See removepkg(8).
In the new removepkg:
Quote:

# Tue Mar 31 03:06:25 UTC 2020
# Support an uninstall script to be executed when the package is removed.
# The script should be a standard sh script with the same name as the package
# (without the .txz or other extension), and should be installed in
# /var/lib/pkgtools/douninst.sh.

brobr 03-31-2020 11:30 AM

Yes, Thanks Pat, noticed this this morning ;-)

bormant 03-31-2020 11:43 AM

Quote:

Tue Mar 31 04:00:43 UTC 2020
a/pkgtools-15.0-noarch-31.txz: Rebuilt.
removepkg: support an uninstall script. See removepkg(8).
Quote:

# Tue Mar 31 03:06:25 UTC 2020
# Support an uninstall script to be executed when the package is removed.
# The script should be a standard sh script with the same name as the package
# (without the .txz or other extension), and should be installed in
# /var/lib/pkgtools/douninst.sh.
@volkerdi
Will installpkg copy install/douninst.sh from package tree to $ADM_DIR/douninst.sh/$shortname as for install/doinst.sh:
Code:

  if [ -r $ROOT/$INSTDIR/doinst.sh ]; then
    cp $ROOT/$INSTDIR/doinst.sh $ADM_DIR/scripts/$shortname


volkerdi 03-31-2020 12:55 PM

Quote:

Originally Posted by bormant (Post 6106305)
@volkerdi
Will installpkg copy install/douninst.sh from package tree to $ADM_DIR/douninst.sh/$shortname as for install/doinst.sh:
Code:

  if [ -r $ROOT/$INSTDIR/doinst.sh ]; then
    cp $ROOT/$INSTDIR/doinst.sh $ADM_DIR/scripts/$shortname


No. It's up to the SlackBuild to put it in /var/lib/pkgtools/douninst.sh/ with the proper name, which (by the way) is not the $shortname. This isn't much increased burden for creating a SlackBuild, but makes it so that installpkg doesn't need any changes and the uninstall scripts are tracked by the existing package file database.

tadgy 03-31-2020 02:46 PM

Quote:

Originally Posted by brobr (Post 6106302)
Yes, Thanks Pat, noticed this this morning ;-)

Seconded!

Really appreciated, Pat :)

volkerdi 03-31-2020 07:04 PM

Quote:

Originally Posted by volkerdi (Post 6106316)
No. It's up to the SlackBuild to put it in /var/lib/pkgtools/douninst.sh/ with the proper name, which (by the way) is not the $shortname. This isn't much increased burden for creating a SlackBuild, but makes it so that installpkg doesn't need any changes and the uninstall scripts are tracked by the existing package file database.

I was wrong, it is the $shortname after all... I'd completely forgotten that it's not the shorter shortname.

Sorry! :)

gus3 03-31-2020 10:44 PM

Quote:

I'd completely forgotten that it's not the shorter shortname.
That sounds like a 21st century situation. Hopefully, it'll be resolved before the 2037 problem.

volkerdi 03-31-2020 10:48 PM

Quote:

Originally Posted by gus3 (Post 6106490)
That sounds like a 21st century situation. Hopefully, it'll be resolved before the 2037 problem.

2038. We have an off-by-one error.

bormant 04-01-2020 01:45 AM

Quote:

Originally Posted by volkerdi (Post 6106316)
No. It's up to the SlackBuild to put it in /var/lib/pkgtools/douninst.sh/ with the proper name

So if anyone rename package file for any reason she breaks this feature. Until now there was only cosmetic "breakage" for this scenario -- lost description from slack-desc and only if shorter shortname is changed.

Another possible scenario for installpkg:
The "half consistent" way for install/douninst.sh is
1) copy it (in installpkg) to /var/lib/pkgtools/douninst.sh/$shortname as for install/doinst.sh and
2) add (echo >>) it's name to /var/lib/pkgtools/packages/$shortname file, so removepkg deletes it as any other regular installed file.

GazL 04-01-2020 03:37 AM

A removepkg --skip-douninst flag to allow control over this new post-processing feature might not be a bad idea.

bifferos 04-01-2020 12:21 PM

Quote:

Originally Posted by GazL (Post 6106558)
A removepkg --skip-douninst flag to allow control over this new post-processing feature might not be a bad idea.

I think if I needed this option I'd be paying a visit to the script directory sooner or later to sort things out. Probably better if it's sooner.

bifferos 04-01-2020 07:07 PM

I added a douninst.sh script to my pypi slackbuild generator and it seems to work much as expected. In this example, the doinst.sh script does the 'pip install' and the douninst.sh does the 'pip uninstall'. You may have good reason to not want your module installs working like this, but at least you have the option now.

https://github.com/bifferos/afterpkg...ce961af493be24

bormant 04-02-2020 09:01 AM

@volkerdi
This half-consistent way for install/doinst.sh and install/douninst.sh can be added with:
Code:

--- a/installpkg        2019-10-04 09:05:27.000000000 +0300
+++ b/installpkg        2020-04-02 17:02:34.359923861 +0300
@@ -697,9 +697,14 @@
      cp $ROOT/$INSTDIR/doinst.sh $ADM_DIR/scripts/$shortname
      chmod 755 $ADM_DIR/scripts/$shortname
    fi
+    if [ -r $ROOT/$INSTDIR/douninst.sh ]; then
+      cp $ROOT/$INSTDIR/douninst.sh $ADM_DIR/douninst.sh/$shortname
+      chmod 755 $ADM_DIR/douninst.sh/$shortname
+      echo var/lib/pkgtools/douninst.sh/$shortname >> $ADM_DIR/packages/$shortname
+    fi
    # /install/doinst.sh and /install/slack-* are reserved locations for the package system.
    # Heh, not any more with a recent tar :-)
-    ( cd $ROOT/$INSTDIR ; rm -f doinst.sh slack-* 1> /dev/null 2>&1 )
+    ( cd $ROOT/$INSTDIR ; rm -f doinst.sh douninst.sh slack-* 1> /dev/null 2>&1 )
    rmdir $ROOT/$INSTDIR 1> /dev/null 2>&1
  fi
  # If we used a scan directory, get rid of it:


NonNonBa 04-04-2020 05:00 AM

Hi,

Sorry to come after the battle, but if we agree the stuff left by a package is just a loss of bits crowding some dirs, wouldn't it be more elegant to leave this to the local admin. A general hook would IMHO be more flexible and powerful. We could imagine, let's say /etc/postpkgtools.sh which would be ran after ANY package-related operation:

Code:

# A package was added
/etc/postpkgtools.sh LONG_NEW_PACKAGE_NAME
# A package was upgraded
/etc/postpkgtools.sh LONG_NEW_PACKAGE_NAME LONG_OLD_PACKAGE_NAME
# A package was removed
/etc/postpkgtools.sh - LONG_OLD_PACKAGE_NAME

It is easy for an administrator to know which file is moved from /var/.../scripts and to determine if it's worth to be removed. Plus, this would allow to automate (among ANYTHING else) the update of the kernel initrd, which could interest many users. :twocents:

bifferos 04-04-2020 11:04 AM

I don't think it's just about some files left behind. What about users? Groups? Could these represent security vulnerabilities? What about generated kernel modules that may get loaded and then forgotten about long after the package that created them has been removed?

I'm unsure as to why you'd want package-specific behaviour put in a system-level file like that. Won't that mean /etc/postpkgtools.sh ends up as a mass of hard to maintain if clauses? Or did you imagine some additional sub-system to execute per-package helpers, e.g. /etc/postpackagetools.d?

IMHO the current implementation is pretty much perfect. It's the absolute minimum you could get away with and that's frequently the best solution. I don't care about package renames, and don't really understand the post from bormant. Is that happening all the time? When did it last happen? Would the package that it happened to have benefited from douninst.sh in the first place?

For kernel initrd, isn't that something that could be done in the doinst.sh (assuming there's desire), and not really anything to do with douninst.sh?

NonNonBa 04-05-2020 08:14 AM

I don't see any issue with groups and users. Unless you do a full install and manually merge your passwd/groups with updated defaults, you always have useless groups and users on a system. Plus, it is far easier to find the leftovers of a removed service if you have still a user name instead of just an empty UID number.

Touching what is loaded, it is the same issue when you upgrade, let's say, openssl. You have to kill and respawn everything is using it in order the old code to be actually removed. I don't think you would want the doinst.sh to even try to do that for you, as you certainly wouldn't call the glibc upgrade going on its own to telinit 1 a great feature.

The fact is that for 25+ years this mechanism hasn't been required by anything, so maybe the rare corner cases for it can be left to the admin (also notice the doinst.sh can handle the upgrading process, this is really useful only when you definitively remove something) rather than adding code you will have to care about as the pkgtools will evolve. That's why I think it's a better idea to just allow to the admin to automate what he already manually does after the package manager has completed its job (whatever it is, not just removing). And yes, the admin could split things in /etc/postpkgtools.d, as well as running any crazy things he wants from the script, that's the idea, but I'd be surprised this one would grow that big. :)

There are also some issues with the current mechanism, as now a removal can break something as soon as two packages share files, which was till now hardly thinkable (only when a tracked file in a package is generated in another). For example, the user has ABC installed, but gives a try to CDE, which is its forks. Then he chooses to keep the last and removes the former, but breaks all, because both generate a required file from the doinst that has been undoinsted since the packager of ABC couldn't even know it would be forked when he did the package.

That's why I think we now have a more complex and weaker code than before, for an unobvious gain. Hence my (IMHO very Slackware) suggestion: trust the admin and let him hammer his own fingers (Le métier qui rentre !, as we say by there).

bifferos 04-05-2020 05:12 PM

Quote:

Originally Posted by NonNonBa (Post 6108021)
The fact is that for 25+ years this mechanism hasn't been required by anything

How would you know?

Quote:

There are also some issues with the current mechanism, as now a removal can break something as soon as two packages share files
I'm unsure why you'd want two packages to share files. Isn't that something to be avoided? Or, perhaps, packages that share files simply don't use douninst.sh? You're not obliged to use it.

Quote:

That's why I think we now have a more complex and weaker code than before, for an unobvious gain.
I don't consider the removepkg change as complex or weak, so I guess you mean the *use* of that code will be complex. If there is no obvious gain, then nobody will use it. If nobody uses it, then there's no complex, weaker code. It just does nothing. I think you're being a little paranoid if you think everyone's going to start using this overnight, given that most slackbuild maintainers have other jobs and want to do the minimum that keeps their builds working. There is little use for it in the core distro either because the maintenance tasks there are well established and documented and nobody wants to change workflows that have been in place for decades. Personally I'll be using this in my own private package builds as my SBo ones have no use for it, but rest assured I will be using it, and it will help me a great deal.

Quote:

Hence my (IMHO very Slackware) suggestion: trust the admin and let him hammer his own fingers
You could also trust the package maintainer to write a pragmatic douninst.sh. Clearly if I fill it with daft code it's not going to work properly. There's a solution to that....:hattip:

Richard Cranium 04-05-2020 05:18 PM

Quote:

Originally Posted by bifferos (Post 6108217)
I'm unsure why you'd want two packages to share files. Isn't that something to be avoided? Or, perhaps, packages that share files simply don't use douninst.sh? You're not obliged to use it.

I guess you'd be surprised on some of the things that happen in Slackware as it is delivered. There's a reason why removepkg checks to see if a file also belongs in another package from the one being removed.

bassmadrigal 04-05-2020 06:46 PM

Quote:

Originally Posted by bifferos (Post 6108217)
I'm unsure why you'd want two packages to share files. Isn't that something to be avoided? Or, perhaps, packages that share files simply don't use douninst.sh? You're not obliged to use it.

Quote:

Originally Posted by Richard Cranium (Post 6108221)
I guess you'd be surprised on some of the things that happen in Slackware as it is delivered. There's a reason why removepkg checks to see if a file also belongs in another package from the one being removed.

It probably also wouldn't be very hard to do a sanity check of any potentially removed files in the douninst.sh to ensure they don't belong to any other packages. This is already done with .new files in the doinst.sh. There's probably a better, cleaner, or more efficient way to do this, but something like this should work:

Code:

checkdup() {
  REMFILE="$1"
  count=0

  for i in /var/lib/pkgtools/packages/*; do
    if grep -q "$REMFILE" "$i"; then
      ((count++))
    fi
  done

  if [ $count == 0 ]; then
    rm -f "$REMFILE"
  else
    echo "$REMFILE was found in $count package(s). Skipping."
  fi
}

checkdup etc/mycustomfile


NonNonBa 04-06-2020 07:02 AM

Quote:

Originally Posted by bifferos (Post 6108217)
How would you know?

Because "required" would mean it would be there yet, at least historically used by one official package. :)

Quote:

Originally Posted by bifferos (Post 6108217)
Personally I'll be using this in my own private package builds as my SBo ones have no use for it, but rest assured I will be using it, and it will help me a great deal.

Then you act as an admin, not a packager (who is per se doing things for others). This is exactly what my suggestion allows you to implement among any other things:

Code:

if [ "$1" = "-" ] && [ -f "/path/to/uninstall/$2" ]; then
  ( cd /; sh "/path/to/uninstall/$2" )
  mv "/path/to/uninstall/$2" "/path/to/was_uninstall/$2"
fi

Quote:

Originally Posted by bifferos (Post 6108217)
You could also trust the package maintainer to write a pragmatic douninst.sh.

You can't trust the packager, that's my point, because a packager just can't know nor anticipate what the other packagers do or will do, as there is no way to guess what a doinst generates (precisely the targets of the undoinst). More generally, except in some rare cases, a packager can't really know what an admin will want to keep or not and is then always prone to fight him.

Quote:

Originally Posted by bassmadrigal (Post 6108243)
It probably also wouldn't be very hard to do a sanity check of any potentially removed files in the douninst.sh to ensure they don't belong to any other packages.

Sure, but you see? You're then partially re-implementing the core functionality of removepkg in a file ran from removepkg.

bifferos 04-06-2020 08:43 AM

Quote:

Originally Posted by NonNonBa (Post 6108373)
Because "required" would mean it would be there yet, at least historically used by one official package. :)

'required' and 'available' are two different concepts. Just because something is required doesn't mean it has to be available. I require it. I don't really understand your argument. Are you telling me I don't require it? How do you know what I require? Granted you can tell me how to achieve what I'm trying to do in some other way but what if I don't want to do it that way?

One could argue that installpkg is not really 'required' couldn't one? You could just make do with the tar command coupled with some manual steps and it'd be great because you as admin would know what's going on, etc..etc.. but at some point one wants to automate things. Slackware has always trodden a fine line between what it does and doesn't do for the admin and I doubt very much everyone will agree on where that line should be it's an ongoing conversation. What I don't understand is why you seem to be telling me I can't/shouldn't use any uninstall steps, and how it creates some kind of problem for either you or the wider Slackware community. It doesn't, this is simply my choice for my own systems.

Quote:

You can't trust the packager, that's my point
Then why would you use his packages? When you download the package, grep the contents for douninst.sh and if you find it choose to refuse on the basis that the maintainer is in your view incompetent :-).

Quote:

Sure, but you see? You're then partially re-implementing the core functionality of removepkg in a file ran from removepkg.
If I wrap a third-party package that is supplied with an install/uninstall script I'm only automating the execution of that script's uninstall option. That has zero to do with removepkg. If my douninst.sh script contains a 'pip uninstall XXX' line, then that also has zero to do with removepkg. Removepkg has no knowledge of pip installed packages at all. I don't really get your point.

NonNonBa 04-06-2020 09:20 AM

Quote:

Originally Posted by bifferos (Post 6108396)
'required' and 'available' are two different concepts. Just because something is required doesn't mean it has to be available. I require it. I don't really understand your argument.

When a software requires another element, it means it won't build without it. When a mechanism is required by a package, it means the system won't work well without it.

Quote:

Originally Posted by bifferos (Post 6108396)
What I don't understand is why you seem to be telling me I can't/shouldn't use any uninstall steps, and how it creates some kind of problem for either you or the wider Slackware community. It doesn't, this is simply my choice for my own systems.

I don't tell you what you should do or not, I even submitted how you could do it in a way that wouldn't alter removepkg. I'm not against a mechanism allowing to automate tasks after a removal, I just think it has little/nothing to do on the packager's side.

Quote:

Originally Posted by bifferos (Post 6108396)
Then why would you use his packages? When you download the package, grep the contents for douninst.sh and if you find it choose to refuse on the basis that the maintainer is in your view incompetent :-).

It has nothing to do with the competency of the packager. It's just you can ask to a packager to provide sane install defaults not sane uninstall defaults.

Quote:

Originally Posted by bifferos (Post 6108396)
If my douninst.sh script contains a 'pip uninstall XXX' line, then that also has zero to do with removepkg. Removepkg has no knowledge of pip installed packages at all. I don't really get your point.

And how do you know any other packages or local script don't need XXX if you are not the admin of the machine? You can't. Here is all the point, precisely.

bassmadrigal 04-06-2020 11:03 AM

Quote:

Originally Posted by NonNonBa (Post 6108373)
You can't trust the packager, that's my point, because a packager just can't know nor anticipate what the other packagers do or will do, as there is no way to guess what a doinst generates (precisely the targets of the undoinst). More generally, except in some rare cases, a packager can't really know what an admin will want to keep or not and is then always prone to fight him.

With Pat adding the ability to skip running the douninst.sh and your ability to alias removepkg to contain --skip-douninst, it seems to be the best of both worlds. Many package managers support some post uninstall script, so it is nice for Slackware to add that functionality and leave it optional.

But if you can't trust a packager, should you be trusting them to write a doinst.sh or the SlackBuild itself? What about overwriting files from other packages? If the packager doesn't have that particular program installed, they may not even realize that their package is overwriting another one. And it is very possible that it could cause breakage. Point blank, if you're using someone else's work, from SBo, from a package repo, you're inherently placing some trust in that person (although, you can still validate that trust by checking the SlackBuild, doinst.sh, douninst.sh, and installpkg --warn to make sure files won't be overwritten.

Quote:

Originally Posted by NonNonBa (Post 6108373)
Sure, but you see? You're then partially re-implementing the core functionality of removepkg in a file ran from removepkg.

But if that file was placed there after the creating the package itself (either in the doinst.sh or during execution of that program), then it would be impossible to use the functionality within removepkg to ensure a file removed won't reside in another package.

Luckily, none of my scripts on SBo will need the douninst.sh functionality (that I can see), and I imagine very few will actually need it, but it's nice to have the option rather than potentially leaving stale files/folders around the system that aren't tracked by the package manager.

brobr 04-07-2020 06:25 AM

FWIF, I just ran into the situation where, I think a douninst.sh is fitting. The dkms-package (on SBo) helps to maintain modules built externally from the kernel after a kernel-upgrade. It creates its own build-area in /var/lib/dkms.

Installing a kernel-module to be built via dkms involves placing the required source files in /usr/src -which can be done via a build-script that then runs dkms from doinst.sh to build and to install the kernel-driver. After running removepkg the build tree for the kernel-module in /var/lib/dkms, some symlinks, and the dkms-installed kernel-modules will be left behind (in /lib/modules/(kernelversion)/extra). Dkms-installs leave original kernel-modules intact.

Ideally, one would 'dkms remove <module/version> -all' before (with removepkg) deleting all source files. This because these sources contain the conf.file dkms needs for completing the 'remove' instruction. As it is uncommon and therefore prone to happen, there is a risk that the dkms step will be skipped. This can be repaired with a dounist.sh that cleans up the bits that would have been removed by the 'dkms remove' command.

Maybe, instead of delving into the -for the likes like me- quite abstract pro-and cons of a douninst.sh, it would be interesting to see examples of how people use it or bypass the use of it.

With the risk of being branded an 'untrusted' maintainer, the discussed dkms example is here. All comments to improve the scripts(s) are more than welcome.

It is early days, but a dounist.sh template on SBo would be very helpful in streamlining intended usage.

bifferos 04-07-2020 08:01 AM

@brobr I posted an example of how you can use it in this thread. Nobody really commented on that either for or against, so that's either because nobody understood it, or perhaps due to my code generating a slackbuild instead of actually *being* a slackbuild people didn't trust it. To remove a layer of abstraction I've now posted the *generated* slackbuild.

initrd is just a simple python module that I registered on pypi. In order to register it certain metadata must be supplied, and this metadata is queryable using the pypi API. So rather than duplicate this metadata on SBo, sbgen.py simply queries the metadata and generates a slackbuild from it. The generated slackbuild has been uploaded here:
https://github.com/bifferos/slackbui.../master/initrd

The readme gives the generated file structure:
https://github.com/bifferos/slackbui.../initrd/README

You can see the only file that matters is:

opt/afterpkg-python/initrd-0.1.tar.gz

This is the package source. The doinst.sh installs it using pip. The douninst.sh removes it using pip. Of course, the Python package must appear twice, once in source form, and also in installed form, but Python packages are not generally that large. For me it doesn't matter.

This allows upwards of 1000 python packages on SBo to be generated. Where dependency management finds a requirement for a specific Python package, instead of looking on SBo, we can generate the slackbuild wrapper package and pip install it, and according to the Slackware package management tools it's listed as being there, whilst not limiting us to the pypi versions that SBo maintainers have got around to pulling in.

All this is highly experimental at the moment and nowhere near ready for people to use in anger, it's just something I'm working on, but the new douninst.sh functionality allows me to experiment with such concepts without having to hack removepkg all the time.

NonNonBa 04-07-2020 08:18 AM

Quote:

Originally Posted by bassmadrigal (Post 6108422)
But if you can't trust a packager, should you be trusting them to write a doinst.sh or the SlackBuild itself?

Not the same thing. You can expect a packager to provide what is required/advised to run a software according to the upstream, but there's nothing as a "sane removal". Breaking things following the INSTALL instructions is not the same as breaking them because you walk in the dark. installpkg already tracks everything has no need to be linked to your particular setup, removepkg shouldn't cross this line, which is the admin's territory.

Quote:

Originally Posted by bassmadrigal (Post 6108422)
Luckily, none of my scripts on SBo will need the douninst.sh functionality (that I can see), and I imagine very few will actually need it.

You partially meet my point. I think on the packager's side this feature is either useless or dangerous, while on the admin's side it could be useful, safe, and far more powerful.

Quote:

Originally Posted by brobr (Post 6108696)
Maybe, instead of delving into the -for the likes like me- quite abstract pro-and cons of a douninst.sh, it would be interesting to see examples of how people use it or bypass the use of it.

You have to hack the pkgtools if you want to test my solution. I think your package does to much things for a source package: the doinst shouldn't try to build anything. With my suggestion, the admin could just automate the remove/rebuild of the modules each time the kernel is upgraded.

Plus, it seems your douninst assumes the running kernel will be the one where the modules where installed. It couldn't be the case if the admin first upgrades the kernel to check it globally works and only then upgrades the modules, and when it is the case, how does the doinst know the targeted new kernel (open question, I don't know dkms)? With my model, the script would get the names of the old/new kernels and could determine where to remove and add things.

bassmadrigal 04-07-2020 10:30 AM

Quote:

Originally Posted by NonNonBa (Post 6108724)
Not the same thing. You can expect a packager to provide what is required/advised to run a software according to the upstream, but there's nothing as a "sane removal". Breaking things following the INSTALL instructions is not the same as breaking them because you walk in the dark. installpkg already tracks everything has no need to be linked to your particular setup, removepkg shouldn't cross this line, which is the admin's territory.

You partially meet my point. I think on the packager's side this feature is either useless or dangerous, while on the admin's side it could be useful, safe, and far more powerful.

I'm not going to continue this debate. It has been determined with a lot of different distro's package managers that a post-uninstall script is a good idea. Slackware has now implemented it and it is up to the package maintainers to use it in a sane manner. If someone is able to convince Pat to remove the douninst support, I doubt it will bother me, but it doesn't bother me that it's included either. I imagine as SlackBuilds are developed that start containing douninst scripts, the SBo admins will be making sure they are unlikely to cause harm to the system.

For users who don't want to leave their system in the package maintainer's hands, they can simply create an alias like the following:

Code:

alias removepkg='removepkg --skip-douninst'
Then if they need to override that sometime, they'd just call removepkg with the absolute path

Code:

/sbin/removepkg

brobr 04-07-2020 11:55 AM

Thanks for the example. And an interesting one (because it deals with python packages):
Quote:

Originally Posted by bifferos (Post 6108714)
@brobr ..

The doinst.sh installs it using pip. The douninst.sh removes it using pip. ..

.. instead of looking on SBo, we can generate the slackbuild wrapper package and pip install it, and according to the Slackware package management tools it's listed as being there, whilst not limiting us to the pypi versions that SBo maintainers have got around to pulling in.

Thus, afterpkg generates "footprints" of pip-installed modules that can be picked up by Slackware's pkgtools; as if these modules have been installed using Slackbuilds. This makes a lot of sense; not only because of the hassle of dependency-tracking (often one needs to update the SBo-scripts to get a correct version or -at the moment- have to change python-version manually). But more so because (new) developers of python modules do not create a setup.py script that caters for a 'python setup.py install' route that works in the traditional way outwith pip (as is needed to create a distributable slackware pkg). Mainly because they are not aware of this.

Instead of 'shipping' built python packages from one computer to another, with afterpkg the admin instructs/distributes pip installs of those (Please correct me if I get this still wrong).

This example makes it much clearer (at least for me) to understand how afterpkg is intended to work.

Such an approach makes management of complex packages like Leo much easier. Also the python-bloat on SBo can be minimized to only those packages that can be considered on the interface between python and the OS. Leo is an interesting use_case in this respect, very fast development, continually changing requirements (dependencies); but maybe (too quickly) linked to novel versions of non-python ware such as QT5 (via PyQT). And it is at these cross-sections that one might get limited to which versions of a program like Leo could be used on Slackware (so there a SLackbuild as published on SBo still comes in useful) or the other way round; a python module (like pysam) that can be linked to a native library (htslib) breaks when the latter is upgraded and the python module is not following in its tracks. Still, such info would be trackable in the 'requires.txt' files any python module/program comes with. So these problems are not immediately related to the way afterpkg uses the doinstall/douninstall. (The relevant bit comes in whether/how easy one can control which version afterpkg takes as a starting point).

Ideally one would be able to run 'afterpkg python.app' and get the stuff installed by pkgtools.

Analogous to sbopkg, maybe consider putting the downloaded source files in /var/cache/afterpkg instead of /opt ( /var gets space for itself on my box so that overflowing kernel-messages are not obstructing other stuff). Also, after installation they could be removed, couldn't they (I won't have space on my laptop for 1000 packages ;-)?

Also note, as mentioned a while back on the SlackBuilds.users list, that any python package distributed via pypi can be downloaded using this (logical) format:
Quote:

https://pypi.python.org/packages/source/<package first letter>/<package name>/<package name-version.extension>
So, python-defusedxml would look like:
Code:

"https://pypi.python.org/packages/source/d/defusedxml/defusedxml-0.6.0.tar.gz"


brobr 04-07-2020 12:04 PM

Quote:

Originally Posted by NonNonBa (Post 6108724)
I think your package does to much things for a source package: the doinst shouldn't try to build anything. With my suggestion, the admin could just automate the remove/rebuild of the modules each time the kernel is upgraded.

Plus, it seems your douninst assumes the running kernel will be the one where the modules where installed. It couldn't be the case if the admin first upgrades the kernel to check it globally works and only then upgrades the modules, and when it is the case, how does the doinst know the targeted new kernel (open question, I don't know dkms)? With my model, the script would get the names of the old/new kernels and could determine where to remove and add things.

Hmm, yes that makes sense; With dkms, you only change the module when there is an upgrade of that. When you upgrade the kernel that module gets automatically installed in the new one; that is how dkms is set up. So the updating of the module is uncoupled of that of the kernel. If you miss uninstalling from an old kernel how important is that (quite often with something like using Virtual box, modules stay behind so that you have to remove old kernel modules manually)? For me my slackbuild calling dkms via doinst.sh is a kind of script you talk about...

bifferos 04-07-2020 12:28 PM

Yes, that's a fairly good description of where Afterpkg is trying to go. However it's better if you consider it as a tool to assist me in experimentation with SBo rather than some kind of finished product. sbgen.py, which generates the Python packages isn't fully integrated into Afterpkg yet, and I don't know if it should be. For instance do we need to actually create real Slackware packages, or should we just be fooling dependency management systems into thinking they're present with some kind of hint files? There are a lot of Python packages, not so many dependency management tools, after all.

Don't be too worried about where I'm putting downloaded source files. This is so experimental and needs so many edge cases ironed it may be impossible to complete in any meaningful way. For instance it's better to use the python wheels, but they don't always exist for our platform, or they may not have been created at all for some older packages. initrd (my example) works and builds, other packages do not.

brobr 04-07-2020 01:41 PM

@biffereos Well, more fantasizing at this end, than being worried ;-).

Quote:

do we need to actually create real Slackware packages, or should we just be fooling dependency management systems
Ideally, would this not be a kind of choice/option that can be build in: say -p for pip-install only and -s for an package.tar.gz as end product, i.e. an SBo-like archive for the generated SlackBuild. This option could then also mean that the directed tar.gz is passed on to say sbopkg, for generating (and installing) the _SBo.tgz. When bypassing the pip-route,the generation of donistsk/dounist.sh could be omitted. But maybe this latter route will become error-prone in the future when pip-install is taking dominance, as mentioned above.

One could imagine that the whole python-tree in SBo would just be a list of 'safe' package-names for python-modules that can be processed by afterpkg. Possibly, a lot of other python-only packages (like some in the 'academic' tree ) could be treated as such as well.

I imagine that such a combo (of afterpkg, sbopkg, pkgtools) would reduce enormously the work load of people maintaining python packages or those that use them but need uncovered versions (absent or more up-to-date).

Management of R-packages, not of R itself, is already completely going through the R-interface and not via slack-builds. I wonder whether it's future-proof to try to keep up with the ever increasing complexity of the python jungle through slackbuilds as we know. Afterpkg seems a nice gel that could keep either style of package-management a valid option.

drumz 04-07-2020 02:34 PM

Quote:

Originally Posted by brobr (Post 6108696)
FWIF, I just ran into the situation where, I think a douninst.sh is fitting. The dkms-package (on SBo) helps to maintain modules built externally from the kernel after a kernel-upgrade. It creates its own build-area in /var/lib/dkms.

Installing a kernel-module to be built via dkms involves placing the required source files in /usr/src -which can be done via a build-script that then runs dkms from doinst.sh to build and to install the kernel-driver. After running removepkg the build tree for the kernel-module in /var/lib/dkms, some symlinks, and the dkms-installed kernel-modules will be left behind (in /lib/modules/(kernelversion)/extra). Dkms-installs leave original kernel-modules intact.

Ideally, one would 'dkms remove <module/version> -all' before (with removepkg) deleting all source files. This because these sources contain the conf.file dkms needs for completing the 'remove' instruction. As it is uncommon and therefore prone to happen, there is a risk that the dkms step will be skipped. This can be repaired with a dounist.sh that cleans up the bits that would have been removed by the 'dkms remove' command.

Yes, I agree. When I uploaded system76-io-dkms (https://slackbuilds.org/repository/1...tem76-io-dkms/) to SlackBuilds I felt "dirty" for having the "dkms install system76-io/$VER" command in doinst.sh, but no way to remove the module when doing "removepkg". I will definitely be updating this SlackBuild for 15.0 to make use of douninst.sh.

Of course, I'll put a big fat warning in the README that "removepkg" will be doing extra work to fully remove system76-io-dkms from the system.

brobr 04-07-2020 05:13 PM

Hi drumz, thanks for the feedback. When cobbling mine together I had a look at your Slackbuild to see how others had used dkms; well, there was only one... possibly because of what NonNonBa put forward. Maybe at one point, after I have not experienced any glitches, I will try to get my script accepted along yours...

bassmadrigal 04-07-2020 08:07 PM

I did have a dkms build working from within a SlackBuild on an older version of the amdgpu-pro driver without needing to do it in doinst.sh.

https://github.com/bassmadrigal/slac...ver.SlackBuild

Code:

if [ "${BUILDDKMS:-no}" == "yes" ]; then

  # Patch the crap outta the source so we can build the module
  for i in $CWD/patches/*.patch; do patch -p1 < $i; done

  # Prevent dkms from trying to rebuild an initrd
  sed -i 's/REMAKE_INITRD="yes"/REMAKE_INITRD="no"/' $PKG/usr/src/${SRCNAM}-${SRCVER}/dkms.conf

  # Add the kernel module to the correct location
  sed -i 's|/updates|/kernel/drivers/gpu/drm/amd/amdgpu|' $PKG/usr/src/${SRCNAM}-${SRCVER}/dkms.conf

  # Set up the source for dkms building
  mkdir -p $PKG/var/lib/dkms/${SRCNAM}/${SRCVER}/source/
  ln -s /var/lib/dkms/${SRCNAM}/${SRCVER}/source usr/src/${SRCNAM}-${SRCVER}

  # Check if dkms is installed
  if [ ! -x /usr/sbin/dkms ]; then
    echo "Please install dkms from SBo"
    exit 1
  fi

  # Let's build it
  mkdir -p $PKG/lib/modules/`uname -r`/kernel/drivers/gpu/drm/amd/amdgpu
  dkms install \
    -m ${SRCNAM} \
    -v ${SRCVER} \
    --sourcetree $PKG/usr/src \
    --dkmstree $PKG/var/lib/dkms/${SRCNAM}/${SRCVER} \
    --installtree lib/modules
fi

But if you have a new kernel added, those modules would never be tracked by pkgtools and wouldn't be removed if the package was removed. douninst.sh would allow you to use dkms to remove all versions of that module.

NonNonBa 04-08-2020 04:03 AM

Quote:

Originally Posted by bassmadrigal (Post 6108760)
It has been determined with a lot of different distro's package managers that a post-uninstall script is a good idea.

And as many that think the same touching tracking dependencies.

Quote:

Originally Posted by bassmadrigal (Post 6108760)
For users who don't want to leave their system in the package maintainer's hands, they can simply create an alias like the following:

Code:

alias removepkg='removepkg --skip-douninst'

This option should then be added to upgradepkg and in any other tool calling the pkgtools.

Quote:

Originally Posted by bassmadrigal (Post 6108760)
I'm not going to continue this debate.

Yep, I think I've also far enough exposed my views. ;)

brobr 04-08-2020 05:47 AM

Quote:

Originally Posted by bassmadrigal (Post 6108943)
I did have a dkms build working from within a SlackBuild
...
Code:

if [ "${BUILDDKMS:-no}" == "yes" ]; then

 ...

  # Set up the source for dkms building
  mkdir -p $PKG/var/lib/dkms/${SRCNAM}/${SRCVER}/source/
  ln -s /var/lib/dkms/${SRCNAM}/${SRCVER}/source usr/src/${SRCNAM}-${SRCVER}

  # Check if dkms is installed
  if [ ! -x /usr/sbin/dkms ]; then
    echo "Please install dkms from SBo"
    exit 1
  fi

  # Let's build it
  mkdir -p $PKG/lib/modules/`uname -r`/kernel/drivers/gpu/drm/amd/amdgpu
  dkms install \
    -m ${SRCNAM} \
    -v ${SRCVER} \
    --sourcetree $PKG/usr/src \
    --dkmstree $PKG/var/lib/dkms/${SRCNAM}/${SRCVER} \
    --installtree lib/modules
fi

..

Ha, so you can call dkms during the package build so that it installs into the package before it is installed via pkgtools?
I am going to test that, thanks !-)

One thing I do not completely understand:
Quote:

douninst.sh would allow you to use dkms to remove all versions of that module.
How would this work when -as is the case for the digimend-drivers- the dkms.conf is removed with the source by pkgtools before the douninst.sh is run?

allend 04-08-2020 09:44 AM

Quote:

Luckily, none of my scripts on SBo will need the douninst.sh functionality (that I can see), and I imagine very few will actually need it, but it's nice to have the option rather than potentially leaving stale files/folders around the system that aren't tracked by the package manager.
I like the introduction of a douninst.sh functionality. It allows a package install to include functionality that can be cleanly removed when the package is removed. Consider the use case of a package that generates log file information. The package can add a logrotate script so the log file information will not grow without control. If the package is removed then the douninst.sh can remove stale logfiles. Rather than few needing it, I see many using it.

bassmadrigal 04-08-2020 10:54 AM

Quote:

Originally Posted by NonNonBa (Post 6109049)
And as many that think the same touching tracking dependencies.

True, but douninst.sh is quite a bit different than tracking dependencies. And really, this is only touching the surface. Some package managers have scripts to support pre-install, post-install, pre-uninstall, and post-uninstall.

Quote:

Originally Posted by NonNonBa (Post 6109049)
This option should then be added to upgradepkg and in any other tool calling the pkgtools.

This would be something that Pat would need to decide, but I can see the benefit in adding support for upgradepkg and slackpkg.

Quote:

Originally Posted by brobr (Post 6109089)
Ha, so you can call dkms during the package build so that it installs into the package before it is installed via pkgtools?
I am going to test that, thanks !-)

Hopefully it still works fine. I was toying with that back in 2016 and 2017 but ultimately removed support for it in the script (and then I stopped providing updates for the script).

Quote:

Originally Posted by brobr (Post 6109089)
One thing I do not completely understand:

How would this work when -as is the case for the digimend-drivers- the dkms.conf is removed with the source by pkgtools before the douninst.sh is run?

This is a good point and I didn't think about that. To use dkms, it would need to be done in a pre-uninstall script, which is not supported or you'd have to do something funky to make sure the dkms.conf wasn't removed (I wouldn't recommend this). I suppose the alternative would be to just manually remove the modules using a wildcard for the kernel version.

Code:

rm -rf /lib/modules/*/location/to/modules.ko

Richard Cranium 04-09-2020 04:59 PM

Quote:

Originally Posted by bassmadrigal (Post 6108760)
I'm not going to continue this debate. It has been determined with a lot of different distro's package managers that a post-uninstall script is a good idea. Slackware has now implemented it and it is up to the package maintainers to use it in a sane manner. If someone is able to convince Pat to remove the douninst support, I doubt it will bother me, but it doesn't bother me that it's included either. I imagine as SlackBuilds are developed that start containing douninst scripts, the SBo admins will be making sure they are unlikely to cause harm to the system.

For users who don't want to leave their system in the package maintainer's hands, they can simply create an alias like the following:

Code:

alias removepkg='removepkg --skip-douninst'
Then if they need to override that sometime, they'd just call removepkg with the absolute path

Code:

/sbin/removepkg

Or just
Code:

\removepkg

bassmadrigal 04-09-2020 05:09 PM

Quote:

Originally Posted by Richard Cranium (Post 6109716)
Or just
Code:

\removepkg

I didn't know that. Thanks!

brobr 04-09-2020 08:30 PM

@bassmadrigal: Thanks again for the suggestion and example how dkms can be directed to build kernel modules into the package. I adapted my Slackbuild to have this as an option. It's very nice to see quickly what dkms does (provided the script covers/reproduces all its steps). With this in mind, the package can work without a douninst,sh. Another thing, though, is that one needs to be aware to call dkms to remove modules from a running kernel (after an upgrade) BEFORE removepkg. As it needs a dkms.conf file for this (see above).

A way around this is to place a copy of the dkms.conf somewhere outwith the files and folders controlled by pkgtools. This can be done from the doinst.sh by means of a simple copy command. Then, a douninst.sh can
a) first call dkms with this 'parked' conf (put in /etc/dkms) for guidance what driver to uninstall and -after completion -
b) can remove the conf-copy from /etc/dkms.

This works quite nicely, but means appreciation of a douninst.sh. Which defies the rationale not to call dkms during installation of the driver. So, as the default option, the Slackbuild is set to make full use of dkms by means of the doinsth.sh and a douninst.sh. And by doing so may better reflect what the developers of the drivers have in mind (they advice to use dkms for installation of their drivers).

You can check it here: digimend-drivers-dkms

cheers


All times are GMT -5. The time now is 08:45 PM.