[SOLVED] Testing of HooRex (SlackBuilds.org dependency calculator)

chris.willing · 04-18-2015, 07:58 PM

Could people please test a new dependency calculator please? I'd appreciate any comments, bug reports etc., prior to a proper release. Its at https://github.com/cwilling/hoorex

It started out as a way to find out which SBo packages would be affected by a change to one package in particular e.g. if package "speex" changes, which other packages might be affected? Simplistically, the answer is found by determining all the SBo packages which list speex in the REQUIRES field of their .info file. A "find" command run over a local copy of the SBo repo will give an answer. However that answer needs some human interpretation - it is no good, the way it is, as input to some other script that I may want to use. Also, that answer only gives a single "level" of packages which require speex. Surely (possibly, at least) there will be other packages which depend on that first load of dependent packages. Who knows how long the trail of dependencies may be? HooRex answers this question (and others).

hoorex speex

will return a list of packages which name speex as a direct dependency (they list speex in their .info file). Running:

hoorex -m speex

will return an extended list of dependent packages i.e. it also includes packages which depend on the first level of dependents, also packages which depend on the dependents' dependents etc., etc.

The output is always listed in "build order" i.e. it works as a queue that you could supply to some build tool. You can specify multiple packages to resolve just by adding their names to the command line. A -l (--long) option includes the SBo package category in the output making it easier to be use in a shell script to automate the building/installation of all the relevant packages. A simple minded example using json-c and speex:

Code:

cd /path/to/slackbuilds
for lpkg in $(hoorex -l speex json-c) ; do
  cd $lpkg
  pkgname=$(basename $lpkg)
  source ./$pkgname.info
  wget $DOWNLOAD
  sh $pkgname.SlackBuild
  installpkg /tmp/$pkgname-$VERSION-*_SBo.tgz
  cd -
done

HooRex also resolves dependencies in the traditional way (which is like the reverse of what has been described so far). To know the complete dependency tree in build order for vlc, use the -r option and run:

hoorex -r vlc

All the available HooRex options are shown from:

hoorex -h

and there's a man page too.

The source tarball is available in releases tab of the link above - at the moment, the direct link to it is https://github.com/cwilling/hoorex/archive/0.5.3.tar.gz (use "wget --content-disposition ..." for a fully named tarball). Just run make install (as root) to install the hoorex script (python) and manpage. The pyxdg package (available at SBo) is a run time dependency.

As I already said, I'm looking for comments, bug reports etc., anything to make HooRex more accurate and robust. Thanks for your time.

chris

aaditya · 04-19-2015, 01:42 AM

Hi,

I tested it and seems to work here

I have two suggestions:

Would it be possible to add a -i option, so that only the installed packages show up? For example,

Code:

$ test/usr/bin/hoorex gtkmm
gtkmm inkscape gparted gstreamermm libglademm bombono-dvd gbgoffice guitarix gigedit ganv seq24 gsmartcontrol gtkmm-utils azr3 libgnomemm gelemental gtkglextmm mysql-workbench harmonySEQ avant-window-navigator k3d amSynth libgnomecanvasmm lvtk nitrogen rawtherapee

But the only installed packages that depend on gtkmm are:

Code:

$ bin/asbt.sh -e --rev gtkmm
/home/aaditya/slackbuilds/desktop/nitrogen/nitrogen.info
/home/aaditya/slackbuilds/system/gparted/gparted.info

Second and somewhat minor, could there be an option to change the way the output appears, for example, instead of getting the output as a list of space separated packages, could we get it in a newline separated format or a tree-like format so that its easy to visualize the dependencies (-r) of a package...

Thx.

chris.willing · 04-19-2015, 07:53 AM

Thanks for the suggestions aaditya. They both seem feasible and I'll add them in the next few days.

chris

chris.willing · 04-20-2015, 03:56 AM

I've pushed out a 0.5.4 version which contains the enhancements suggested by aaditya, along with fixes for some minor bugs I found along the way. Any more feedback would be greatly appreciated.

The direct link to the source tarball is https://github.com/cwilling/hoorex/archive/0.5.4.tar.gz or clone the repo from https://github.com/cwilling/hoorex.git

aaditya · 04-20-2015, 07:12 AM

Seems to work as expected, thx

Code:

$ ./hoorex -i gtkmm
gtkmm nitrogen gparted

Code:

$ ./hoorex -r -1 gparted
mm-common
libsigc++
glibmm
cairomm
pangomm
atkmm
gtkmm
gparted

solarfields · 04-20-2015, 07:56 AM

chris.willing,

this is great. Is there a way for it to distinguish between packages installed from SlackBuilds.org and those from other sources?

a4z · 04-20-2015, 08:22 AM

Quote:

Originally Posted by chris.willing

I've pushed out a 0.5.4 version which contains the enhancements suggested by aaditya, along with fixes for some minor bugs I found along the way. Any more feedback would be greatly appreciated.

The direct link to the source tarball is https://github.com/cwilling/hoorex/archive/0.5.4.tar.gz or clone the repo from https://github.com/cwilling/hoorex.git

hoorex line 21

Code:

HOOREX_VERSION = '0.5.3'

did you forget to update the variable?

ponce · 04-20-2015, 08:36 AM

seems the Makefile takes care of it

https://github.com/cwilling/hoorex/c...05983988241fe4

BTW, thanks Chris for this!

chris.willing · 04-20-2015, 05:07 PM

Quote:

Originally Posted by aaditya

Seems to work as expected, thx

Code:

$ ./hoorex -i gtkmm
gtkmm nitrogen gparted

Code:

$ ./hoorex -r -1 gparted
mm-common
libsigc++
glibmm
cairomm
pangomm
atkmm
gtkmm
gparted

Thanks for testing the changes you suggested aaditya.

BTW something similar to what you tested as above (first find who requires gtkmm, then find build order for one of them) can be done using a hoorex pipeline - it feeds output of first hoorex instance as input to a second hoorex instance e.g.

Code:

hoorex -i gtkmm | hoorex -r -1

which finds who (that you have installed) requires gtkmm and then displays a build order for _all_ of them - not just build order for gpartd. Of course, if you want to view the individual build order(s) then you would have to do it as you already did.

Thanks again for your suggestions - restricting output to packages already installed is very useful.

chris

kikinovak · 04-20-2015, 05:11 PM

This looks like a very interesting and useful project. I'm curious: why the name HooRex?

chris.willing · 04-20-2015, 05:53 PM

Quote:

Originally Posted by solarfields

chris.willing,

this is great. Is there a way for it to distinguish between packages installed from SlackBuilds.org and those from other sources?

On first use, hoorex builds and saves an index of dependency relationships by walking through a local copy of your slackbuilds repository (which we presume you already downloaded by git, rsync etc.). That process uses the REQUIRES field of each package's .info file. In a sense, this restricts things to SBo packages. When hoorex checks its "raw" output against packages already installed, it uses the directory listing at /var/log/packages - those packages could be from anywhere (but contain no dependency information themselves).

Something I have in mind (and partially implemented already) is to enable the initial index building to use something other than the REQUIRES field of the .info files. Some people already enhance their local .info files with additional build requirements to suit how they like particular packages built. This can be done by adding to the existing REQUIRES field; however this leads to lots of additional work when merging SBo updates (pretty much weekly, recently). I personally add an extra field PREREQS at the very end of the .info file. In the PREREQS field, I add whatever additional packages I want along with $REQUIRES e.g.

Code:

PREREQS="$REQUIRES foo bar"

This keeps intact REQUIRES field intact so git merges are mostly seamless.

The reason to go through all that is to suggest that the current hoorex mechanism to index the dependencies of an SBo repo may be adaptable to other repositories. Provided those other repos have a consistent strategy for registering dependency information, hoorex could probably parse them too. Do you have a particular "other" source in mind?

chris

chris.willing · 04-20-2015, 06:01 PM

Quote:

Originally Posted by a4z

hoorex line 21

Code:

HOOREX_VERSION = '0.5.3'

did you forget to update the variable?

Well, yes and no. I didn't explicitly change it (so looks like I forgot) but, as ponce already found, it is set correctly at build/install time via the Makefile.

Its interesting you noticed that in the first place - you must be reading the code

. Hope there's nothing too ugly in there ...

chris

chris.willing · 04-20-2015, 06:05 PM

Quote:

Originally Posted by ponce

seems the Makefile takes care of it

https://github.com/cwilling/hoorex/c...05983988241fe4

BTW, thanks Chris for this!

Thanks for the thanks! Hope you may find it useful and, don't forget, all suggestions are gratefully accepted.

chris

chris.willing · 04-20-2015, 06:59 PM

Quote:

Originally Posted by kikinovak

This looks like a very interesting and useful project. I'm curious: why the name HooRex?

Finding project names can be difficult, as I think you've experienced recently yourself

HooRex is a corruption of "who requires", so that running 'hoorex gtkmm' is like asking 'who requires gtkmm'. I guess it makes packages seem to have some human aspect; maybe 'what requires ..." would be more accurate but whatrex (or wotrex) doesn't roll off the tongue quite as well. I hope it doesn't mean something awful in french ...

chris

55020 · 04-20-2015, 07:03 PM

Well if you're going to take the approach of having what you call a 'cache', you'd be better off using a proper sqlite database -- then your complex queries are just trivial SQL. And then you can keep all sorts of other useful things in there.

cf. https://github.com/idlemoor/slackrep...ctions.sh#L401
and https://github.com/idlemoor/slackrep...ctions.sh#L479