[SOLVED] Testing of HooRex (SlackBuilds.org dependency calculator)
SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Well if you're going to take the approach of having what you call a 'cache', you'd be better off using a proper sqlite database -- then your complex queries are just trivial SQL. And then you can keep all sorts of other useful things in there.
I've avoided calling it a database for that very reason - if its a database, I should use a proper database tool. I did consider it but, in truth, the queries I need are not complex but quite trivial python dictionary lookups into the cache. That means I can avoid the additional complexity (and learning) of the additional sqlite interface code that would be needed. I wanted to keep HooRex simple - its not a complete build infrastructure like slackrepo; I already have my own build system and just wanted a simple dependency calculator to drive the build system with a list of package names (in the correct order).
Thanks again for your suggestions - restricting output to packages already installed is very useful.
chris
I was working on this for my own tool (asbt), but its just a grep of the info files and not very efficient, so I am probably going to be using hoorex now
I've got a python topological sort package that you can use as you see fit:
Code:
#! /usr/bin/env python
# -*- Mode: Python -*-
#
# Figure out strongly connected components from a graph.
#
# See http://en.wikipedia.org/wiki/Strongly_connected_component
#
# That has a sample graph which I will use as a test case.
#
import os
class GraphNode(object):
def __init__(self, **kwargs):
# expect "name" and "child_list"
# will set transpose_finish if found
super(GraphNode, self).__init__(kwargs)
self.name = kwargs["name"]
# child_list is a list of GraphNode names.
self.child_list = kwargs["child_list"]
self.transpose_finish = kwargs.get("transpose_finish", -1)
self.discover = -1
self.finish = -1
def __str__(self):
return ("GraphNode s. child_list %s. discover %d. finish %d. finishT: %d." %
(self.name, repr(self.child_list), self.discover, self.finish, self.transpose_finish ))
def isWhite(self):
return self.discover == -1 and self.finish == -1
def isBlack(self):
return self.finish != -1
def isGrey(self):
return self.discover != -1 and self.finish == -1
class Graph(object):
def __init__(self, **kwargs):
"""nodes is a list of GraphNode objects.
This constructor will create a dictionary (hashmap) of those
nodes with the node name as a key."""
super(Graph, self).__init__(kwargs)
if kwargs.has_key("nodes"):
self.nodes = dict([ (x.name, x) for x in kwargs["nodes"] ])
else:
self.nodes = {}
self.time = 0
def addNode(self, node):
self.nodes[node.name] = node
def transpose(self):
"""Return a Graph object which describes the transpose of this one."""
retval = Graph(nodes=[GraphNode(name=x.name,
child_list=[],
transpose_finish=x.finish)
for x in self.nodes.itervalues()])
for x in self.nodes:
for y in self.nodes[x].child_list:
retval.nodes[y].child_list.append(x)
return retval
def dfs(self):
"""Perform a DFS on the graph, setting the discover and finish values."""
# Normally, white, grey, and black are used to mark the nodes.
# discover == finish == -1 means the node is white.
# discover != -1 and finish == -1 means the node is grey
# discover != -1 and finish != -1 means the node is black.
# (Actually, finish != -1 is sufficient to declare a node to be black.)
#
# More wierdness. If none of the transposeFinish values are
# different from the default, you essentially get all the nodes in
# no particular order.
#
for aval in self.nodes.values():
aval.discover = aval.finish = -1
theList = [ x for x in self.nodes.values() ]
theList.sort(None, lambda x: x.transpose_finish, True)
self.time = 0
return [ self.__visit(x, []) for x in theList if x.isWhite() ]
def __visit(self, node, retlist):
self.time += 1
node.discover = self.time
for x in node.child_list:
if self.nodes[x].isWhite():
self.__visit(self.nodes[x], retlist)
self.time += 1
node.finish = self.time
retlist.append(node)
return retlist
class SCC(object):
"""This is a Strongly Connected Component object."""
IDENT = 1
def __init__(self, nodeList, verbose):
self.import_list = set() # this is a set of GraphNode ids
self.nodes = set() # this is a set of GraphNode objects
self.ident = SCC.IDENT
SCC.IDENT += 1
for aNode in nodeList:
self.addNode(aNode, verbose)
def addNode(self, theNode, verbose):
if verbose:
print "Adding %s to SCC %d" % (theNode, self.ident)
self.nodes.add(theNode)
self.import_list.update(theNode.child_list)
def __str__(self):
temp = "SCC ident: %d" % self.ident
for aNode in self.nodes:
temp += (os.linesep + str(aNode))
return temp + os.linesep
class StronglyConnectedComponentFinder(object):
def __init__(self, theGraph, verbose=False):
# verbose here is for debugging
self.graph = theGraph
self.graphT = None
self.verbose = verbose
self.SCCList = []
self.SCCMap = {}
def findSCC(self):
if self.graphT:
return
self.graph.dfs()
self.graphT = self.graph.transpose()
theList = self.graphT.dfs()
# theList is a list of lists of GraphNodes. Python can be rather Lispy at times.
for x in theList:
newSCC = SCC(x, self.verbose)
self.SCCList.append(newSCC)
self.SCCMap[newSCC.ident] = newSCC
if __name__ == '__main__':
print "Creating the Graph instance."
aGraph = Graph(nodes=[ GraphNode(name="a", child_list=["b"]),
GraphNode(name="b", child_list=["e", "f", "c"]),
GraphNode(name="c", child_list=["d", "g"]),
GraphNode(name="d", child_list=["c", "h"]),
GraphNode(name="e", child_list=["a", "f"]),
GraphNode(name="f", child_list=["g"]),
GraphNode(name="g", child_list=["f"]),
GraphNode(name="h", child_list=["g", "d"]) ])
print "Create StronglyConnectedComponentFinder object."
finder = StronglyConnectedComponentFinder(aGraph, True)
print "Calculate dfs of original graph, the transpose, and the DFS of the transpose."
finder.findSCC()
print "the list of SCCs is: "
for scc in finder.SCCList:
print scc
print "Note that the nodes in this SCC list are from the *transposed* graph!"
This is a by-product of a Java build tool that I used to maintain. It would compile a package worth of java files at a time, using this to determine the correct order to compile them. If two or more packages referenced each other, you'd get a strongly connected component that you'd have to compile all at once.
I think we were using python 2.5.x when we stopped doing things this way. It's using the 2.x print statement versus the 3.x print function, but that's an easy change to make.
Last edited by Richard Cranium; 04-20-2015 at 11:27 PM.
Reason: The python version would be nice to know.
I was working on this for my own tool (asbt), but its just a grep of the info files and not very efficient, so I am probably going to be using hoorex now
Thats nice to hear, thanks aaditya.
The problem with doing a grep over all .info files for each query is the time needed to work through the entire SBo repo which is currently over 5000 packages. Thats a lot of file io and was my main motivation for making HooRex. Also, the output from grep is fine for human reading but needs some work to be used by other apps e.g. for package building. My strategy was to work through the repo just once and store the information I need in a format (Python dictionary) that is more or less instantaneous to query.
I've got a python topological sort package that you can use as you see fit:
Code:
#! /usr/bin/env python
...
This is a by-product of a Java build tool that I used to maintain. It would compile a package worth of java files at a time, using this to determine the correct order to compile them. If two or more packages referenced each other, you'd get a strongly connected component that you'd have to compile all at once.
Thanks, that is quite interesting. I've often wondered about representing dependencies as a graph - mainly for visualization though. I see from your test case that the nodes' internal data of name and child_list is already quite close to my dictionary of names as keys and package dependencies as values. It might be interesting to generate a Graph of the whole SBo repo to look for circular dependencies.
I've pushed out a 0.5.4 version which contains the enhancements suggested by aaditya, along with fixes for some minor bugs I found along the way. Any more feedback would be greatly appreciated.
Hi,
I just installed it on my buildbox and played around with it. It's a great tool, and I think I'll use it quite regularly. One detail: at the moment, hoorex runs fine when invoked as a normal user. Example:
[root@raymonde:~] # hoorex lame
Traceback (most recent call last):
File "/usr/bin/hoorex", line 427, in <module>
main()
File "/usr/bin/hoorex", line 32, in main
default_config = load_user_config()
File "/usr/bin/hoorex", line 372, in load_user_config
with open(config_file, 'wb') as configfile:
IOError: [Errno 2] No such file or directory: '/root/.config/hoorex/defaults.cfg'
I think it would come in handy if hoorex could also run as root, since building packages normally happens as root.
I just installed it on my buildbox and played around with it. It's a great tool, and I think I'll use it quite regularly. One detail: at the moment, hoorex runs fine when invoked as a normal user. Example:
[root@raymonde:~] # hoorex lame
Traceback (most recent call last):
File "/usr/bin/hoorex", line 427, in <module>
main()
File "/usr/bin/hoorex", line 32, in main
default_config = load_user_config()
File "/usr/bin/hoorex", line 372, in load_user_config
with open(config_file, 'wb') as configfile:
IOError: [Errno 2] No such file or directory: '/root/.config/hoorex/defaults.cfg'
I think it would come in handy if hoorex could also run as root, since building packages normally happens as root.
I think you need to recreate its internal database as you do as user also as root as the datas are generated in the invoking user's home (they're not available system wide), like
I just installed it on my buildbox and played around with it. It's a great tool, and I think I'll use it quite regularly. One detail: at the moment, hoorex runs fine when invoked as a normal user. Example:
[root@raymonde:~] # hoorex lame
Traceback (most recent call last):
File "/usr/bin/hoorex", line 427, in <module>
main()
File "/usr/bin/hoorex", line 32, in main
default_config = load_user_config()
File "/usr/bin/hoorex", line 372, in load_user_config
with open(config_file, 'wb') as configfile:
IOError: [Errno 2] No such file or directory: '/root/.config/hoorex/defaults.cfg'
I think it would come in handy if hoorex could also run as root, since building packages normally happens as root.
I had made the assumption that a user's XDG configuration directory would already exist. Clearly (now) thats not always a correct assumption so, just in case, we create it ourselves before trying to write into it.
Thanks for reporting the problem.
chris
Last edited by chris.willing; 04-21-2015 at 04:43 AM.
Reason: typo
Thats great Niki, I'm thrilled you like it so much.
I had a quick look around your repo and noticed that pyxdg (which is a runtime dependency for HooRex) is included in your desktop release but not server release, so if HooRex makes it out of testing into server release, pyxdg would need to go into server release too.
Thats great Niki, I'm thrilled you like it so much.
I had a quick look around your repo and noticed that pyxdg (which is a runtime dependency for HooRex) is included in your desktop release but not server release, so if HooRex makes it out of testing into server release, pyxdg would need to go into server release too.
chris
Yeah, I know. For the moment, I'm test-driving it. Once everything works as it should, I'll push it over to the main server-* and desktop-* repos, with a thought for the dependency.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.