LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 04-20-2015, 07:51 PM   #16
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 916

Original Poster
Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619

Quote:
Originally Posted by 55020 View Post
Well if you're going to take the approach of having what you call a 'cache', you'd be better off using a proper sqlite database -- then your complex queries are just trivial SQL. And then you can keep all sorts of other useful things in there.

cf. https://github.com/idlemoor/slackrep...ctions.sh#L401
and https://github.com/idlemoor/slackrep...ctions.sh#L479
I've avoided calling it a database for that very reason - if its a database, I should use a proper database tool. I did consider it but, in truth, the queries I need are not complex but quite trivial python dictionary lookups into the cache. That means I can avoid the additional complexity (and learning) of the additional sqlite interface code that would be needed. I wanted to keep HooRex simple - its not a complete build infrastructure like slackrepo; I already have my own build system and just wanted a simple dependency calculator to drive the build system with a list of package names (in the correct order).

chris
 
Old 04-20-2015, 10:37 PM   #17
aaditya
Member
 
Registered: Oct 2013
Location: India
Distribution: Slackware
Posts: 272
Blog Entries: 2

Rep: Reputation: 85
Quote:
Originally Posted by chris.willing View Post
Thanks again for your suggestions - restricting output to packages already installed is very useful.

chris
I was working on this for my own tool (asbt), but its just a grep of the info files and not very efficient, so I am probably going to be using hoorex now
 
Old 04-20-2015, 11:24 PM   #18
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: McKinney, Texas
Distribution: Slackware64 15.0
Posts: 3,858

Rep: Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225
I've got a python topological sort package that you can use as you see fit:
Code:
#! /usr/bin/env python
#                             -*- Mode: Python -*-
#
# Figure out strongly connected components from a graph.
#
# See http://en.wikipedia.org/wiki/Strongly_connected_component
#
# That has a sample graph which I will use as a test case.
#
import os


class GraphNode(object):

    def __init__(self, **kwargs):
        # expect "name" and "child_list"
        # will set transpose_finish if found
        super(GraphNode, self).__init__(kwargs)
        self.name = kwargs["name"]
        # child_list is a list of GraphNode names.
        self.child_list = kwargs["child_list"]
        self.transpose_finish = kwargs.get("transpose_finish", -1)
        self.discover = -1
        self.finish = -1

    def __str__(self):
        return ("GraphNode s. child_list %s. discover %d. finish %d. finishT: %d."  %
                (self.name, repr(self.child_list), self.discover, self.finish, self.transpose_finish ))

    def isWhite(self):
        return self.discover == -1 and self.finish == -1

    def isBlack(self):
        return self.finish != -1

    def isGrey(self):
        return self.discover != -1 and self.finish == -1
    

class Graph(object):

    def __init__(self, **kwargs):
        """nodes is a list of GraphNode objects.
        This constructor will create a dictionary (hashmap) of those
        nodes with the node name as a key."""
        super(Graph, self).__init__(kwargs)
        if kwargs.has_key("nodes"):
            self.nodes = dict([ (x.name, x) for x in kwargs["nodes"] ])
        else:
            self.nodes = {}
        self.time = 0

    def addNode(self, node):
        self.nodes[node.name] = node

    def transpose(self):
        """Return a Graph object which describes the transpose of this one."""
        retval = Graph(nodes=[GraphNode(name=x.name,
                                        child_list=[],
                                        transpose_finish=x.finish)
                              for x in self.nodes.itervalues()])

        for x in self.nodes:
            for y in self.nodes[x].child_list:
                retval.nodes[y].child_list.append(x)
            
        return retval

    def dfs(self):
        """Perform a DFS on the graph, setting the discover and finish values."""
        # Normally, white, grey, and black are used to mark the nodes.
        # discover == finish == -1 means the node is white.
        # discover != -1 and finish == -1 means the node is grey
        # discover != -1 and finish != -1 means the node is black.
        # (Actually, finish != -1 is sufficient to declare a node to be black.)
        #
        # More wierdness.  If none of the transposeFinish values are
        # different from the default, you essentially get all the nodes in
        # no particular order.
        #
        for aval in self.nodes.values():
            aval.discover = aval.finish = -1
        theList = [ x for x in self.nodes.values() ]
        theList.sort(None, lambda x: x.transpose_finish, True)
        self.time = 0
        return [ self.__visit(x, []) for x in theList if x.isWhite() ]

    def __visit(self, node, retlist):
        self.time += 1
        node.discover = self.time
        for x in node.child_list:
            if self.nodes[x].isWhite():
                self.__visit(self.nodes[x], retlist)
        self.time += 1
        node.finish = self.time
        retlist.append(node)
        return retlist

            
class SCC(object):
    """This is a Strongly Connected Component object."""
    
    IDENT = 1
    
    def __init__(self, nodeList, verbose):
        self.import_list = set() # this is a set of GraphNode ids
        self.nodes = set()        # this is a set of GraphNode objects
        self.ident = SCC.IDENT
        SCC.IDENT += 1

        for aNode in nodeList:
            self.addNode(aNode, verbose)

    def addNode(self, theNode, verbose):
        if verbose:
            print "Adding %s to SCC %d" % (theNode, self.ident)
        self.nodes.add(theNode)
        self.import_list.update(theNode.child_list)

    def __str__(self):
        temp = "SCC ident: %d" % self.ident
        for aNode in self.nodes:
            temp += (os.linesep + str(aNode))
        return temp + os.linesep

class StronglyConnectedComponentFinder(object):

    def __init__(self, theGraph, verbose=False):
        # verbose here is for debugging
        self.graph = theGraph
        self.graphT = None
        self.verbose = verbose
        self.SCCList = []
        self.SCCMap = {}

    def findSCC(self):
        if self.graphT:
            return
        self.graph.dfs()
        self.graphT = self.graph.transpose()
        theList = self.graphT.dfs()
        # theList is a list of lists of GraphNodes.  Python can be rather Lispy at times.
        for x in theList:
            newSCC = SCC(x, self.verbose)
            self.SCCList.append(newSCC)
            self.SCCMap[newSCC.ident] = newSCC
        

if __name__ == '__main__':
    print "Creating the Graph instance."
    aGraph = Graph(nodes=[ GraphNode(name="a", child_list=["b"]),
                           GraphNode(name="b", child_list=["e", "f", "c"]),
                           GraphNode(name="c", child_list=["d", "g"]),
                           GraphNode(name="d", child_list=["c", "h"]),
                           GraphNode(name="e", child_list=["a", "f"]),
                           GraphNode(name="f", child_list=["g"]),
                           GraphNode(name="g", child_list=["f"]),
                           GraphNode(name="h", child_list=["g", "d"]) ])
    
    print "Create StronglyConnectedComponentFinder object."
    finder = StronglyConnectedComponentFinder(aGraph, True)
    print "Calculate dfs of original graph, the transpose, and the DFS of the transpose."
    finder.findSCC()
    print "the list of SCCs is: "
    for scc in finder.SCCList:
        print scc
    print "Note that the nodes in this SCC list are from the *transposed* graph!"
This is a by-product of a Java build tool that I used to maintain. It would compile a package worth of java files at a time, using this to determine the correct order to compile them. If two or more packages referenced each other, you'd get a strongly connected component that you'd have to compile all at once.

I think we were using python 2.5.x when we stopped doing things this way. It's using the 2.x print statement versus the 3.x print function, but that's an easy change to make.

Last edited by Richard Cranium; 04-20-2015 at 11:27 PM. Reason: The python version would be nice to know.
 
Old 04-21-2015, 12:39 AM   #19
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 916

Original Poster
Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
Quote:
Originally Posted by aaditya View Post
I was working on this for my own tool (asbt), but its just a grep of the info files and not very efficient, so I am probably going to be using hoorex now
Thats nice to hear, thanks aaditya.

The problem with doing a grep over all .info files for each query is the time needed to work through the entire SBo repo which is currently over 5000 packages. Thats a lot of file io and was my main motivation for making HooRex. Also, the output from grep is fine for human reading but needs some work to be used by other apps e.g. for package building. My strategy was to work through the repo just once and store the information I need in a format (Python dictionary) that is more or less instantaneous to query.

chris
 
Old 04-21-2015, 01:26 AM   #20
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 916

Original Poster
Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
Quote:
Originally Posted by Richard Cranium View Post
I've got a python topological sort package that you can use as you see fit:
Code:
#! /usr/bin/env python
...
This is a by-product of a Java build tool that I used to maintain. It would compile a package worth of java files at a time, using this to determine the correct order to compile them. If two or more packages referenced each other, you'd get a strongly connected component that you'd have to compile all at once.
Thanks, that is quite interesting. I've often wondered about representing dependencies as a graph - mainly for visualization though. I see from your test case that the nodes' internal data of name and child_list is already quite close to my dictionary of names as keys and package dependencies as values. It might be interesting to generate a Graph of the whole SBo repo to look for circular dependencies.

chris
 
Old 04-21-2015, 02:18 AM   #21
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: CentOS, OpenSUSE
Posts: 3,453

Rep: Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154
Quote:
Originally Posted by chris.willing View Post
I've pushed out a 0.5.4 version which contains the enhancements suggested by aaditya, along with fixes for some minor bugs I found along the way. Any more feedback would be greatly appreciated.
Hi,

I just installed it on my buildbox and played around with it. It's a great tool, and I think I'll use it quite regularly. One detail: at the moment, hoorex runs fine when invoked as a normal user. Example:

Code:
[kikinovak@raymonde:~] $ hoorex lame
lame plex-home-theater avidemux ffmpeg kdenlive ebook2cw soundconverter Pd-extended mythtv
When I invoke it as root, I get this:

Code:
[root@raymonde:~] # hoorex lame
Traceback (most recent call last):
  File "/usr/bin/hoorex", line 427, in <module>
    main()
  File "/usr/bin/hoorex", line 32, in main
    default_config = load_user_config()
  File "/usr/bin/hoorex", line 372, in load_user_config
    with open(config_file, 'wb') as configfile:
IOError: [Errno 2] No such file or directory: '/root/.config/hoorex/defaults.cfg'
I think it would come in handy if hoorex could also run as root, since building packages normally happens as root.

Good work! Cheers!

Niki

Last edited by kikinovak; 04-21-2015 at 03:03 AM.
 
Old 04-21-2015, 02:45 AM   #22
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: CentOS, OpenSUSE
Posts: 3,453

Rep: Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154
I've built a package for HooRex.

Slackware 14.1:

http://www.microlinux.fr/microlinux/...bit/slackware/

Slackware64 14.1:

http://www.microlinux.fr/microlinux/...t/slackware64/

Source:

http://www.microlinux.fr/microlinux/...source/hoorex/

Cheers,

Niki
 
2 members found this post helpful.
Old 04-21-2015, 02:47 AM   #23
solarfields
Senior Member
 
Registered: Feb 2006
Location: slackalaxy.com
Distribution: Slackware, CRUX
Posts: 1,449

Rep: Reputation: 997Reputation: 997Reputation: 997Reputation: 997Reputation: 997Reputation: 997Reputation: 997Reputation: 997
Quote:
I think it would come in handy if hoorex could also run as a normal user, since building packages normally happens as root.
I suppose you meant "root".
 
1 members found this post helpful.
Old 04-21-2015, 03:03 AM   #24
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: CentOS, OpenSUSE
Posts: 3,453

Rep: Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154
Quote:
Originally Posted by solarfields View Post
I suppose you meant "root".
Indeed. I just corrected it. Thanks
 
Old 04-21-2015, 03:27 AM   #25
ponce
LQ Guru
 
Registered: Aug 2004
Location: Pisa, Italy
Distribution: Slackware
Posts: 7,098

Rep: Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175Reputation: 4175
Quote:
Originally Posted by kikinovak View Post
Hi,

I just installed it on my buildbox and played around with it. It's a great tool, and I think I'll use it quite regularly. One detail: at the moment, hoorex runs fine when invoked as a normal user. Example:

Code:
[kikinovak@raymonde:~] $ hoorex lame
lame plex-home-theater avidemux ffmpeg kdenlive ebook2cw soundconverter Pd-extended mythtv
When I invoke it as root, I get this:

Code:
[root@raymonde:~] # hoorex lame
Traceback (most recent call last):
  File "/usr/bin/hoorex", line 427, in <module>
    main()
  File "/usr/bin/hoorex", line 32, in main
    default_config = load_user_config()
  File "/usr/bin/hoorex", line 372, in load_user_config
    with open(config_file, 'wb') as configfile:
IOError: [Errno 2] No such file or directory: '/root/.config/hoorex/defaults.cfg'
I think it would come in handy if hoorex could also run as root, since building packages normally happens as root.
I think you need to recreate its internal database as you do as user also as root as the datas are generated in the invoking user's home (they're not available system wide), like
Code:
# hoorex -f -s SBO_PATH
 
Old 04-21-2015, 04:40 AM   #26
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 916

Original Poster
Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
Quote:
Originally Posted by kikinovak View Post
Hi,

I just installed it on my buildbox and played around with it. It's a great tool, and I think I'll use it quite regularly. One detail: at the moment, hoorex runs fine when invoked as a normal user. Example:

Code:
[kikinovak@raymonde:~] $ hoorex lame
lame plex-home-theater avidemux ffmpeg kdenlive ebook2cw soundconverter Pd-extended mythtv
When I invoke it as root, I get this:

Code:
[root@raymonde:~] # hoorex lame
Traceback (most recent call last):
  File "/usr/bin/hoorex", line 427, in <module>
    main()
  File "/usr/bin/hoorex", line 32, in main
    default_config = load_user_config()
  File "/usr/bin/hoorex", line 372, in load_user_config
    with open(config_file, 'wb') as configfile:
IOError: [Errno 2] No such file or directory: '/root/.config/hoorex/defaults.cfg'
I think it would come in handy if hoorex could also run as root, since building packages normally happens as root.

Good work! Cheers!

Niki

I've pushed out version 0.5.5 which includes a fix for this bug. The source tarball is at https://github.com/cwilling/hoorex/archive/0.5.5.tar.gz

I had made the assumption that a user's XDG configuration directory would already exist. Clearly (now) thats not always a correct assumption so, just in case, we create it ourselves before trying to write into it.

Thanks for reporting the problem.

chris

Last edited by chris.willing; 04-21-2015 at 04:43 AM. Reason: typo
 
1 members found this post helpful.
Old 04-21-2015, 04:55 AM   #27
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 916

Original Poster
Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
Quote:
Originally Posted by kikinovak View Post
Thats great Niki, I'm thrilled you like it so much.

I had a quick look around your repo and noticed that pyxdg (which is a runtime dependency for HooRex) is included in your desktop release but not server release, so if HooRex makes it out of testing into server release, pyxdg would need to go into server release too.

chris
 
1 members found this post helpful.
Old 04-21-2015, 05:12 AM   #28
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: CentOS, OpenSUSE
Posts: 3,453

Rep: Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154
Quote:
Originally Posted by chris.willing View Post
Thats great Niki, I'm thrilled you like it so much.

I had a quick look around your repo and noticed that pyxdg (which is a runtime dependency for HooRex) is included in your desktop release but not server release, so if HooRex makes it out of testing into server release, pyxdg would need to go into server release too.

chris
Yeah, I know. For the moment, I'm test-driving it. Once everything works as it should, I'll push it over to the main server-* and desktop-* repos, with a thought for the dependency.

BTW, could it work on Slackware 14.0 also?
 
Old 04-21-2015, 06:32 AM   #29
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 916

Original Poster
Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
Quote:
Originally Posted by kikinovak View Post
BTW, could it work on Slackware 14.0 also?
I just tried it in a fresh 14.0 64bit VM and it seems to be OK.

chris
 
1 members found this post helpful.
Old 04-22-2015, 04:12 AM   #30
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: CentOS, OpenSUSE
Posts: 3,453

Rep: Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154Reputation: 2154
Quote:
Originally Posted by chris.willing View Post
I just tried it in a fresh 14.0 64bit VM and it seems to be OK.

chris
Thanks. I'll look into it as soon as I have a spare moment.

Cheers.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Call for testing - MATE SlackBuilds chess Slackware 298 12-21-2013 09:17 PM
Visualization of package dependency graphs in SlackBuilds.org gabrielmagno Slackware 8 07-24-2013 02:50 AM
openbox on slack13.37 (x86) -- Thanks to linuxquestions.org, slackbuilds.org & many vectrum Linux - Member Desktop Screenshots 5 02-03-2013 12:22 PM
slackbuilds.org harkonen Slackware 16 08-22-2007 02:01 PM
Use SlackBuilds.org or my own hosting to offer up SlackBuilds? hollywoodb Slackware 6 11-30-2006 08:56 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 12:10 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration