Faster implementation of sbopkg's sqg queue file generator.
SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Faster implementation of sbopkg's sqg queue file generator.
It's written in python and is available at GitHub.
It's faster than the bash script that now comes with sbopkg.
Both runs are on my main desktop, running Slackware64 14.2.
The bash script with sbopkg...
Code:
time /usr/sbin/sqg -a
Processing all SlackBuilds in the SBo/14.2 repo................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.
real 24m8.911s
user 20m6.206s
sys 8m26.916s
It's written in python and is available at GitHub.
It's faster than the bash script that now comes with sbopkg.
woah - big difference! First question - am I right in thinking your sqg installs to /usr/bin, while the default sqg installs to /usr/sbin? If so, I presume yours needs to be called with the full path.
Right now it does. If I dig a little into setuptools, I am fairly certain that it can be changed to /usr/sbin. This way, at least, people can install this one in parallel with the one that comes with sbopkg.
hoorex (available through SBo and source code at https://github.com/cwilling/hoorex) also uses python to do this sort of stuff. After setting up a cache of build dependencies, accessing results is close to instantaneous:
chris@d6:~$ time hoorex -r1 -g all > all.sqf
real 0m0.223s
user 0m0.219s
sys 0m0.003s
That doesn't include prior calculation of the cache contents (which I do once a week after the SBo repo is updated). That would be:
chris@d6:~$ time hoorex -f
real 0m17.448s
user 0m0.469s
sys 0m0.560s
It takes ~17 seconds only because my default slackbuilds tree is on an nfs mounted directory. It's much faster when on a local file system:
chris@d6:~$ time hoorex -f -s /tmp/slackbuilds
real 0m0.333s
user 0m0.269s
sys 0m0.054s
I haven't looked at the detail of the regex itself for a while - it may be more complicated than absolutely necessary. I wanted to deal with the possibility of additional fields in the .info files which might express different dependency relationships. In particular, fields which then include other fields e.g.
PREREQS="foo bar $REQUIRES"
What filesystem type? Is there raid underneath? SSD?
Hmm. I'll have to run /usr/sbin/sqg on my testbed rig (currently Slackware64 14.2) to get timings, upgrade to -current and see the delta. Probably later this week.
Turns out that I never converted my testbed from -current to 14.2.
I'm still using sbopkg-0.38.0-noarch-1_wsr on that box. Running /usr/sbin/sqg -a over the 14.2 repo did not show anything like your speedup.
When I configured sbopkg to use the -current repo in github, I got
Code:
root@testbed:~# time sqg -a
ERROR: /var/lib/sbopkg/queues or /var/lib/sbopkg/SBo/14.2 do not exist or are not writable.
Check the configurable variables at the top of the script.
real 0m0.378s
user 0m0.003s
sys 0m0.001s
I'd like to reproduce your findings. I suspect my environment doesn't match yours, but I don't know what exactly to ask to find out what is different.
Here's my results on a Slackware64 -current setup, using the 14.2 repo data:
Code:
root@testbed:~# time sqg -a
Processing all SlackBuilds in the SBo/14.2 repo......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.
real 15m32.741s
user 12m17.285s
sys 4m31.311s
Code:
root@testbed:~# time /usr/bin/sqg -a -s
Done!
real 0m27.551s
user 0m14.942s
sys 0m9.995s
root@testbed:~#
Code:
root@testbed:~# uname -a
Linux testbed 4.4.16 #1 SMP Thu Jul 28 04:02:04 CDT 2016 x86_64 AMD Athlon(tm) II X2 260 Processor AuthenticAMD GNU/Linux
Last edited by Richard Cranium; 08-07-2016 at 01:59 PM.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.