LinuxQuestions.org - Faster implementation of sbopkg's sqg queue file generator.

- Slackware (https://www.linuxquestions.org/questions/slackware-14/)

- - Faster implementation of sbopkg's sqg queue file generator. (https://www.linuxquestions.org/questions/slackware-14/faster-implementation-of-sbopkgs-sqg-queue-file-generator-4175586448/)

Faster implementation of sbopkg's sqg queue file generator.

It's written in python and is available at GitHub.
It's faster than the bash script that now comes with sbopkg.

Both runs are on my main desktop, running Slackware64 14.2.

The bash script with sbopkg...

Code:

time /usr/sbin/sqg -a

Processing all SlackBuilds in the SBo/14.2 repo................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.



real        24m8.911s

user        20m6.206s

sys        8m26.916s

The python version at GitHub:

Code:

# time /usr/bin/sqg -a

Done!



real        0m56.907s

user        0m32.647s

sys        0m20.885s

Any questions, ask here.

Quote:

Originally Posted by Richard Cranium (Post 5587055)

It's written in python and is available at GitHub.
It's faster than the bash script that now comes with sbopkg.

woah - big difference! First question - am I right in thinking your sqg installs to /usr/bin, while the default sqg installs to /usr/sbin? If so, I presume yours needs to be called with the full path.

Right now it does. If I dig a little into setuptools, I am fairly certain that it can be changed to /usr/sbin. This way, at least, people can install this one in parallel with the one that comes with sbopkg.

hoorex (available through SBo and source code at https://github.com/cwilling/hoorex) also uses python to do this sort of stuff. After setting up a cache of build dependencies, accessing results is close to instantaneous:

chris@d6:~$ time hoorex -r1 -g all > all.sqf
real 0m0.223s
user 0m0.219s
sys 0m0.003s

That doesn't include prior calculation of the cache contents (which I do once a week after the SBo repo is updated). That would be:

chris@d6:~$ time hoorex -f
real 0m17.448s
user 0m0.469s
sys 0m0.560s

It takes ~17 seconds only because my default slackbuilds tree is on an nfs mounted directory. It's much faster when on a local file system:

chris@d6:~$ time hoorex -f -s /tmp/slackbuilds
real 0m0.333s
user 0m0.269s
sys 0m0.054s

chris

Yeah, I didn't bother to pickle the results.

I didn't know about hoorex; I might steal your regex over a generic solution (that, frankly, I already had on hand).

Having quickly waded through the hoorex license, I'll ensure to credit you for anything that I abstract from your code base.

No worries.

I haven't looked at the detail of the regex itself for a while - it may be more complicated than absolutely necessary. I wanted to deal with the possibility of additional fields in the .info files which might express different dependency relationships. In particular, fields which then include other fields e.g.
PREREQS="foo bar $REQUIRES"

chris

a little bit OOT, but it seems sqg is faster in current

Code:

real    1m13.693s

user    0m23.941s

sys    0m7.421s

this is run on my desktop which has 4 GB and dual core CPU 5300 @ 2.60GHz

What filesystem type? Is there raid underneath? SSD?

Hmm. I'll have to run /usr/sbin/sqg on my testbed rig (currently Slackware64 14.2) to get timings, upgrade to -current and see the delta. Probably later this week.

Turns out that I never converted my testbed from -current to 14.2.

I'm still using sbopkg-0.38.0-noarch-1_wsr on that box. Running /usr/sbin/sqg -a over the 14.2 repo did not show anything like your speedup.

When I configured sbopkg to use the -current repo in github, I got

Code:

root@testbed:~# time sqg -a

ERROR: /var/lib/sbopkg/queues or /var/lib/sbopkg/SBo/14.2 do not exist or are not writable.

Check the configurable variables at the top of the script.



real        0m0.378s

user        0m0.003s

sys        0m0.001s

I'd like to reproduce your findings. I suspect my environment doesn't match yours, but I don't know what exactly to ask to find out what is different.

Here's my results on a Slackware64 -current setup, using the 14.2 repo data:

Code:

root@testbed:~# time sqg -a

Processing all SlackBuilds in the SBo/14.2 repo......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.



real        15m32.741s

user        12m17.285s

sys        4m31.311s

Code:

root@testbed:~# time /usr/bin/sqg -a -s

Done!



real        0m27.551s

user        0m14.942s

sys        0m9.995s

root@testbed:~#

Code:

root@testbed:~# uname -a

Linux testbed 4.4.16 #1 SMP Thu Jul 28 04:02:04 CDT 2016 x86_64 AMD Athlon(tm) II X2 260 Processor AuthenticAMD GNU/Linux