LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 08-06-2016, 01:10 PM   #1
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: McKinney, Texas
Distribution: Slackware64 15.0
Posts: 3,858

Rep: Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225
Faster implementation of sbopkg's sqg queue file generator.


It's written in python and is available at GitHub.
It's faster than the bash script that now comes with sbopkg.

Both runs are on my main desktop, running Slackware64 14.2.

The bash script with sbopkg...
Code:
time /usr/sbin/sqg -a
Processing all SlackBuilds in the SBo/14.2 repo................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.

real	24m8.911s
user	20m6.206s
sys	8m26.916s
The python version at GitHub:
Code:
# time /usr/bin/sqg -a
Done!

real	0m56.907s
user	0m32.647s
sys	0m20.885s
Any questions, ask here.
 
Old 08-06-2016, 02:46 PM   #2
Gerard Lally
Senior Member
 
Registered: Sep 2009
Location: Leinster, IE
Distribution: Slackware, NetBSD
Posts: 2,176

Rep: Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761
Quote:
Originally Posted by Richard Cranium View Post
It's written in python and is available at GitHub.
It's faster than the bash script that now comes with sbopkg.
woah - big difference! First question - am I right in thinking your sqg installs to /usr/bin, while the default sqg installs to /usr/sbin? If so, I presume yours needs to be called with the full path.
 
Old 08-06-2016, 05:18 PM   #3
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: McKinney, Texas
Distribution: Slackware64 15.0
Posts: 3,858

Original Poster
Rep: Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225
Right now it does. If I dig a little into setuptools, I am fairly certain that it can be changed to /usr/sbin. This way, at least, people can install this one in parallel with the one that comes with sbopkg.
 
Old 08-06-2016, 08:03 PM   #4
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 914

Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
hoorex (available through SBo and source code at https://github.com/cwilling/hoorex) also uses python to do this sort of stuff. After setting up a cache of build dependencies, accessing results is close to instantaneous:

chris@d6:~$ time hoorex -r1 -g all > all.sqf
real 0m0.223s
user 0m0.219s
sys 0m0.003s

That doesn't include prior calculation of the cache contents (which I do once a week after the SBo repo is updated). That would be:

chris@d6:~$ time hoorex -f
real 0m17.448s
user 0m0.469s
sys 0m0.560s

It takes ~17 seconds only because my default slackbuilds tree is on an nfs mounted directory. It's much faster when on a local file system:

chris@d6:~$ time hoorex -f -s /tmp/slackbuilds
real 0m0.333s
user 0m0.269s
sys 0m0.054s


chris
 
Old 08-06-2016, 08:12 PM   #5
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: McKinney, Texas
Distribution: Slackware64 15.0
Posts: 3,858

Original Poster
Rep: Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225
Yeah, I didn't bother to pickle the results.

I didn't know about hoorex; I might steal your regex over a generic solution (that, frankly, I already had on hand).

Having quickly waded through the hoorex license, I'll ensure to credit you for anything that I abstract from your code base.
 
Old 08-06-2016, 09:54 PM   #6
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 914

Rep: Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619Reputation: 619
No worries.

I haven't looked at the detail of the regex itself for a while - it may be more complicated than absolutely necessary. I wanted to deal with the possibility of additional fields in the .info files which might express different dependency relationships. In particular, fields which then include other fields e.g.
PREREQS="foo bar $REQUIRES"

chris
 
Old 08-07-2016, 09:11 AM   #7
willysr
Senior Member
 
Registered: Jul 2004
Location: Jogja, Indonesia
Distribution: Slackware-Current
Posts: 4,651

Rep: Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772Reputation: 1772
a little bit OOT, but it seems sqg is faster in current

Code:
real    1m13.693s
user    0m23.941s
sys     0m7.421s
this is run on my desktop which has 4 GB and dual core CPU 5300 @ 2.60GHz
 
Old 08-07-2016, 11:12 AM   #8
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: McKinney, Texas
Distribution: Slackware64 15.0
Posts: 3,858

Original Poster
Rep: Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225Reputation: 2225
What filesystem type? Is there raid underneath? SSD?

Hmm. I'll have to run /usr/sbin/sqg on my testbed rig (currently Slackware64 14.2) to get timings, upgrade to -current and see the delta. Probably later this week.


Turns out that I never converted my testbed from -current to 14.2.

I'm still using sbopkg-0.38.0-noarch-1_wsr on that box. Running /usr/sbin/sqg -a over the 14.2 repo did not show anything like your speedup.

When I configured sbopkg to use the -current repo in github, I got
Code:
root@testbed:~# time sqg -a
ERROR: /var/lib/sbopkg/queues or /var/lib/sbopkg/SBo/14.2 do not exist or are not writable.
Check the configurable variables at the top of the script.

real	0m0.378s
user	0m0.003s
sys	0m0.001s
I'd like to reproduce your findings. I suspect my environment doesn't match yours, but I don't know what exactly to ask to find out what is different.

Here's my results on a Slackware64 -current setup, using the 14.2 repo data:
Code:
root@testbed:~# time sqg -a
Processing all SlackBuilds in the SBo/14.2 repo......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done.

real	15m32.741s
user	12m17.285s
sys	4m31.311s
Code:
root@testbed:~# time /usr/bin/sqg -a -s
Done!

real	0m27.551s
user	0m14.942s
sys	0m9.995s
root@testbed:~#
Code:
root@testbed:~# uname -a
Linux testbed 4.4.16 #1 SMP Thu Jul 28 04:02:04 CDT 2016 x86_64 AMD Athlon(tm) II X2 260 Processor AuthenticAMD GNU/Linux

Last edited by Richard Cranium; 08-07-2016 at 01:59 PM.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sbopkg/sqg vs sbotools? Didier Spaier Slackware 2 11-11-2015 04:39 AM
sqg - sbopkg queuefile generator chess Slackware 22 12-30-2014 03:31 AM
difference in sbopkg queue file and slackbuilds dependencies the_zone Slackware 15 04-24-2014 11:18 AM
Sbopkg : building a queue file with CFLAGS and CXXFLAGS set. Linux.tar.gz Slackware 4 03-11-2012 10:24 PM
[SOLVED] Dount on Queue files and Sbopkg vikrang Slackware 5 07-02-2011 06:02 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 12:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration