LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 06-13-2018, 08:43 AM   #1
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Rep: Reputation: 20
How to rebuild all SBo packages installed on the system


I'm looking for a way to rebuild all installed packages created with SBo scripts.
I use to use sbopkg on my slackware 14.2, the stable branch.

Recently I upgraded many slackware official packages, the most are patches provided by Pat. There were also gcc, glibc, and kernel.
I also upgraded some AlienBoB packages to the version provided on his slack 14.2 repository.

I think would be a good choice to rebuild also all installed SBo packages, because some of them could be based on an old library version no longer installed (upgraded to a new version).

So my question.. is there a way to tell sbopkg/sqg to automatically:
- check all SBo packages (which packages are installed on the system?)
- create a right sorted queue, with a dependency priority (eg. "if find qbittorrent and libtorrent-rasterbar, put in the queue libtorrent-rasterbar before qbittorrent").
- finally process the queue

Thanks in avance!
 
Old 06-13-2018, 08:58 AM   #2
orbea
Senior Member
 
Registered: Feb 2015
Distribution: Slackware64-current
Posts: 1,435

Rep: Reputation: Disabled
This was recently asked on the Slackbuilds-users mailing list, I would suggest reading the thread there as it had some good replies.

https://lists.slackbuilds.org/piperm...ne/021580.html
 
Old 06-13-2018, 10:03 AM   #3
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Original Poster
Rep: Reputation: 20
Seems the answer was this:
http://idlemoor.github.io/slackrepo/

I've never used slackrepo, I have to read the friendly manual.
Thanks for your hint!
If I'll have some doubts abaout that tool I will ask there!
Again, thank you.
Bye!
 
Old 06-19-2018, 05:49 PM   #4
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Original Poster
Rep: Reputation: 20
Hi all, I read a bit about slackrepo but I'm not sure it is intended to rebuild all _SBo packages installed on the system, "against" all libraries versions actually running on the system.
If I well understand, it is intended to build packages "against" a vanilla slackware OS, for redistribution of binary packages ready to be installed on any other "vanilla slackware".

Looking for a way to re-build all SBo packages against my own slackware system (after an upgrade of a very large number of packages, patched by pat, upgraded by alienbob and so on), I tried to write a very simple bash script.

If I'm not wrong, the problem in rebuilding of installed SBo packages could be: to create a queue of packages sorted by "dependency" order, so that the tool "sbopkg" (for example) will be able to process it without missed deps errors.

- Thinking about that procedure, my script attempt to download the SBo queue for each SBo package found on the system (thanks to "sqg" script provided along with "sbopkg" tool).

- Then it concatenates all downloaded queues, removing commented lines sometimes present in ".sqf" files. Here we have a raw list of concatenated queues.

- the script removes double lines: the same package repeated many times, we need just the first occurrence.
That output queue contains all packages needed to rebuild all software previously installed via SBo. And I should follow the right building order.

- At this point the script removes from the queue all packages it finds already installed on the system, but not from SBo.
Eg.
I installed "ffmpeg" from AlienBob and "mpv" from SBo.
When I rebuild all SBo packages, I want to keep Alien "ffmpeg" and rebuild just "mpv" through its SBo slackbuild script. I also want to re-build mpv against AlienBob "ffmpeg" already installed.
So I have to remove "ffmpeg" from the rebuild queue. I hope I explained it well.

- that final queue is recorded as a log file and should be possible to give it to sbopkg as input queue. "sbopkg -i queue.log".


What do you think about?
I would be very grateful if you could take a look. I put it as attachment to this message.

Thanks in advance for your answers!

PS.
The script just creates a queue file without any install or any rebuild procedure, so it shouldn't be invasive, anyway read it before test.
I verified the number of SBo packages installed and it matches the number of packages in the output queue (in my case 255 software to rebuild). This is just really raw check...
Attached Files
File Type: txt sbo-rebuild.sh.TXT (871 Bytes, 17 views)
 
Old 06-19-2018, 07:40 PM   #5
montagdude
Senior Member
 
Registered: Apr 2016
Distribution: Slackware
Posts: 1,240

Rep: Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931
One way would be using sboui:

1. Filter by installed packages
2. Tag all categories (this will tag everything that is installed)
3. Type "e" to rebuild

Last edited by montagdude; 06-19-2018 at 07:46 PM.
 
1 members found this post helpful.
Old 06-19-2018, 07:56 PM   #6
montagdude
Senior Member
 
Registered: Apr 2016
Distribution: Slackware
Posts: 1,240

Rep: Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931
Though, thinking about this some more, if you are just rebuilding everything, does the order really matter? I think you can just rebuild them one at a time in whatever order desired; all the dependencies for a particular package will already be present at build time, even if some of them are linked against old library versions.
 
Old 06-20-2018, 05:58 AM   #7
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Original Poster
Rep: Reputation: 20
I could be wrong, but let's consider an example:

Code:
- package A-1_alien: installed from AlienBob repo as pre-compiled.
- package B-1_SBo: installed from SBo compiled against A-1_alien and depends on this one.
- package C-1_SBo: built with SBo script against B-1_SBo and depends on it.
Now what happens if I upgrade "A" package?
Maybe nothing.
But "B" could stop working properly, due to to lack of its dependency which is compiled against. And the same could be for "C".
We would have:

Code:
- A-2_alien (upgraded)
- B-1_SBo ---> against A-1_alien
- C-1_SBo ---> B-1, A-1
If I try to rebuild C-1_SBo before "B" I would compile it against B-1_SBo, so we will obtain:

Code:
- A-2_alien
- B-1_SBo ---> A-1
- C-2_SBo ---> B-1, A-1
Then, if I try to rebuild also "B":

Code:
- A-2_alien
- B-2_SBo ---> against A-2
- C-2_SBo ---> against the old B-1 (which in turn, had been built against A-1)
So, C-2_SBo could be not working properly, due to the lack of B-1 (and A-1) it was compiled against.

Anyway, likely could not be any problem at all in process an unsorted queue.
But IMHO it is more safe to follow a dependency based queue.
Any comments, suggests, about this point will be very useful and well accepted.

As regards "sboui", what exactly did you refer to?
Sbopkg tool interface when launched without arguments?
I gave it a closer look, actually it has an option to create a queue with all installed SBo packages. Maybe you referred to that feature.
I tried: it created a 255 packages queue. This is an other raw confirm on my bash tool results.
I inspected the queue and it is sorted by name.
For example I find "abiword" before "wv", which abiword depends on. So if I process that queue as is, sbopkg would build and install "abiword" before its dep "wv". May be does not produce any issue as regards abiword, but It could be unsafe.
An other example from my queue is ardour:
Code:
ardour: REQUIRES="aubio jack-audio-connection-kit liblo liblrdf vamp-plugin-sdk rubberband lv2 cwiid lilv"
It would be processed before all its dependencies. And as I showed above could be potentially unsafe.

I checked the queue with this command:
Code:
for i in $(cat /var/lib/sbopkg/queues/all_SBo_packages.sqf); do echo "$i: $(echo | sbopkg -s "$i" | grep ^REQUIRES)" ; done|less


Anyway I try to compare sbopkg generated queued containing all installed packages (I saved it as "all_SBo_packages"):
Code:
/var/lib/sbopkg/queues/all_SBo_packages.sqf
... with the one generated using my script:
Code:
/var/log/sbo-rebuild/2018.06.14-18.35-sborebuild.queue.log
I could attach the two quues to this message... in case you want to check it.
I tried the following check and it seems no missed packages are found:
Code:
for i in $(cat /var/log/sbo-rebuild/2018.06.14-18.35-sborebuild.queue.log); do if ! grep -q $i /var/lib/sbopkg/queues/all_SBo_packages.sqf; then echo "$i not found in all packages sbo queue"; fi; done

for i in $(cat /var/lib/sbopkg/queues/all_SBo_packages.sqf); do if ! grep -q $i /var/log/sbo-rebuild/2018.06.14-18.35-sborebuild.queue.log; then echo "$i not found in all packages sbo queue"; fi; done
Attached Files
File Type: log 2018.06.14-18.35-sborebuild.queue.log (3.3 KB, 2 views)
File Type: txt all_SBo_packages.sqf.txt (3.3 KB, 3 views)
 
Old 06-20-2018, 06:37 AM   #8
elcore
Member
 
Registered: Sep 2014
Distribution: Slackware
Posts: 528

Rep: Reputation: Disabled
I see here in your code, a whole lot of complications for no good reason.

Thing is, if you are able to do successful upgrade manually; you only have to do it once.
Then you could just pull everything you wrote from .bash_history into your shell script, and you won't have to do it ever again.

Personally, I scripted mostly everything I've ever built, it removepkg all the relevant packages first, builds and upgrades them in a very specific sequence, then stores all *.t?z as backup.
Sure I could've chosen to depend on someone's tool, but I prefer not to. It's easier for me to patch my own shell code (with only coreutils dep) than having external tools which can change anytime.
 
Old 06-20-2018, 10:10 AM   #9
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Original Poster
Rep: Reputation: 20
Thanks for your reply.
What do you mean exactly with "...you only have to do it once"?
I have to upgrade 255 SBo packages, I need an automated procedure to accomplish it. Don't you agree?
(I don't want to write all that procedure manually at bash prompt! )
 
Old 06-20-2018, 11:09 AM   #10
elcore
Member
 
Registered: Sep 2014
Distribution: Slackware
Posts: 528

Rep: Reputation: Disabled
I mean if you do it properly, the next time you get into same situation you'll only have to write one command instead of 255 commands.
But if you get an automation tool elsewhere and something breaks, you'll have to debug code you did not write. (or wait for the new version of the 3rd party tool)
This is not something I'm comfortable with because; (1) it's an incredible waste of time, and (2) it makes you dependent on automation tool author.

What I can do to help, is advise you to split packages which depend on each other into package groups, so you'll have like 6 sequences, rather than one sequence of 255.
Most of those 255 don't need a specific order, and for those that do I just make a group and build them in that order.
 
Old 06-20-2018, 11:35 AM   #11
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 5,436

Rep: Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211
I did request a feature I think you'd find handy to sqg (the companion package of sbopkg that generates queue files). willysr pulled through and made some changes to sqg to allow it, but I don't think it is in a released version yet, so you'd have to grab it directly from github. With this change, you could parse your /var/log/packages for SBo packages, extract the name from the files, then pass that to sqg to generate a gigantic sqf file that you can use with sbopkg to rebuild everything. However, this would keep any dependencies that are required by SBo, even if you used an alternate (like Alien Bob's). So, you'd need to manually remove those from that queue file.

In regards to slackrepo, it does do what you want, just in a way you may not expect. slackrepo ensures every package built is built against a clean system (except for any dependencies it needs to build). This ensures your packages aren't accidentally built against unknown dependencies. Now, how does this help in your situation? Well, slackrepo keeps track of every dependency, and when one is changed, slackrepo will rebuild any packages that are attached to that. And slackrepo allows you to do mass rebuilds of all your packages.

By default, slackrepo prefers to be in a clean environment, where it will do all its compiling in a chroot that is destroyed after the package is created and saved elsewhere. However, you can override that and have slackrepo operate in a "dirty" environment, where it will just operate on the computer it's installed on and will keep packages installed as they're compiled.
 
Old 06-20-2018, 12:41 PM   #12
montagdude
Senior Member
 
Registered: Apr 2016
Distribution: Slackware
Posts: 1,240

Rep: Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931Reputation: 931
Quote:
Originally Posted by joenew View Post
I could be wrong, but let's consider an example:

Code:
- package A-1_alien: installed from AlienBob repo as pre-compiled.
- package B-1_SBo: installed from SBo compiled against A-1_alien and depends on this one.
- package C-1_SBo: built with SBo script against B-1_SBo and depends on it.
...

If I try to rebuild C-1_SBo before "B" I would compile it against B-1_SBo, so we will obtain:

...

So, C-2_SBo could be not working properly, due to the lack of B-1 (and A-1) it was compiled against.
Assuming that you are only rebuilding your SBo packages and not upgrading them, then I don't think this would be a problem. Let's say that A and B are packages producing libraries libA.so.x and libB.so.y, respectively. When you rebuild B against A-2, it will simply produce a new version of libB.so.y, linking against libA.so.x, but having the same exact name as before. Therefore, when you run C-2, it will pull in the new libB.so.y at runtime, coming from the B-2 package, and libB.so.y will pull in libA.so.x. So all the links will be valid and satisfied, even though you rebuilt them out of order.

Quote:
Originally Posted by joenew View Post
As regards "sboui", what exactly did you refer to?
Sbopkg tool interface when launched without arguments?
No, it's a separate tool adding a new ncurses user interface to sbopkg, sbotools, etc., but it also has some additional features including dependency resolution. It could do what you want, but after thinking through your problem some more, it would take a bit of tedious user intervention to make it rebuild everything in the proper order without duplicating rebuilds. So it's probably not worth the effort in your case, unless you decide you want to try it out.

Last edited by montagdude; 06-20-2018 at 12:43 PM.
 
Old 06-20-2018, 01:37 PM   #13
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Original Poster
Rep: Reputation: 20
Quote:
Originally Posted by bassmadrigal View Post
I did request a feature I think you'd find handy to sqg (the companion package of sbopkg that generates queue files). willysr pulled through and made some changes to sqg to allow it, but I don't think it is in a released version yet, so you'd have to grab it directly from github. With this change, you could parse your /var/log/packages for SBo packages, extract the name from the files, then pass that to sqg to generate a gigantic sqf file that you can use with sbopkg to rebuild everything. However, this would keep any dependencies that are required by SBo, even if you used an alternate (like Alien Bob's). So, you'd need to manually remove those from that queue file.
Have you look at my attached script?
In my solution I've added:
Code:
for pacchetto in $(awk '!a[$0]++' $RAW_QUEUE); do
	find /var/log/packages -regex "/var/log/packages/$pacchetto-[^-]*-[^-]*-[^-]*_SBo" -exec echo $pacchetto \; >> $SBO_Q
done
My var called RAW_QUEUE is the big list of concatenated queues of all installed SBo packages. So the same package is listed also two, three or "N" times...
The "for" cycle above read the RAW_QUEUE after it has been cleaned with "awk": awk takes the dirty RAW_QUEUE and remove from it the second, third, "N_th" occurrence of the same package, keeping just the first one. It performs this "edit" preserving the sorting as it find... that is crucial to maintain a rebuild ordered by dependency.

After that thanks to the "for" cycle, for each package of the "cleaned list", it looks for that package in installed list (/var/log/packages) and just if it is tagged "_SBo", it write the package name at the bottom of a new cleaned queue (SBO_Q). In that way, the new queue doesn't contain any "non SBo" packages.
Example:

ffmpeg
mpv

On my system it will find ffmpeg from "-alien", so it doesn't add to the final queue.
It will find mpv from SBo and it will add it to the final queue.
So when I will give that final queue to sbopkg it will rebuild just mpv and it will rebuild it against the AlienBob "ffmpeg" that is on the system.

In my script the user doesn't have to manually remove any packages from the final queue, it contains the same list of packages installed just from SBo on the system.
 
Old 06-20-2018, 02:13 PM   #14
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 5,436

Rep: Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211Reputation: 3211
Quote:
Originally Posted by joenew View Post
Have you look at my attached script?
In my solution I've added:
Code:
for pacchetto in $(awk '!a[$0]++' $RAW_QUEUE); do
	find /var/log/packages -regex "/var/log/packages/$pacchetto-[^-]*-[^-]*-[^-]*_SBo" -exec echo $pacchetto \; >> $SBO_Q
done
My var called RAW_QUEUE is the big list of concatenated queues of all installed SBo packages. So the same package is listed also two, three or "N" times...
The "for" cycle above read the RAW_QUEUE after it has been cleaned with "awk": awk takes the dirty RAW_QUEUE and remove from it the second, third, "N_th" occurrence of the same package, keeping just the first one. It performs this "edit" preserving the sorting as it find... that is crucial to maintain a rebuild ordered by dependency.

After that thanks to the "for" cycle, for each package of the "cleaned list", it looks for that package in installed list (/var/log/packages) and just if it is tagged "_SBo", it write the package name at the bottom of a new cleaned queue (SBO_Q). In that way, the new queue doesn't contain any "non SBo" packages.
Example:

ffmpeg
mpv

On my system it will find ffmpeg from "-alien", so it doesn't add to the final queue.
It will find mpv from SBo and it will add it to the final queue.
So when I will give that final queue to sbopkg it will rebuild just mpv and it will rebuild it against the AlienBob "ffmpeg" that is on the system.

In my script the user doesn't have to manually remove any packages from the final queue, it contains the same list of packages installed just from SBo on the system.
I haven't looked at the script since LQ attachments are blocked at my work. But based on your explanation, I do foresee an issue with the build order. If you're just basing your rebuild order on the time they were originally installed, you could run into issues if any dependencies have changed on SBo (or if you upgraded any packages).

But you could probably pass that list to sqg to generate a queue with a proper build order based on the latest info from SBo.
 
Old 06-20-2018, 03:20 PM   #15
joenew
Member
 
Registered: Mar 2010
Distribution: slackware 14.2 64bit
Posts: 112

Original Poster
Rep: Reputation: 20
To clear up any confusion about my script, I report it "inline" on this messagge:
Code:
#!/bin/bash


SBO_PKGS=/tmp/installed-SBo.txt
RAW_QUEUE=/tmp/rawqueue-SBo-rebuilder.tmp
SBO_Q=/tmp/SBo-queue-rebuilder.txt
QDIR=/var/lib/sbopkg/queues
LOGDIR=/var/log/sbo-rebuild
LOG=$(date +%Y.%m.%d-%H.%M-sborebuild.queue.log)

[ -e $RAW_QUEUE ] && rm $RAW_QUEUE
[ -e $SBO_Q ] && rm $SBO_Q
[ ! -d $LOGDIR ] && mkdir $LOGDIR

find /var/log/packages -name "*_SBo"|sed 's/^.*\/\(.*\)-.*-.*-.*$/\1/' > $SBO_PKGS
while read pacchetto; do
	[ ! -e $QDIR/$pacchetto.sqf ] && /usr/sbin/sqg -p $pacchetto
	if [ -e $QDIR/$pacchetto.sqf ]; then
		sed '/^#/d' $QDIR/$pacchetto.sqf >> $RAW_QUEUE
	else
		echo $pacchetto >> $RAW_QUEUE
	fi
done < $SBO_PKGS
for pacchetto in $(awk '!a[$0]++' $RAW_QUEUE); do
	find /var/log/packages -regex "/var/log/packages/$pacchetto-[^-]*-[^-]*-[^-]*_SBo" -exec echo $pacchetto \; >> $SBO_Q
done
cp $SBO_Q $LOGDIR/$LOG
#rm $SBO_Q $SBO_PKGS $RAW_QUEUE
It's very raw, I think, but it is also quite simple.
The last line is commented because temp files could be useful for debug purpose.

Yes you forsaw right!
It is based on sqg (as I explained in my previous messages).
So the build order should be the same of the concatenated queues. And this should prevent any dependency issue.

Last edited by joenew; 06-20-2018 at 03:22 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sbopkg and packages outside of SBo solarfields Slackware 1 03-22-2016 12:56 PM
Getting list of installed packages from a dead system ultrasawblade Debian 4 01-14-2016 01:52 PM
Listing of packages installed on system borgibo Linux - Newbie 4 06-16-2008 04:42 PM
Listing of packages installed on system kushalkoolwal Debian 12 06-22-2006 03:59 AM
What are the packages required to installed for Linux Printing System? Akhran Debian 1 10-18-2005 02:58 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 02:48 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration