Using sbopkg to move from 14.0 to 14.1

Woodsman · 06-08-2014, 09:15 PM

I have been browsing the forum but haven't found my answer.

I want to migrate my 14.0 list of SBo packages such that when I reboot into my 14.1 partition, I can use sbopkg to build the same packages along with dependencies.

I created a single sqf of SBo packages installed in 14.0. I can copy that file to my 14.1 partition. That list has no information about dependencies --- just a list of what I have installed from SBo.

I pulled all sqf files from gitorious.

Now, how to coordinate the entire shebang?

Basically I need to convert my 14.0 sqf list into a mega sqf by merging appropriate sqf files as needed.

I played with sqg but all I got was duplication of all files I downloaded from gitorious.

Is there a sane way to create a super master sqf file?

Thanks.

willysr · 06-08-2014, 10:06 PM

sqg built a queue files per package, not a super master sqf file.

once you built the queue files using sqg, you can use sbopkg -c to check whether your package has an update on 14.1 and then you can use sbopkg -i "package1 package2 ...packageN" to build and install your packages

Woodsman · 06-08-2014, 10:31 PM

Quote:

sqg built a queue files per package, not a super master sqf file.

Then sqg is not what I seek. I might as well download the sqf files --- much faster than building locally and once installed, 'git pull' takes a few seconds.

All I am trying to do is automate the entire SBo rebuild process. I know what I have installed. I want to consolidate that list. For example, two packages: ffmpeg and gparted. Both have dependencies. I could write a simple script like this:

sbopkg -i ffmpeg
sbopkg -i gparted
sbopkg -i ...

That would work and any coincidental dependencies are skipped in subsequent package builds.

I am seeking to reduce my list of 14.0 SBo packages into something like the above. Perhaps I could parse my 14.0 list, look for a related gitorious sqf file, and if the gitorious sqf contains "@" references, then remove each respective "@" references from my 14.0 list. What remains should be a short list of what to build that sbopkg should handle.

moisespedro · 06-08-2014, 11:11 PM

I am still not sure on what you want to accomplish. I've read your posts twice and I am clueless.

ruario · 06-09-2014, 01:24 AM

I think you want to recreate the build/install order you used last time so you can rebuild again with dependencies satisfied.

Quote:

Originally Posted by ruario

If you installed all of your SBo packages once and have never upgraded them then "/var/log/packages" alone can provide you with a nice sorted list based on when they were installed, which you could use as a queue file, e.g. try something like this:

Code:

$ ls -tr /var/log/packages | sed -nr 's#^([[:graph:]]+)(-[[:alnum:]\.\+_]+){3}SBo#\1#p'

If however you have been upgrading SBo packages from time to time (most likely), it gets a little more complex. The above might still work but because the upgraded packages will be out of sync in the list, there is a fair chance they will appear after one of more of their build dependencies causing a build failure. To get an idea of how big a problem this could be for you, try creating a list of the SBo packages that have been upgraded at some point in time, like so:

Code:

$ ls --full-time -tr /var/log/removed_packages | sed -nr 's#.* ([0-9]{4}-[0-9]{2}-[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2})\..* ([[:graph:]]+)(-[[:alnum:]\.\+_]+){3}SBo-upgraded-(.*)#Installed: \1,\2\tUpgraded: \5\t\3#p'

If this list is fairly small, you might get away with using your original list or perhaps you could use the new one to give you some clues about what to shift around in the build order of the original list. If it is fairly long however you will probably need some help to fix the build order. In which case the following steps should give you a rough list:

Code:

$ ls --full-time -tr /var/log/packages | sed -nr 's#.* ([0-9]{4}-[0-9]{2}-[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2})\..* ([[:graph:]]+)(-[[:alnum:]\.\+_]+){3}SBo#\1,\2\t\3#p' > /tmp/SBo-install-times
$ ls --full-time -tr /var/log/removed_packages | sed -nr 's#.* ([0-9]{4}-[0-9]{2}-[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2})\..* ([[:graph:]]+)(-[[:alnum:]\.\+_]+){3}SBo-upgraded.*#\1,\2\t\3#p' >> /tmp/SBo-install-times
$ sort /tmp/SBo-install-times | sed "s,.*\t,," > /tmp/SBo-install-order-with-dupes

The final output file (/tmp/SBo-install-order-with-dupes) shows the actual install order but packages that have been upgraded will be listed multiple times (once for each install/upgrade). You will need to edit this list and remove all but the first reference to each package before you can use it as a queue file, otherwise you will end up building some of your packages more than once.

Note: If any of the packages you are rebuilding have gained additionally dependencies that might cause the queue to fail part way through.

ruario · 06-09-2014, 01:57 AM

I forgot I wrote a second post:

Quote:

Originally Posted by ruario

EDIT: I have realised however that this isn't perfect because the method listed in my earlier post could have included stuff that was upgraded and later uninstalled. The best way to deal with those problems would be to recreate the list of currently installed SBo packages:

Code:

$ ls -tr /var/log/packages | sed -nr 's#^([[:graph:]]+)(-[[:alnum:]\.\+_]+){3}SBo#\1#p' > /tmp/SBo-list

Then use this list to find the dupes within /tmp/SBo-install-order-with-dupes, e.g.:

Code:

$ while read p; do printf "$p:" ;grep -xcF "$p" /tmp/SBo-install-order-with-dupes; done < /tmp/SBo-list | grep -v ':1$'

Afterwards, you can also use /tmp/SBo-list again to check for packages that shouldn't be listed in /tmp/SBo-install-order-with-dupes at all (because you removed them at some later stage):

Code:

$ grep -vxFf /tmp/SBo-list /tmp/SBo-install-order-with-dupes

Once you have finished manually editing /tmp/SBo-install-order-with-dupes to remove duplicate entries or packages you don't want, you could do the following to rebuild and reinstall your current selection of SBo packages:

Code:

# mv /tmp/SBo-install-order-with-dupes /var/lib/sbopkg/queues/mypackages.sqf 
# sbopkg -i mypackages

Better yet make that last line:

Code:

# mv /tmp/SBo-install-order-with-dupes /var/lib/sbopkg/queues/mypackages.sqf 
# sbopkg -ki mypackages

And you won't have to worry about cleaning out the dupes.

moisespedro · 06-09-2014, 02:17 AM

A bit off topic: ruario never finishes his messages in one post :P

ruario · 06-09-2014, 02:37 AM

Quote:

Originally Posted by moisespedro

A bit off topic: ruario never finishes his messages in one post :P

This is 100% true!

kikinovak · 06-09-2014, 06:01 AM

Quote:

Originally Posted by Woodsman

I want to migrate my 14.0 list of SBo packages such that when I reboot into my 14.1 partition, I can use sbopkg to build the same packages along with dependencies.

KISS approach:

slackpkg clean-system
upgrade system from 14.0 to 14.1
rebuild everything manually step by step

Cheers,

Niki

jtsn · 06-09-2014, 08:45 AM

Quote:

Originally Posted by kikinovak

KISS approach

1. Upgrade system
2. Rebuild SBo packages that don't work anymore (or keep old library versions around until they get rebuilt)
3. Leave everything else unchanged

It's the same principle used for maintaing Slackware itself.

ruario · 06-09-2014, 09:36 AM

I think his whole point was about getting the build order right for those he needs to rebuild. I might be wrong but that is how I read it.

Woodsman · 06-09-2014, 04:10 PM

Quote:

I've read your posts twice and I am clueless.

Oh well. I never claimed to be the brightest bulb in the pack. My apologies for not expressing myself clearly.

Quote:

rebuild everything manually step by step

Sure, as long as I can automate the process with a script. I keep reading that computers are supposed to be good at that kind of thing.

Quote:

Rebuild SBo packages that don't work anymore

I have been doing that for years. I decided that when I move to 14.1 I would perform a rebuild of everything. Start fresh.

Quote:

I think his whole point was about getting the build order right for those he needs to rebuild.

That's all.

sbopkg doesn't know my intentions and if I want to rebuild all SBo packages I need appropriate sqf files to automate the rebuilds. My 14.0 package list does not include dependencies. The gitorious sqf files contain dependencies. I needed to merge the two lists. I did that after I posted last night. Refer to the last paragraph, post #3. Sometimes just talking things out loud with others gets the creative juices humming in the background.

I have been skipping every other Slackware release. I haven't been updating with each Slackware release because of the work involved. I don't have a plain stock system. Updating is a lot of work for me because of years of customization going back to about Slackware 10. The challenge with not updating to the latest release is the general mindset among free/libre development is "tough luck --- we won't wait for you." This time around, I want to tinker with Mate, but only version 1.6 is available for 14.0. I want to test the mate-system-tools package, which comes only with 1.8, which is available only for 14.1. So screw me unless I update.

Or take a lot of time to set up a full Mate build environment. This is one of the few aspects I dislike about free/libre software: Bleeding freaking edge all the time.

In all the years I have been updating Slackware releases I never wrote any kind of update check list. I got away with that laziness until the past few years when I started getting involved with development testing. Now I have two hard drives, a laptop, an HTPC and keeping everything straight during a system update is stressful. With each release something always breaks along the way. Always. That is the nature of free/libre software: Bleeding freaking edge all the time.

Thus I am trying to write such a check list, which includes rebuilding SBo packages as necessary. I do not want to sit at my computer manually cd'ing to each build directory, running ./*.SlackBuild, manually installing the package, and repeating for several days. I have 118 SBo packages installed.

Sometimes I think I ought to move to Current and treat my system as a semi-rolling release. Perhaps then the breakages would be smaller in nature, probably happen one at a time, and be more manageable. Yet even if I do that, I need to first get 14.1 installed and running smoothly.

Habitual · 06-09-2014, 04:42 PM

Quote:

Originally Posted by ruario

This is 100% true!

I can...

---------- Post added 06-09-14 at 05:43 PM ----------

Quote:

Originally Posted by ruario

This is 100% true!

vouch for that.

Didier Spaier · 06-09-2014, 04:59 PM

As suggested by Willy in post #2, I would use sqg + sbopkg:

I assume that pkglist is a text file, containing a list of the 112 packages names, and sqflist the same, but with each package name being followed by .sqf
run sqg -p `cat pkglist` to build your 112 queue files
run removepkg `cat pkglist` to be able to safely use the -k option in the next command
run sbopkg -i -k -B "`cat sqflist`" so that sbopkg knows it has to build and install queue files, not packages

If all goes well I believe that should build and install all your packages, building and installing each dependency only once. I'm not sure this can be followed to the letter, but you get the idea.

EDIT I edited the last command adding a -B option so that you don't have to give a confirmation before building and installing each package. But bear in mind that reading the logs after processing the whole list of packages is recommended

Woodsman · 06-09-2014, 05:33 PM

Ha! Instructions! Thank you.

Although I am beginning to think gslapt might be easier.