LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (https://www.linuxquestions.org/questions/slackware-14/)
-   -   Script to get/set dependencies/required packages in a slackbuild (https://www.linuxquestions.org/questions/slackware-14/script-to-get-set-dependencies-required-packages-in-a-slackbuild-4175563496/)

franzen 01-09-2016 11:21 AM

Script to get/set dependencies/required packages in a slackbuild
 
1 Attachment(s)
To get the dependencies of my packages, i used to use "requiredbuilder", but
it does not exactly what i want to. So, i wrote my own "get_deps.sh" which does the
following:
  • detect only hard dependencies(using "readelf -d", not "ldd") in all libs/binaries in the current package
  • check if the current package itself already covers some needed libs( e.g. ffmpeg does), and so don't list the package itself
  • prefer listing *solibs*txz if dependencies are found in multiple packages
  • avoid listing aaa_elflibs and cxxlibs, if dependencies are also covered in other packages
  • Better performance than requiredbuilder
Usage: chmod 755 get_deps.sh, put it somewhere in $PATH, then execute it in a slackbuild right before makepkg:

Code:

...
EXCLUDE="libffi icu4c" ADD="libxyz" get_deps.sh > $PKG/install/slack-required
makepkg ...

The Script is attached as .txt, as i didn't found a way to upload it as .sh.

Feedback is welcome :-)

DarkVision 01-09-2016 12:40 PM

Nice :)

I was giving it a try since my own buildscripts have support for installed requiredbuilder and will create a slack-required if requested.

I don't know much about package managers and slack-required files, the code that is included in my buildscripts is mature and i do not use it myself anymore.

Two questions:
getdeps will produce a slack-required file like that:
Code:

glibc-solibs,libdvdread
while requiredbuilder gives a package each line:
Code:

glibc-solibs
libdvdread

and i can use the -v option to add version info to slack-required:
Code:

glibc-solibs >= 2.17-x86_64-10_slack14.1
libdvdread >= 5.0.3-x86_64-1

Is there a difference between comma-separated and "one dep each line" and can version information be added to the output?

I did another test for ffmpeg:
requiredbuider:
Code:

alsa-lib
bzip2
cxxlibs | gcc-g++
db48
faac
gcc
glibc-solibs
jack
lame
libX11
libXau
libXdmcp
libXext
libXrandr
libXrender
libXv
libogg
libtheora
libva
libvdpau
libvorbis
libvpx
libwebp
libxcb
openjpeg2
opus
sdl
speex
svgalib
twolame
x264
x265
xvidcore
xz
zlib

getdeps:
Code:

alsa-lib,bzip2,faac,glibc-solibs,jack,lame,libX11,libXext,libXv,libtheora,libva,libvdpau,libvorbis,libvpx,libwebp,libxcb,openjpeg2,opus,sdl,speex,twolame,x264,x265,xvidcore,xz,zlib
There are a few differences:
Code:

--- slack-required.requiredbuilder        2016-01-09 19:31:22.699107496 +0100
+++ slack-required.getdeps.sorted        2016-01-09 19:32:55.302029712 +0100
@@ -3,2 +2,0 @@
-cxxlibs | gcc-g++
-db48
@@ -6 +3,0 @@
-gcc
@@ -11,2 +7,0 @@
-libXau
-libXdmcp
@@ -14,2 +8,0 @@
-libXrandr
-libXrender
@@ -17 +9,0 @@
-libogg
@@ -29 +20,0 @@
-svgalib

If i do a ldd libavcodec.so.56.60.100 i get a dependency to libXrandr which is not listed for getdeps. readelf -d does not list libXrandr. Not sure which dep-finder is correct here...

willysr 01-09-2016 09:55 PM

That package is already included in slackware packages i believe

DarkVision 01-09-2016 11:57 PM

Quote:

Originally Posted by willysr (Post 5475833)
That package is already included in slackware packages i believe

Which package? This thread is about the getdeps script.

I have a smiliar script but that is not like getdeps, it just checks all packages installed for bin/lib-files and their dependencies. I'm using ldd but the OP said he is using readelf. So me is just interested in his script (and maybe a replacement of requiredbuilder which is not maintained anymore... AFAIK) and the differences between ldd and readelf -d.

DarkVision 01-10-2016 01:23 AM

@franzen : again, nice job :cool:
I just compared requiredbuilder with getdeps for ffmpeg:

requiredbuilder:
Quote:

real 1m4.860s
user 0m43.089s
sys 0m29.961s
getdeps:
Quote:

real 0m6.748s
user 0m6.534s
sys 0m2.826s
If kcalc is doing right this is about 90% faster :D i like that!

franzen 01-10-2016 01:35 AM

Quote:

Originally Posted by DarkVision (Post 5475625)
Two questions:
getdeps will produce a slack-required file like that:
Code:

glibc-solibs,libdvdread
while requiredbuilder gives a package each line:
Code:

glibc-solibs
libdvdread


I use comma-separated deps in my slack-repo since years(generated with requiredbuilder -b -c -p -z ), client is slapt-get, maybe it works also with linebreaks

Quote:

and i can use the -v option to add version info to slack-required:
Code:

glibc-solibs >= 2.17-x86_64-10_slack14.1
libdvdread >= 5.0.3-x86_64-1

Is there a difference between comma-separated and "one dep each line" and can version information be added to the output?
Version information can't be added. For me, this was never useful. I (almost)always have only one version of a package in my repo, and have to keep track that it fits together with the rest of the repo. In the rare cases i needed two version(lets say guile and guile1.8), the package has another name anyway. Versions might be be useful when mixing diffrent repos , but in that case, there might a lot of other things go wrong(in my opinion). In short, i don't need Version in the required string, they just make the dep-text longer.

Quote:

I did another test for ffmpeg:
...

Code:

--- slack-required.requiredbuilder        2016-01-09 19:31:22.699107496 +0100
+++ slack-required.getdeps.sorted        2016-01-09 19:32:55.302029712 +0100
@@ -3,2 +2,0 @@
-cxxlibs | gcc-g++
-db48
@@ -6 +3,0 @@
-gcc
@@ -11,2 +7,0 @@
-libXau
-libXdmcp
@@ -14,2 +8,0 @@
-libXrandr
-libXrender
@@ -17 +9,0 @@
-libogg
@@ -29 +20,0 @@
-svgalib


Beside the possibililty that getdeps contain bugs ;-), different output is intentional, depending on the package. ldd lists "too much", it lists dependencies of dependecies.
ffmpeg actually has no dependency e.g. on db48, ffmpeg depends on a package that depends on db48. While this will for now work to install ffmpeg, it might not work if the repo is combinded with the next slackware-release, as dependencies might have changed/packages removed and so on. You might be forced to rebuild working software, not for a new version, but that the required-string is correct again.

willysr 01-10-2016 02:13 AM

Quote:

Originally Posted by DarkVision (Post 5475853)
Which package? This thread is about the getdeps script.

i mean libXrandr is already included in Slackware

a4z 01-10-2016 02:45 AM

Quote:

Originally Posted by DarkVision (Post 5475866)
@franzen : again, nice job :cool:
I just compared requiredbuilder with getdeps for ffmpeg:

requiredbuilder:


getdeps:


If kcalc is doing right this is about 90% faster :D i like that!

you can run sbbdep against the DESTDIR,
http://a4z.bitbucket.org/docs/sbbdep...e_query_option

in this way it's possible to include it in a build
I think it will be even faster, with the right options for sure
and more correct, since it checks also for some internals, eg just including a required lib in a package does not necessary mean it will be found, on the other hand also files can be found that are not in a linker path but somewhere else on the system.

whole documentation:
http://a4z.bitbucket.org/docs/sbbdep/

I am moving the doc from here to the new location , so if something missing, check here
https://bitbucket.org/a4z/sbbdep/wiki/Home

ReaperX7 01-10-2016 04:50 AM

Some tools parse the .info file for dependencies from the SBo tree.

Mostly everything already in the main Slackware tree is generally not listed.

franzen 01-10-2016 05:39 AM

Quote:

Originally Posted by a4z (Post 5475880)
you can run sbbdep against the DESTDIR,
http://a4z.bitbucket.org/docs/sbbdep...e_query_option

in this way it's possible to include it in a build

I wasn't aware that this tool exists, it seems much more powerful and may do all
that i wanted to get from my get_dep-script. Unfortunately, i can't build it
on slackware 14.1, i used the instructions from your slackbuilds-buildscript, make says:
Code:

[  3%] Building CXX object CMakeFiles/sbbdep.dir/src/sbbdep/cache.cpp.o
In file included from /usr/include/c++/4.8.2/thread:35:0,
                from /usr/src/build/kaffee/sbbdep/sbbdep-0.2.5/include/sbbdep/utils/backgroundjob.hpp:6,
                from /usr/src/build/kaffee/sbbdep/sbbdep-0.2.5/src/sbbdep/cache.cpp:35:
/usr/include/c++/4.8.2/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.

Quote:

I think it will be even faster, with the right options for
sure
and more correct, since it checks also for some internals, eg just including a required lib in a package does not necessary mean it will be found, on the other hand also files can be found that are not in a linker path but somewhere else on the system.
Even faster is nice to have. My script does also search for everything in the current package what has to do with ELF, and therefore should find everything needed in /var/log/packages/.
In the installed packages, it only searches for deps which are in /etc/ld.so.conf, as the libs have to be located there to work with ldconfig and so on. If libs are somewhere in the package, this should be corrected when building a package, in my opinion.

Question to sbbdep: according to your docs, it will for my usecase output something like this:
Code:

[~]$sbbdep  --quiet --short /var/adm/packages/boost-1.49.0-x86_64-3
aaa_elflibs | bzip2, aaa_elflibs | gcc, aaa_elflibs | zlib, cxxlibs | gcc-g++, glibc | glibc-solibs, icu4c

If i understand it right from elf_libs.txt,
aaa_elflibs should be avoided to be a dependency-package where possible, as the containing libs will never get (security)updates until the next slackware-release. Therefore i remove the "or-pipes" and aaa_elflibs in my script, if the dependency is covered by another package. Do you agree with that, or is there another intention to add aaa_elflibs?

franzen 01-10-2016 06:28 AM

Quote:

Originally Posted by franzen (Post 5475917)
I wasn't aware that this tool exists, it seems much more powerful and may do all
that i wanted to get from my get_dep-script. Unfortunately, i can't build it

I worked with "cmake . -DCMAKE_INSTALL_PREFIX=/usr" for some reason.
Beside aaa_elflibs,cxxlibs and preferred solibs-packages, "sbbdep --quiet --short" seems to give me the same output as my script does.
Sbbdep is very fast, if one doesn't take in account the initial indexing. Does the index get updated, if there are new/updated packages in /var/log/packages/ ?
Sbbdep will help me with many tasks, thank you very much for creating such a great tool :cool:

For my usecase described in the first post, i think will stick to my script, as i wanted to replace the external dependency(requiredbuilder) from my build-tools by bash-code, and i
would have to "sed" the output of sbbdep to get what i want.

a4z 01-10-2016 11:05 AM

sbbdep is also available via slackbuilds.org, you can have a look at the build script in how to use the options(or not to set the CMAKE_CXX_FLAGS ;-)
http://slackbuilds.org/repository/14...?search=sbbdep

the index updates automatically, except if you disable it explicit via option of course.
so after install/remove/update its good to not disable it at least once
or run it without any options to just refresh the cache after install/remove/update.

if there is nothing to update, the internal check does not take to much time, on ssd it's nearly irrelevant, on hds it depends if files are in the cache or not if it is measurable.

I have to think about the aaa_elflibs, good that you mention this, thanks.
maybe Slackware should care about the aaa_elflibs since it can, and therefore IMHO shoud be updated, see this thread.
http://www.linuxquestions.org/questi...ge-4175537158/

not sure about the cxxlibs name, the salix dep files contain them, and as the aaa_elflibs, if you want a minimal installation you can use them.

and both packages might be a real alternative for a minimum install

I have to do some research for those packages, I already know that my personal opinion is that the distribution has to keep them actual, but my opinion might be for sure irrelevant.

do you want to file an issue?
https://bitbucket.org/a4z/sbbdep/issues

it's not a big deal to avoid these packages, or enable user filter in general what might be the better way

DarkVision 01-17-2016 05:16 AM

1 Attachment(s)
Quote:

Originally Posted by a4z (Post 5476018)
sbbdep is also available via slackbuilds.org, you can have a look at the build script in how to use the options(or not to set the CMAKE_CXX_FLAGS ;-)

I wrote my own buildscript using CMAKE_CXX_FLAGS by adding "-std=c++11" and "-lpthread" ;)

My scripts do now support requiredbuilder and sbbdep. The missing feature in sbbdep from requiredbuilder using ADD or EXCLUDE to modify the slack-required file is a pitty but i added some code to emulate that missing feature. While i'm still interested in the get_deps.sh script it looks like this script does support that in the same way as requiredbuilder does.

Since no one was able to answer my question about the comma-separated list of dependencies i modified the get_deps.sh script, see attached patch.

I also added some code to add the version of the dependency, just use get_deps.sh --version.

Default output:
Code:

ffmpeg
gcc
gcc-g++
glibc-solibs

Using --version:
Code:

ffmpeg >= 2.8.3-x86_64-1slp
gcc >= 5.3.0-x86_64-1
gcc-g++ >= 5.3.0-x86_64-1
glibc-solibs >= 2.22-x86_64-4

Patch:
Code:

--- get_deps.sh.orig        2016-01-10 07:49:56.000000000 +0100
+++ get_deps.sh        2016-01-17 11:55:57.559366402 +0100
@@ -18,6 +18,9 @@
 #along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+VERSION=false
+[ x$1 = x"--version" ] && VERSION=true
+
 EXCLUDE="${EXCLUDE:-}"
 ADD="${ADD:-}"
 
@@ -86,7 +89,20 @@
                                        tmpvar="$(echo "$package" | tr ' ' '\n' | sed "/^$/d" | sort -u | tr '\n' '|' | sed 's/|$//' )"
                                        package="$tmpvar"
                                fi
-                                echo "$package" >> $tmpfile
+                                if ${VERSION:-false}; then
+                                    for pkg in `find /var/log/packages -type f -name "$package*" -printf "%f\n" | sort`; do
+                                        pkgname=${pkg%-*} # build
+                                        pkgname=${pkgname%-*} # arch
+                                        pkgname=${pkgname%-*} # version
+                                        if [ x"$pkgname" == x"$package" ]; then
+                                                pkgver=" >= ${pkg#$pkgname-}" # name
+                                                break
+                                            fi
+                                    done
+                                else
+                                pkgver=""
+                                fi
+                                echo "$package$pkgver" >> $tmpfile
                                # remove $elfneeded from inputfile($needed), to check if something wasn't found in the ld-directories
                                sed -i "/$elfneeded/d" $needed
                        fi
@@ -116,7 +132,8 @@
        sed -i "/$dep/d" $tmpfile
 done
 
-[ -s "$tmpfile" ] && sort -u < $tmpfile | tr '\n' ',' | sed "s/,$//"
+#[ -s "$tmpfile" ] && sort -u < $tmpfile | tr '\n' ',' | sed "s/,$//"
+[ -s "$tmpfile" ] && sort -u < $tmpfile | sed "s/,$//"
 
 # cleanup
 for cleanup in $internal $needed $tmpfile

I will have a deeper look into the get_deps.sh script... it really looks interesting and might be a good way to include creating slack-required files automatically to my own buildscripts if neither requiredbuilder nor sbbdep is installed.

franzen 01-20-2016 01:51 AM

Quote:

Originally Posted by DarkVision (Post 5479591)
Since no one was able to answer my question about the comma-separated list of dependencies i modified the get_deps.sh script, see attached patch.

I also added some code to add the version of the dependency, just use get_deps.sh --version.

Thanks, i'll have a closer look and will try to build the version-feature into, hopefully
avoiding additional "find"s for better performance.

The comma-separated list has no real technical difference to multiple rows, so far i know.
But if listet in an a PACKAGES.TXT in a repository, i think it disturbs if there are hundreds of extralines.

franzen 01-20-2016 02:01 AM

Quote:

Originally Posted by a4z (Post 5476018)
not sure about the cxxlibs name, the salix dep files contain them, and as the aaa_elflibs, if you want a minimal installation you can use them.

cxxlibs seem to be removed from -current, these libs seem to be added aaa_elflibs.
One of the ideas of aaa_elflibs is to avoid upgrade-issues, so old libraries will have
to there, also if newer versions gonna be added. It will grow if is has to contain "everything", which isn't very suitable for a minimal install i think.


All times are GMT -5. The time now is 07:40 AM.