LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (https://www.linuxquestions.org/questions/slackware-14/)
-   -   Call for testers - script to resolve dependencies for SBo packages (https://www.linuxquestions.org/questions/slackware-14/call-for-testers-script-to-resolve-dependencies-for-sbo-packages-4175637160/)

igadoter 08-26-2018 08:28 AM

Call for testers - script to resolve dependencies for SBo packages
 
It is just included below. I tested for several packages - seems it works. I use lynx to dump web page - but there should be easier way. The script is called by dep-resolv.sh
Code:

$ ./dep-resolv.sh link-to-top-level-package
you are of course can name it as you wish. There are two parameters DEPS - name of file links are stored in, HASH - it should be complete gibberish as it is used to hash lines in the DEPS, default values are DEPS=deps and HASH=AnzUiWw

The command
Code:

$ DEPS=foo HASh=OlaIARE ./depres.sh http://www.slackbuilds.org/repositor...ipboardViewer/
generates the file ./foo in the same directory as the script
Code:

$ cat foo
http://www.slackbuilds.org/repository/14.2/desktop/ClipboardViewer/
http://www.slackbuilds.org/repository/14.2/libraries/libgnomeui/
http://www.slackbuilds.org/repository/14.2/libraries/libbonoboui/
http://www.slackbuilds.org/repository/14.2/libraries/libgnomecanvas/
http://www.slackbuilds.org/repository/14.2/libraries/libgnome/
http://www.slackbuilds.org/repository/14.2/system/gnome-vfs/
http://www.slackbuilds.org/repository/14.2/libraries/libbonobo/
http://www.slackbuilds.org/repository/14.2/misc/gnome-mime-data/
http://www.slackbuilds.org/repository/14.2/libraries/ORBit2/

The order of compilation is from bottom to top. Use tac to get in reverse order.

Here is the script itself
Code:

#!/bin/bash

#
# Create dependencies file for sbopkg packages
#
 
DEPS=${DEPS:=deps} ; touch $DEPS
DEPSHASH=${DEPS}hash ; touch $DEPSHASH
TMPF=tmpfile ; touch $TMPF
TMPF_OUT=tmpfile_out ; touch $TMPF
HASH_POSTFIX=${HASH:=AnzUiWw}  # It should complete gibberish
LYNX_OPT="-nonumbers -listonly -dump"

# Remove repeated links
function slip_and_add () {
        cat $DEPS | grep -v "$ADDL" > $TMPF
        echo "$ADDL" >> $TMPF
        cat  $TMPF > $DEPS
 }

# Pick deps for package pointed by package addres 
function get_deps () {
                lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT
}

# Build hash for dependencies file
function hash () {
                  for i in $(seq 1 `cat $DEPS | wc -l` )
                  do
                  echo ${i}$HASH_POSTFIX
                  done |  paste - $DEPS > $DEPSHASH 
}

echo $1 > $DEPS  # Put the link of top package to be installed into dependencies file 
hash
let PROC=1
# Terminate if there is no more links to be processsed
  until  [ $PROC -gt `cat $DEPS | wc -l` ]; do
      LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"
      get_deps                    # Get dependencies for processed package 
      while read addr; do 
              ADDL=$addr slip_and_add 
      done < $TMPF_OUT                # Add them to dependencies file of the top package
      hash                      # Rebuild hash for dependencies file
      let PROC=$PROC+1            # Point to next link to be processed in dependencies file
  done
 
# Make clean 
rm $DEPSHASH 
rm $TMPF
rm $TMPF_OUT

Dependencies which are not present on SlackBuilds.org cannot be resolved (clearly) by this scripte.

igadoter 08-27-2018 01:40 PM

Here is script which can made things little more interesting
Code:

#!/bin/bash

 
cat $1 |sed 's/repository/slackbuilds/' | while read addr ; do
    pdir=`echo $addr | cut -d'/' -f7` ;
    mkdir $pdir ;         
    cd $pdir ; 
    curl  `echo ${addr}`\{README,${pdir}.info,${pdir}.SlackBuild,doinst.sh,slack-desc\} -o \#1 ;
    cd ..  ; 
done

Run the script in the first post, say html addresses are in 'foo' file. Then run
Code:

$ cat foo | dc_copy.sh -
Code:

dc_copy.sh
it is just the name of the second script given by me. Notice command ends with '-' sign - usually it means standard input.

Now do something like that
Code:

$ ls -R
On my computer it is
Code:

$ ls -R
.:
ClipboardViewer  dc_copy.sh  dep-resol-v2.sh  depres.sh  foo    gnome-mime-data  iso.old    libbonoboui  libgnomecanvas  scratch
ORBit2          dc_copy.sh~  dep-resolv.sh    deps      foobar  gnome-vfs        libbonobo  libgnome    libgnomeui

./ClipboardViewer:
ClipboardViewer.SlackBuild  ClipboardViewer.info  README  doinst.sh  slack-desc

./ORBit2:
ORBit2.SlackBuild  ORBit2.info  README  doinst.sh  slack-desc

./gnome-mime-data:
README  doinst.sh  gnome-mime-data.SlackBuild  gnome-mime-data.info  slack-desc

./gnome-vfs:
README  doinst.sh  gnome-vfs.SlackBuild  gnome-vfs.info  slack-desc

./libbonobo:
README  doinst.sh  libbonobo.SlackBuild  libbonobo.info  slack-desc

./libbonoboui:
README  doinst.sh  libbonoboui.SlackBuild  libbonoboui.info  slack-desc

./libgnome:
README  doinst.sh  libgnome.SlackBuild  libgnome.info  slack-desc

./libgnomecanvas:
README  doinst.sh  libgnomecanvas.SlackBuild  libgnomecanvas.info  slack-desc

./libgnomeui:
README  doinst.sh  libgnomeui.SlackBuild  libgnomeui.info  slack-desc


zsd 08-27-2018 02:19 PM

You might consider adding a trailing / to $1 if there isn't one there. E.g.,
Code:

# Put the link of top package to be installed into dependencies file
case "$1" in
  (*/)
    echo $1 > $DEPS 
    ;;
  (*)
    echo $1/ > $DEPS
    ;; 
esac

Also, is your regexp the same as '(/.*){7}' ? In any case, you might consider putting '' around your regexp and losing the \s.

You might also consider the virtue of replacing
Code:

DEPSHASH=${DEPS}hash ; touch $DEPSHASH
TMPF=tmpfile ; touch $TMPF
TMPF_OUT=tmpfile_out ; touch $TMPF

(the second line is presumably supposed to end with $TMPF_OUT?)
with
Code:

DEPSHASH=`mktemp`
TMPF=`mktemp`
TMPF_OUT=`mktemp`

And maybe even adding
Code:

trap "rm -f $TMPF $TMPF_OUT $DEPSHASH" 0 1 2 3 15
(or similar) after the calls to mktemp, and then you can delete the three rm's at the end of the program.

Having said that, it worked for the couple of cases I tried (after I realized I needed the trailing / on the link).

igadoter 08-28-2018 03:42 AM

Quote:

Originally Posted by zsd (Post 5896670)
You might consider adding a trailing / to $1 if there isn't one there

Done. Say someone forget to add trailing / to the link.
Quote:

Also, is your regexp the same as '(/.*){7}' ? In any case, you might consider putting '' around your regexp and losing the \s.
yes. I tried '(/.*){7} but I was unable to make it works as I expected. The regexp is very primitive: it just counts number of / in link.

Thanks for feedback and comments. I just done everything as you suggested.

Below is rewritten script
Code:

#!/bin/bash

#
# Create dependencies file for sbopkg packages
#
 
DEPS=deps ; touch $DEPS

DEPSHASH=`mktemp`
TMPF=`mktemp`
TMPF_OUT=`mktemp`
HASH_POSTFIX=${HASH:=AnzUiWw}  # It should be complete gibberish
LYNX_OPT="-nonumbers -listonly -dump"

# Remove duplicated links
function slip_and_add () {
        cat $DEPS | grep -v "$ADDL" > $TMPF
        echo "$ADDL" >> $TMPF
        cat  $TMPF > $DEPS
 }

# Pick deps for package pointed by package addres 
function get_deps () {
                lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT
}

# Build hash for dependencies file
function hash () {
                  for i in $(seq 1 `cat $DEPS | wc -l` )
                  do
                  echo ${i}$HASH_POSTFIX
                  done |  paste - $DEPS > $DEPSHASH 
}

echo $1 > $DEPS  # Put the link of top package to be installed into dependencies file 
hash
let PROC=1
# Terminate if there is no more links to be processsed
  until  [ $PROC -gt `cat $DEPS | wc -l` ]; do
      LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"
      get_deps                    # Get dependencies for processed package 
      while read addr; do 
              ADDL=$addr slip_and_add 
      done < $TMPF_OUT                # Add them to dependencies file of the top package
      hash                      # Rebuild hash for dependencies file
      let PROC=$PROC+1            # Point to next link to be processed in dependencies file
  done
# Make clean 
trap "rm -f $TMPF $TMPF_OUT $DEPSHASH" 0 1 2 3 15
cat $DEPS

there are changes in the script I forgot to mention: I added at the last line
Code:

cat $DEPS
to make it possible to run script in pipe, eg.
Code:

./depres.sh | sed 's/foo/bar/`
so to run it silently redirect to /dev/null
Code:

./depres.sh > /dev/null

zsd 08-28-2018 06:47 AM

Quote:

Quote:

Originally Posted by zsd 
You might consider adding a trailing / to $1 if there isn't one there
Done. Say someone forget to add trailing / to the link.
It seems you didn't add the "add / to the end of $1 if it isn't there" change.

I think the change to output $DEPS at the end is a big improvement.
1) You might want use tac instead of cat, so that the dependencies will come out in the order one has to build them, and
2) since you output the contents of that file, you might just use DEPS=`mktemp` and add $DEPS to the list of files that the trap command rm's.

The trap should go right after the files are created with mktemp... if someone ^C's the script while it is working, before it gets to the trap command, the temp files won't be deleted, which is probably not what is desired. (But, having said that, it is your program.)

I have my version of your program working with grep -E '(/.*){7}' > $TMPF_OUT; while yours (clearly) works too, I am surprised you had issues with the .* version. Huh.

You might also put in this code (feel free to use it if you like) before using $1 for the first time:

Code:

if [ -z "$1" ]
then
    echo >&2 "$0: need exactly one argument, a link to a Slackbuild"
    echo >&2 "Example:  $0 http://www.slackbuilds.org/repository/14.2/graphics/shotwell/"
    exit 1
fi

My last (?) suggestion for you is to put a comment at the top saying what the program does and who wrote it. (Credit where credit is due!) And maybe even some license statement, saying it is in the public domain, or free for any use, or for people to send you 1 kg of gold when it is used, or ...

igadoter 08-28-2018 08:39 AM

@zsd thank you. I greatly appreciate your help. I am just trying to do some useful script. What I started is just hand-made - any improvement is welcomed. I need to read about trap command ;(

Edit: @zsd it seems that '(/*){7}' works due to the presence of apostrophes. I will have a look at this.

zsd 08-29-2018 08:55 AM

Quote:

Originally Posted by igadoter (Post 5896998)
@zsd thank you. I greatly appreciate your help. I am just trying to do some useful script. What I started is just hand-made - any improvement is welcomed. I need to read about trap command ;(

Edit: @zsd it seems that '(/*){7}' works due to the presence of apostrophes. I will have a look at this.

Glad to help out. Writing shell scripts can be tricky because there are so many ways to do any given thing. (And the shell is a much more powerful programming language than most people recognize.)

igadoter 08-29-2018 07:01 PM

Below is script for testing dependency file. It just gathers information from all *.info files. So the first thing is to download the *.info file for every package in dependency file, I'm making directory and put inside *.info file. https://www.linuxquestions.org/quest...0/#post5896649 This is very basic test but it gives some assurance.
Code:

#!/bin/bash

#
# Compare dependencies with the contents of *.info files


DEPS=$1  # file of dependencies
ROOT=$2  # root directory
          # in $ROOT are located directories of package from $DEPS

TMPF_NFO=`mktemp`
TMPF_DPS=`mktemp`
trap "rm -f $TMPF_NFO $TMPF_DPS"  0 1 2 3 15

find $ROOT -name \*.info -exec cat {} \; | grep -e PRGNAM -e REQUIRES |       
                                          cut -d'"' -f2 |
                                          tr ' ' '\n' |
                                          sort -u | grep -v \^\$ > $TMPF_NFO

cat $DEPS | cut -d'/' -f7 | sort > $TMPF_DPS 

# Assertion $TMPF_NFO and $TMPF_DPS are identical
diff $TMPF_NFO $TMPF_DPS
if [ $? -eq 0 ]; then
    echo test: $DEPS OK!
else
    echo test: $DEPS FAILURE! 
fi

you can also redirect the contents of files $TMPF_NFO and $TMPF_DPS to terminal.

igadoter 08-30-2018 04:25 PM

I extended the first basic script to be able to work with multiple packages. Just feed the script with file which contains links to packages. Links should have form as being dumped from depositories web pages. I run script on bunch of SBo packages for some X terminals: gnome-terminal, etc. The output should contain all dependencies to build all these terminals
Code:


http://www.slackbuilds.org/repository/14.2/system/Eterm/
http://www.slackbuilds.org/repository/14.2/system/aterm/
http://www.slackbuilds.org/repository/14.2/system/cool-retro-term/
http://www.slackbuilds.org/repository/14.2/system/fbterm/
http://www.slackbuilds.org/repository/14.2/system/gnome-terminal/
http://www.slackbuilds.org/repository/14.2/system/graphterm/
http://www.slackbuilds.org/repository/14.2/system/lilyterm/
http://www.slackbuilds.org/repository/14.2/system/lxterminal/
http://www.slackbuilds.org/repository/14.2/system/mlterm/
http://www.slackbuilds.org/repository/14.2/system/qterminal/
http://www.slackbuilds.org/repository/14.2/system/roxterm/
http://www.slackbuilds.org/repository/14.2/system/termenu/
http://www.slackbuilds.org/repository/14.2/system/terminator/
http://www.slackbuilds.org/repository/14.2/system/terminology/
http://www.slackbuilds.org/repository/14.2/system/termite/
http://www.slackbuilds.org/repository/14.2/system/termsaver/
http://www.slackbuilds.org/repository/14.2/system/tinyterm/
http://www.slackbuilds.org/repository/14.2/system/vinterm/
http://www.slackbuilds.org/repository/14.2/system/wterm/

I named script as dpsres.sh and put all links in trms, now issue command
Code:

$ ./dpsres.sh trms
it creates dependency file trms.dps and print the same file on output
Code:

http://www.slackbuilds.org/repository/14.2/system/Eterm/
http://www.slackbuilds.org/repository/14.2/libraries/imlib2/
http://www.slackbuilds.org/repository/14.2/libraries/libast/
http://www.slackbuilds.org/repository/14.2/system/aterm/
http://www.slackbuilds.org/repository/14.2/system/cool-retro-term/
http://www.slackbuilds.org/repository/14.2/libraries/qmltermwidget/
http://www.slackbuilds.org/repository/14.2/system/fbterm/
http://www.slackbuilds.org/repository/14.2/system/gnome-terminal/
http://www.slackbuilds.org/repository/14.2/libraries/libgnome/
http://www.slackbuilds.org/repository/14.2/development/rarian/
http://www.slackbuilds.org/repository/14.2/system/gnome-vfs/
http://www.slackbuilds.org/repository/14.2/libraries/libbonobo/
http://www.slackbuilds.org/repository/14.2/misc/gnome-mime-data/
http://www.slackbuilds.org/repository/14.2/libraries/ORBit2/
http://www.slackbuilds.org/repository/14.2/system/graphterm/
http://www.slackbuilds.org/repository/14.2/network/tornado/
http://www.slackbuilds.org/repository/14.2/libraries/futures/
http://www.slackbuilds.org/repository/14.2/python/python-certifi/
http://www.slackbuilds.org/repository/14.2/python/singledispatch/
http://www.slackbuilds.org/repository/14.2/python/backports-abc/
http://www.slackbuilds.org/repository/14.2/python/six/
http://www.slackbuilds.org/repository/14.2/system/lilyterm/
http://www.slackbuilds.org/repository/14.2/system/lxterminal/
http://www.slackbuilds.org/repository/14.2/system/mlterm/
http://www.slackbuilds.org/repository/14.2/system/qterminal/
http://www.slackbuilds.org/repository/14.2/libraries/qtermwidget/
http://www.slackbuilds.org/repository/14.2/libraries/lxqt-build-tools/
http://www.slackbuilds.org/repository/14.2/libraries/qt5/
http://www.slackbuilds.org/repository/14.2/libraries/libinput/
http://www.slackbuilds.org/repository/14.2/libraries/libxkbcommon/
http://www.slackbuilds.org/repository/14.2/graphics/graphviz/
http://www.slackbuilds.org/repository/14.2/python/pyudev/
http://www.slackbuilds.org/repository/14.2/python/python-evdev/
http://www.slackbuilds.org/repository/14.2/libraries/libwacom/
http://www.slackbuilds.org/repository/14.2/system/roxterm/
http://www.slackbuilds.org/repository/14.2/system/termenu/
http://www.slackbuilds.org/repository/14.2/system/terminator/
http://www.slackbuilds.org/repository/14.2/python/psutil/
http://www.slackbuilds.org/repository/14.2/system/terminology/
http://www.slackbuilds.org/repository/14.2/development/meson/
http://www.slackbuilds.org/repository/14.2/libraries/efl/
http://www.slackbuilds.org/repository/14.2/python/python3/
http://www.slackbuilds.org/repository/14.2/development/ninja/
http://www.slackbuilds.org/repository/14.2/system/termite/
http://www.slackbuilds.org/repository/14.2/libraries/vte3-ng/
http://www.slackbuilds.org/repository/14.2/development/vala/
http://www.slackbuilds.org/repository/14.2/libraries/pcre2/
http://www.slackbuilds.org/repository/14.2/system/termsaver/
http://www.slackbuilds.org/repository/14.2/system/tinyterm/
http://www.slackbuilds.org/repository/14.2/system/vinterm/
http://www.slackbuilds.org/repository/14.2/libraries/libconfig/
http://www.slackbuilds.org/repository/14.2/system/wterm/

dependency file is created so the order of build is as previous from below to top. As you see it is quite a lot number of dependencies to build these terminals. So the script also creates directory for every package (terminal, not every dependency !) with its own full dependency file and 'run-script' which allows only some packages to be built - say I don't want to build gnome-terminal - to avoid this just chmod -x 'run-script'. By default all run scripts are non-executable. So far these are just empty files.

Here is the script, it is short, clearly no warranties!
Code:


    1        #!/bin/bash
    2       
    3        #
    4        # WMD - wpn of mss dstr
    5        # Create dependencies file for multiple SBo packages
    6        # Command: ./depres.sh file-with-sbo-links
    7        # 
    8       
    9        # TODO: *.run script
    10        # it just do the thing one is doing by hand:
    11        # grab source tarball,  extract file with slackbuild script and run
    12        # TODO: manage broken links - they may appear due to behavior of grep \(/.?+\)\{7\}
    13        # simple regexp '(/*){7}' was the cause of broken  links and script went in infinite loop
    14           
    15        #
    16        # ASS - assertion for dep file: dependencies for name are below the name
    17        #
    18       
    19        W_DIR=`pwd`/                      # working directory
    20        PACKAGES=$1
    21       
    22        if [ ! -f "$PACKAGES" ]; then
    23          exit 1
    24        fi
    25       
    26        if [  -s ${PACKAGES}.dps ]; then
    27            rm -f $PACKAGES.dps
    28        fi   
    29            DEPS=${PACKAGES}.dps; touch $DEPS      # common deps file for all packages in $PACKAGES
    30           
    31            HASH_POSTFIX=${HP:=AnuiZWw} 
    32            LYNX_OPT="-nonumbers -listonly -dump "
    33           
    34        function get_deps () {
    35                    TMPF=`mktemp`
    36                    TMPF_OUT=`mktemp`
    37                    trap " rm -f $TMPF $TMPF_OUT " 0 1 2 3 15
    38                    # grab all links pointing to dependencies
    39                        lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT
    40                        if [  -s $TMPF_OUT ]; then
    41                            while read  _addr; do
    42                      cat $DEPS_PKG | grep -v "$_addr" > $TMPF
    43                      echo "$_addr" >> $TMPF
    44                      cat $TMPF > $DEPS_PKG
    45                    done < $TMPF_OUT
    46                    _addr=""
    47                fi
    48                rm -f $TMPF $TMPF_OUT  # clear garbage   
    49        }
    50       
    51        # Build hash for dependencies file
    52        function hash () {
    53                          for i in $(seq 1 `cat $DEPS_PKG | wc -l` )
    54                          do
    55                          echo ${i}$HASH_POSTFIX
    56                          done |  paste - $DEPS_PKG > $DEPSHASH 
    57        }
    58       
    59        function merge () {
    60                  TMPF=`mktemp`
    61                  trap " rm -f $TMPF "  0 1 2 3 15
    62              if [ -s $DEPS_PKG ]; then
    63                while read _addr ; do
    64                          cat $DEPS | grep -v "$_addr" > $TMPF
    65                          echo "$_addr" >> $TMPF
    66                          cat $TMPF > $DEPS
    67                done < $DEPS_PKG
    68                _addr=""                           
    69              fi   
    70              rm -f $TMPF        # clean garbage
    71        }
    72       
    73        #
    74        # Main: loop through all links in $PACKAGES
    75        #
    76       
    77        while read PKG; do                    # pick next package from $PACKAGES
    78              DEPSHASH=`mktemp`
    79              DEPS_PKG=`mktemp`                # $DEPS_PKG - deps file for package $PKG
    80              trap "rm -f $DEPSHASH $DEPS_PKG" 0 1 2 3 15
    81              # 
    82              # build  $DEPS_PKG 
    83              #
    84              echo "$PKG" > $DEPS_PKG     
    85              let PROC=1
    86                  hash
    87                  # Terminate if there is no more links to be processsed
    88              until  [ $PROC -gt `cat $DEPS_PKG| wc -l` ]; do
    89                  LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"  # get link by its hash 
    90                  # trap: regexp \(/.?+\)\{7} may return broken link
    91                  # strict policy: exit  at the first error
    92                  # or don't care: go to process next package 
    93                  #if [ `echo "$LNK" | grep -e 'libraries/$' -e 'system/$'` ]; then # strict policy
    94                      #exit 1
    95                  #fi 
    96                  get_deps                    # get dependencies for $LNK and store them in $DEPS_PKG 
    97                  hash                        # rebuild hash for $DEPS_PKG
    98                  # find next package to process
    99                  PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
  100                  let PROC="$PROC"+1            # point to next link to process in $DEPS_PKG
  101              done
  102             
  103              # make package directory
  104              PKGNAM=`echo $PKG | cut -d'/' -f7`      # it should be the same as $PKGNAM in *.info
  105              mkdir "${W_DIR}$PKGNAM"
  106              cd "${W_DIR}$PKGNAM"
  107       
  108              # find link web page for package with *.info
  109              # grab all files and store in package directory
  110              PKG_DATA=`echo $PKG | sed 's/repository/slackbuilds/'`
  111              curl  "$PKG_DATA"\{README,${PKGNAM}.info,${PKGNAM}.SlackBuild,doinst.sh,slack-desc\} -o \#1 ;
  112             
  113              cat $DEPS_PKG >  ${PKGNAM}.deps  # full deps file for package is also stored
  114             
  115              RBTT="${PKGNAM}.run                # have choice which packages from $PACKAGES to build"
  116              if [ -s "$RBTT" ]; then
  117                rm -f "$RBTT"
  118                touch "$RBTT"
  119              else
  120                  touch "$RBTT"
  121              fi
  122                 
  123              cd ${W_DIR}                      # go back to worling directory
  124             
  125              # final loop step: merge $DEPS_PKG  with $DEPS - keeping ASS valid
  126              merge
  127             
  128              rm -f $DEPS_PKG $DEPSHASH    # clean garbage                         
  129        done  <  $PACKAGES             
  130                                       
  131        cat $DEPS       
  132       
  133        # marks:
  134        # once there are all *.info ar in place test dependency file running test.sh script
  135        #
  136        # chmod +x W_DIR/*.run to build all packages
  137        # build just follow dependency file
  138        # 
  139       
  140        # if all this is correct one can build all SBo which depends only on SBo  in one run
  141        # prerequsities: full Slackware install

I hope at final stage it will be fully functional - so I don't have to any more do these by my own - step by step.

igadoter 09-03-2018 04:39 AM

The WMD script as it is now works not quite correct. In the code posted here https://www.linuxquestions.org/quest...0/#post5898172 before the line 100 put this
Code:

PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
so now code reads
Code:

    98      hash                        # rebuild hash for $DEPS_PKG
    99      # find next package to process
    100      PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
    101      let PROC="$PROC"+1            # point to next link to process in $DEPS_PKG

simple
Code:

PROC="$PROC"+1
was not enough to find next link to process.

igadoter 09-04-2018 07:23 PM

Hi, I have reached milestone and jut created something really useful, this is new version of script
Code:

    1        #!/bin/bash -x
    2       
    3        #
    4        # Create dependency file for multiple SBo packages
    5        # 
    6           
    7        #
    8        # assertion for dependency file: dependencies for name are below the name
    9        # keep assertion to be true
    10        #
    11       
    12        PACKAGES=$1
    13       
    14        if [ ! -f "$PACKAGES" ]; then
    15          exit 1
    16        fi
    17       
    18        if [  -s ${PACKAGES}.deps ]; then
    19            rm -f $PACKAGES.deps
    20        fi   
    21       
    22        DEPS=${PACKAGES}.deps; touch $DEPS      # common deps file for all packages in $PACKAGES
    23        HASH_POSTFIX=${HP:=AnuiZWw}
    24       
    25        # -dump  dump web page
    26        # -listonly list only links
    27        # -nonumbers links are without numbers 
    28        LYNX_OPT="-nonumbers -listonly -dump "
    29           
    30        function get_deps () {
    31                    TMPF=`mktemp`
    32                    TMPF_OUT=`mktemp`
    33                    trap " rm -f $TMPF $TMPF_OUT " 0 1 2 3 15
    34                    # grab all links pointing to dependencies
    35                        lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT
    36                        if [  -s $TMPF_OUT ]; then
    37                            while read  _addr; do
    38                      cat $DEPS_PKG | grep -v "$_addr" > $TMPF
    39                      echo "$_addr" >> $TMPF
    40                      cat $TMPF > $DEPS_PKG
    41                    done < $TMPF_OUT
    42                    _addr=""
    43                fi
    44                rm -f $TMPF $TMPF_OUT  # clear garbage   
    45        }
    46       
    47        # Build hash for dependencies file
    48        function hash () {
    49                          for i in $(seq 1 `cat $DEPS_PKG | wc -l` )
    50                          do
    51                          echo ${i}$HASH_POSTFIX
    52                          done |  paste - $DEPS_PKG > $DEPSHASH 
    53        }
    54       
    55        function merge () {
    56                  TMPF=`mktemp`
    57                  trap " rm -f $TMPF "  0 1 2 3 15
    58              if [ -s $DEPS_PKG ]; then
    59                while read _addr ; do
    60                          cat $DEPS | grep -v "$_addr" > $TMPF
    61                          echo "$_addr" >> $TMPF
    62                          cat $TMPF > $DEPS
    63                done < $DEPS_PKG
    64                _addr=""                           
    65              fi   
    66              rm -f $TMPF        # clean garbage
    67        }
    68       
    69        #
    70        # main: perform  all links in $PACKAGES
    71        #
    72        while read PKG; do                   
    73              DEPSHASH=`mktemp`
    74              DEPS_PKG=`mktemp`                # $DEPS_PKG: dependency file for $PKG
    75              trap "rm -f $DEPSHASH $DEPS_PKG" 0 1 2 3 15
    76              # 
    77              # perform  $DEPS_PKG 
    78              #
    79              echo "$PKG" > $DEPS_PKG     
    80              let PROC=1
    81                  hash
    82                  # Terminate if there is no more links to be processsed
    83              until  [ $PROC -gt `cat $DEPS_PKG| wc -l` ]; do
    84                  LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"  # get link by its hash 
    85                  # trap: regexp \(/.?+\)\{7} may return broken link
    86                  # strict policy: exit  at the first error
    87                  # or don't care: go to process next package 
    88                  #if [ `echo "$LNK" | grep -e 'libraries/$' -e 'system/$'` ]; then # strict policy
    89                      #exit 1
    90                  #fi 
    91                  get_deps                    # get dependencies for $LNK and store them in $DEPS_PKG 
    92                  hash                        # rebuild hash for $DEPS_PKG
    93                  # find next package to process
    94                  PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
    95                  let PROC="$PROC"+1            # point to next link to process in $DEPS_PKG
    96              done
    97             
    98              # merge $DEPS_PKG  with $DEPS - keeping ASS valid
    99              merge
  100             
  101              rm -f $DEPS_PKG $DEPSHASH    # clean garbage                         
  102        done  <  $PACKAGES             
  103       
  104       
  105        # main loop for creating directories and downloading package data
  106        while read PKG ; do
  107              PKGNAME=`basename $PKG`
  108              PKG_DATA=`echo $PKG | sed 's/repository/slackbuilds/; s/\/$/.tar.gz/'`
  109              curl  "$PKG_DATA" | tar -zxvf -
  110             
  111              TOP_DIR=`pwd`
  112              cd "$PKGNAME"
  113              # script to download sources and additonal data
  114              cat <<'EOF' > "${PKGNAME}-run.sh"       
  115        #!/bin/bash               
  116       
  117        # run basic things
  118        # download packages verify checksum
  119       
  120        . `ls *.info`              # read contents of *.info file
  121       
  122        if [  "`uname -m`" = "_x86_64" ]; then
  123            ARCH="_x86_64"
  124        else
  125            ARCH=""
  126        fi       
  127       
  128        # curl options
  129        # -L means follow redirections
  130        # -O save target with remote name
  131        CURL_OPT=" -L -O "             
  132       
  133        declare URLS=(${DOWNLOAD}$ARCH)  # for multiple downloads
  134        declare CHCKS=(${MD5SUM}$ARCH)
  135       
  136        # build checksums file
  137        echo -n > $PRGNAM.md5
  138        for i in ${!URLS[@]} ; do
  139            echo ${CHCKS[$i]} ${URLS[$i]##*/} >> $PRGNAM.md5
  140        done
  141       
  142        #grab sources   
  143        for url in  ${URLS[@]} ; do
  144          curl -L -O "$url"
  145        done 
  146       
  147        # verify checksums 
  148        md5sum -c $PRGNAM.md5
  149        if [ $? -ne 0 ] ; then
  150            echo Checksum failed!
  151            exit 1
  152        fi   
  153        EOF
  154              cd "$TOP_DIR"                     
  155        done < $DEPS
  156       
  157        # flush to stdout                               
  158        cat $DEPS       
  159       
  160  # End of script

It creates dependency file with suffix .deps and sub-directories for all packages in dependency file. Also contents of dependency file is flushed to stdout. In all sub-directories there are *-run.sh scripts. They download sources and and any other data, verify checksums. They have to be run in the same subdirectory as other files - *info -slackbuilds, etc. Here example. I called the script sbo-res.sh. Run
Code:


$ ./sbo-res.sh sbo-links

It creates sbo-links.deps. Now make all *-run.sh scripts executable
Code:

$ find . -name \*-run.sh -exec chmod +x {} \;
then run these scripts
Code:

find . -name \*-run.sh  | while read line ; do DIR=`dirname $line` ; RUNS=`basename $line` ;  cd "$DIR" ;  ./"$RUNS"  ;  cd  .. ; done
at the end you obtain all targets. Builds goes along sbo-links.deps from bottom to the top. Enjoy!

TODO: there are the two things: 1) to make better test script 2) handle alternative: 64 versus 32 bit. Some packages don't compile on 32 bits systems and vice versa

pan64 09-05-2018 12:57 AM

If I can give you some hints (not about the functionality, but quality and speed):
1. please try www.shellcheck.net to check your script and fix the errors/issues reported.
2. useless use of cat
3. instead of basename and dirname you can use ${VAR##*/} and ${VAR%/*}
4. in general avoid backticks or $( ) if possible, use shell built-ins instead
5. do not use awk|grep|cut|cat|sed|wc|whatever chains, usually a single awk (or perl/python/... ) oneliner (script) will do the job

6. use mktemp only once, and all the other tmp dirs should be located inside this temp dir - in this case you can clean up by removing that single dir.
7. there is no error handling.

igadoter 09-05-2018 02:48 AM

Quote:

Originally Posted by pan64 (Post 5900030)
1. please try www.shellcheck.net to check your script and fix the errors/issues reported.

tested: looks good.
Quote:

4. in general avoid backticks or $( ) if possible, use shell built-ins instead
I think this just improves readability.
Quote:

5. do not use awk|grep|cut|cat|sed|wc|whatever chains, usually a single awk (or perl/python/... ) oneliner (script)
of course but this is bash script with minimum requirements. Essentially I am using 'mktemp` to mimic local variables.
Quote:

7. there is no error handling
It is one-run script. At any error just exit. So I think killing parent process should be enough. Plus kind of log.


But saying that thanks for your feedback.

individual 09-05-2018 02:36 PM

Hey, I like the idea of generating links to dependencies for a SlackBuild, but your script is difficult to follow because of bash's oddities and the use of global variables. In my opinion a traditional programming language would be a better choice for this. I'm not trying to hijack your thread, but I wrote an equivalent script (for your first version) in Perl. I'd love to hear your thoughts on it.
Code:

#!/usr/bin/perl
# Resolve dependencies for a SlackBuild.

use strict;
use warnings;
use HTTP::Tiny;

use constant BASE_URL => 'http://www.slackbuilds.org';

sub resolve_deps {
    my $url = shift || die;

    print "$url\n";

    my $html = HTTP::Tiny->new->get($url)->{content};
    resolve_deps(BASE_URL . $_) for ($html =~ m!(/repository/.*?)'!g);
}

resolve_deps(@ARGV);

EDIT: I forgot to add that this works with Perl 5.8.4 and up, so it is portable.

igadoter 09-07-2018 10:37 AM

Quote:

Originally Posted by individual (Post 5900227)
Hey, I like the idea of generating links to dependencies for a SlackBuild, but your script is difficult to follow because of bash's oddities and the use of global variables.

This is my fault with these oddities. I am just rewriting the script - it will be looking even more odd - but more like any other programming language. I don't really feel to be someone who can you give advice. I was running for years Linux - I get to use to work with shell. Your script is calling recursively function: It would be good to keep trace of this and limit the number of recursive calls. You may also look at so-called tail-recursion - function with tail-recursion can be easily rewritten as common loop.


All times are GMT -5. The time now is 03:09 PM.