LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 08-26-2018, 08:28 AM   #1
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557
Blog Entries: 1

Rep: Reputation: Disabled
Call for testers - script to resolve dependencies for SBo packages


It is just included below. I tested for several packages - seems it works. I use lynx to dump web page - but there should be easier way. The script is called by dep-resolv.sh
Code:
$ ./dep-resolv.sh link-to-top-level-package
you are of course can name it as you wish. There are two parameters DEPS - name of file links are stored in, HASH - it should be complete gibberish as it is used to hash lines in the DEPS, default values are DEPS=deps and HASH=AnzUiWw

The command
Code:
$ DEPS=foo HASh=OlaIARE ./depres.sh http://www.slackbuilds.org/repositor...ipboardViewer/
generates the file ./foo in the same directory as the script
Code:
$ cat foo
http://www.slackbuilds.org/repository/14.2/desktop/ClipboardViewer/
http://www.slackbuilds.org/repository/14.2/libraries/libgnomeui/
http://www.slackbuilds.org/repository/14.2/libraries/libbonoboui/
http://www.slackbuilds.org/repository/14.2/libraries/libgnomecanvas/
http://www.slackbuilds.org/repository/14.2/libraries/libgnome/
http://www.slackbuilds.org/repository/14.2/system/gnome-vfs/
http://www.slackbuilds.org/repository/14.2/libraries/libbonobo/
http://www.slackbuilds.org/repository/14.2/misc/gnome-mime-data/
http://www.slackbuilds.org/repository/14.2/libraries/ORBit2/
The order of compilation is from bottom to top. Use tac to get in reverse order.

Here is the script itself
Code:
#!/bin/bash

#
# Create dependencies file for sbopkg packages
#
 
DEPS=${DEPS:=deps} ; touch $DEPS
DEPSHASH=${DEPS}hash ; touch $DEPSHASH
TMPF=tmpfile ; touch $TMPF
TMPF_OUT=tmpfile_out ; touch $TMPF
HASH_POSTFIX=${HASH:=AnzUiWw}  # It should complete gibberish 
LYNX_OPT="-nonumbers -listonly -dump"

# Remove repeated links 
function slip_and_add () { 
         cat $DEPS | grep -v "$ADDL" > $TMPF 
         echo "$ADDL" >> $TMPF 
         cat  $TMPF > $DEPS
 } 

# Pick deps for package pointed by package addres  
function get_deps () {
		lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT 
}

# Build hash for dependencies file 
function hash () { 
                  for i in $(seq 1 `cat $DEPS | wc -l` ) 
                  do 
                   echo ${i}$HASH_POSTFIX 
                  done |  paste - $DEPS > $DEPSHASH  
}

echo $1 > $DEPS  # Put the link of top package to be installed into dependencies file  
hash 
let PROC=1
# Terminate if there is no more links to be processsed
   until  [ $PROC -gt `cat $DEPS | wc -l` ]; do
      LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"
      get_deps                     # Get dependencies for processed package   
      while read addr; do  
               ADDL=$addr slip_and_add  
      done < $TMPF_OUT                 # Add them to dependencies file of the top package 
      hash                       # Rebuild hash for dependencies file 
      let PROC=$PROC+1             # Point to next link to be processed in dependencies file 
   done 
   
# Make clean   
rm $DEPSHASH   
rm $TMPF
rm $TMPF_OUT
Dependencies which are not present on SlackBuilds.org cannot be resolved (clearly) by this scripte.

Last edited by igadoter; 08-26-2018 at 09:14 AM.
 
Old 08-27-2018, 01:40 PM   #2
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Here is script which can made things little more interesting
Code:
#!/bin/bash 

 
cat $1 |sed 's/repository/slackbuilds/' | while read addr ; do 
     pdir=`echo $addr | cut -d'/' -f7` ;
     mkdir $pdir ; 	 
     cd $pdir ;  
     curl  `echo ${addr}`\{README,${pdir}.info,${pdir}.SlackBuild,doinst.sh,slack-desc\} -o \#1 ; 
     cd ..  ;  
done
Run the script in the first post, say html addresses are in 'foo' file. Then run
Code:
$ cat foo | dc_copy.sh -
Code:
 dc_copy.sh
it is just the name of the second script given by me. Notice command ends with '-' sign - usually it means standard input.

Now do something like that
Code:
$ ls -R
On my computer it is
Code:
$ ls -R
.:
ClipboardViewer  dc_copy.sh   dep-resol-v2.sh  depres.sh  foo     gnome-mime-data  iso.old    libbonoboui  libgnomecanvas  scratch
ORBit2           dc_copy.sh~  dep-resolv.sh    deps       foobar  gnome-vfs        libbonobo  libgnome     libgnomeui

./ClipboardViewer:
ClipboardViewer.SlackBuild  ClipboardViewer.info  README  doinst.sh  slack-desc

./ORBit2:
ORBit2.SlackBuild  ORBit2.info  README  doinst.sh  slack-desc

./gnome-mime-data:
README  doinst.sh  gnome-mime-data.SlackBuild  gnome-mime-data.info  slack-desc

./gnome-vfs:
README  doinst.sh  gnome-vfs.SlackBuild  gnome-vfs.info  slack-desc

./libbonobo:
README  doinst.sh  libbonobo.SlackBuild  libbonobo.info  slack-desc

./libbonoboui:
README  doinst.sh  libbonoboui.SlackBuild  libbonoboui.info  slack-desc

./libgnome:
README  doinst.sh  libgnome.SlackBuild  libgnome.info  slack-desc

./libgnomecanvas:
README  doinst.sh  libgnomecanvas.SlackBuild  libgnomecanvas.info  slack-desc

./libgnomeui:
README  doinst.sh  libgnomeui.SlackBuild  libgnomeui.info  slack-desc
 
Old 08-27-2018, 02:19 PM   #3
zsd
Member
 
Registered: Dec 2005
Location: Nova Scotia
Distribution: Slackware
Posts: 84

Rep: Reputation: 41
You might consider adding a trailing / to $1 if there isn't one there. E.g.,
Code:
# Put the link of top package to be installed into dependencies file
case "$1" in
  (*/)
    echo $1 > $DEPS  
    ;;
  (*)
    echo $1/ > $DEPS 
    ;;   
esac
Also, is your regexp the same as '(/.*){7}' ? In any case, you might consider putting '' around your regexp and losing the \s.

You might also consider the virtue of replacing
Code:
DEPSHASH=${DEPS}hash ; touch $DEPSHASH
TMPF=tmpfile ; touch $TMPF
TMPF_OUT=tmpfile_out ; touch $TMPF
(the second line is presumably supposed to end with $TMPF_OUT?)
with
Code:
DEPSHASH=`mktemp`
TMPF=`mktemp`
TMPF_OUT=`mktemp`
And maybe even adding
Code:
trap "rm -f $TMPF $TMPF_OUT $DEPSHASH" 0 1 2 3 15
(or similar) after the calls to mktemp, and then you can delete the three rm's at the end of the program.

Having said that, it worked for the couple of cases I tried (after I realized I needed the trailing / on the link).

Last edited by zsd; 08-27-2018 at 02:26 PM.
 
Old 08-28-2018, 03:42 AM   #4
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by zsd View Post
You might consider adding a trailing / to $1 if there isn't one there
Done. Say someone forget to add trailing / to the link.
Quote:
Also, is your regexp the same as '(/.*){7}' ? In any case, you might consider putting '' around your regexp and losing the \s.
yes. I tried '(/.*){7} but I was unable to make it works as I expected. The regexp is very primitive: it just counts number of / in link.

Thanks for feedback and comments. I just done everything as you suggested.

Below is rewritten script
Code:
#!/bin/bash

#
# Create dependencies file for sbopkg packages
#
 
DEPS=deps ; touch $DEPS

DEPSHASH=`mktemp`
TMPF=`mktemp`
TMPF_OUT=`mktemp`
HASH_POSTFIX=${HASH:=AnzUiWw}  # It should be complete gibberish 
LYNX_OPT="-nonumbers -listonly -dump"

# Remove duplicated links 
function slip_and_add () { 
         cat $DEPS | grep -v "$ADDL" > $TMPF 
         echo "$ADDL" >> $TMPF 
         cat  $TMPF > $DEPS
 } 

# Pick deps for package pointed by package addres  
function get_deps () {
		lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT 
}

# Build hash for dependencies file 
function hash () { 
                  for i in $(seq 1 `cat $DEPS | wc -l` ) 
                  do 
                   echo ${i}$HASH_POSTFIX 
                  done |  paste - $DEPS > $DEPSHASH  
}

echo $1 > $DEPS  # Put the link of top package to be installed into dependencies file  
hash 
let PROC=1
# Terminate if there is no more links to be processsed
   until  [ $PROC -gt `cat $DEPS | wc -l` ]; do
      LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"
      get_deps                     # Get dependencies for processed package   
      while read addr; do  
               ADDL=$addr slip_and_add  
      done < $TMPF_OUT                 # Add them to dependencies file of the top package 
      hash                       # Rebuild hash for dependencies file 
      let PROC=$PROC+1             # Point to next link to be processed in dependencies file 
   done 
# Make clean   
trap "rm -f $TMPF $TMPF_OUT $DEPSHASH" 0 1 2 3 15
cat $DEPS
there are changes in the script I forgot to mention: I added at the last line
Code:
cat $DEPS
to make it possible to run script in pipe, eg.
Code:
./depres.sh | sed 's/foo/bar/`
so to run it silently redirect to /dev/null
Code:
./depres.sh > /dev/null
 
Old 08-28-2018, 06:47 AM   #5
zsd
Member
 
Registered: Dec 2005
Location: Nova Scotia
Distribution: Slackware
Posts: 84

Rep: Reputation: 41
Quote:
Quote:
Originally Posted by zsd 
You might consider adding a trailing / to $1 if there isn't one there
Done. Say someone forget to add trailing / to the link.
It seems you didn't add the "add / to the end of $1 if it isn't there" change.

I think the change to output $DEPS at the end is a big improvement.
1) You might want use tac instead of cat, so that the dependencies will come out in the order one has to build them, and
2) since you output the contents of that file, you might just use DEPS=`mktemp` and add $DEPS to the list of files that the trap command rm's.

The trap should go right after the files are created with mktemp... if someone ^C's the script while it is working, before it gets to the trap command, the temp files won't be deleted, which is probably not what is desired. (But, having said that, it is your program.)

I have my version of your program working with grep -E '(/.*){7}' > $TMPF_OUT; while yours (clearly) works too, I am surprised you had issues with the .* version. Huh.

You might also put in this code (feel free to use it if you like) before using $1 for the first time:

Code:
if [ -z "$1" ]
then
    echo >&2 "$0: need exactly one argument, a link to a Slackbuild"
    echo >&2 "Example:  $0 http://www.slackbuilds.org/repository/14.2/graphics/shotwell/"
    exit 1
fi
My last (?) suggestion for you is to put a comment at the top saying what the program does and who wrote it. (Credit where credit is due!) And maybe even some license statement, saying it is in the public domain, or free for any use, or for people to send you 1 kg of gold when it is used, or ...
 
Old 08-28-2018, 08:39 AM   #6
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
@zsd thank you. I greatly appreciate your help. I am just trying to do some useful script. What I started is just hand-made - any improvement is welcomed. I need to read about trap command ;(

Edit: @zsd it seems that '(/*){7}' works due to the presence of apostrophes. I will have a look at this.

Last edited by igadoter; 08-28-2018 at 08:46 AM.
 
Old 08-29-2018, 08:55 AM   #7
zsd
Member
 
Registered: Dec 2005
Location: Nova Scotia
Distribution: Slackware
Posts: 84

Rep: Reputation: 41
Quote:
Originally Posted by igadoter View Post
@zsd thank you. I greatly appreciate your help. I am just trying to do some useful script. What I started is just hand-made - any improvement is welcomed. I need to read about trap command ;(

Edit: @zsd it seems that '(/*){7}' works due to the presence of apostrophes. I will have a look at this.
Glad to help out. Writing shell scripts can be tricky because there are so many ways to do any given thing. (And the shell is a much more powerful programming language than most people recognize.)
 
Old 08-29-2018, 07:01 PM   #8
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Below is script for testing dependency file. It just gathers information from all *.info files. So the first thing is to download the *.info file for every package in dependency file, I'm making directory and put inside *.info file. https://www.linuxquestions.org/quest...0/#post5896649 This is very basic test but it gives some assurance.
Code:
#!/bin/bash 

#
# Compare dependencies with the contents of *.info files 
#  

DEPS=$1   # file of dependencies
ROOT=$2   # root directory 
          # in $ROOT are located directories of package from $DEPS 

TMPF_NFO=`mktemp`
TMPF_DPS=`mktemp`
trap "rm -f $TMPF_NFO $TMPF_DPS"  0 1 2 3 15

find $ROOT -name \*.info -exec cat {} \; | grep -e PRGNAM -e REQUIRES |	 
                                           cut -d'"' -f2 | 
                                           tr ' ' '\n' | 
                                           sort -u | grep -v \^\$ > $TMPF_NFO

cat $DEPS | cut -d'/' -f7 | sort > $TMPF_DPS   

# Assertion $TMPF_NFO and $TMPF_DPS are identical 
diff $TMPF_NFO $TMPF_DPS
if [ $? -eq 0 ]; then
    echo test: $DEPS OK!
else
     echo test: $DEPS FAILURE!   
fi
you can also redirect the contents of files $TMPF_NFO and $TMPF_DPS to terminal.

Last edited by igadoter; 08-29-2018 at 07:05 PM.
 
Old 08-30-2018, 04:25 PM   #9
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
I extended the first basic script to be able to work with multiple packages. Just feed the script with file which contains links to packages. Links should have form as being dumped from depositories web pages. I run script on bunch of SBo packages for some X terminals: gnome-terminal, etc. The output should contain all dependencies to build all these terminals
Code:
 
http://www.slackbuilds.org/repository/14.2/system/Eterm/
http://www.slackbuilds.org/repository/14.2/system/aterm/
http://www.slackbuilds.org/repository/14.2/system/cool-retro-term/
http://www.slackbuilds.org/repository/14.2/system/fbterm/
http://www.slackbuilds.org/repository/14.2/system/gnome-terminal/
http://www.slackbuilds.org/repository/14.2/system/graphterm/
http://www.slackbuilds.org/repository/14.2/system/lilyterm/
http://www.slackbuilds.org/repository/14.2/system/lxterminal/
http://www.slackbuilds.org/repository/14.2/system/mlterm/
http://www.slackbuilds.org/repository/14.2/system/qterminal/
http://www.slackbuilds.org/repository/14.2/system/roxterm/
http://www.slackbuilds.org/repository/14.2/system/termenu/
http://www.slackbuilds.org/repository/14.2/system/terminator/
http://www.slackbuilds.org/repository/14.2/system/terminology/
http://www.slackbuilds.org/repository/14.2/system/termite/
http://www.slackbuilds.org/repository/14.2/system/termsaver/
http://www.slackbuilds.org/repository/14.2/system/tinyterm/
http://www.slackbuilds.org/repository/14.2/system/vinterm/
http://www.slackbuilds.org/repository/14.2/system/wterm/
I named script as dpsres.sh and put all links in trms, now issue command
Code:
$ ./dpsres.sh trms
it creates dependency file trms.dps and print the same file on output
Code:
http://www.slackbuilds.org/repository/14.2/system/Eterm/
http://www.slackbuilds.org/repository/14.2/libraries/imlib2/
http://www.slackbuilds.org/repository/14.2/libraries/libast/
http://www.slackbuilds.org/repository/14.2/system/aterm/
http://www.slackbuilds.org/repository/14.2/system/cool-retro-term/
http://www.slackbuilds.org/repository/14.2/libraries/qmltermwidget/
http://www.slackbuilds.org/repository/14.2/system/fbterm/
http://www.slackbuilds.org/repository/14.2/system/gnome-terminal/
http://www.slackbuilds.org/repository/14.2/libraries/libgnome/
http://www.slackbuilds.org/repository/14.2/development/rarian/
http://www.slackbuilds.org/repository/14.2/system/gnome-vfs/
http://www.slackbuilds.org/repository/14.2/libraries/libbonobo/
http://www.slackbuilds.org/repository/14.2/misc/gnome-mime-data/
http://www.slackbuilds.org/repository/14.2/libraries/ORBit2/
http://www.slackbuilds.org/repository/14.2/system/graphterm/
http://www.slackbuilds.org/repository/14.2/network/tornado/
http://www.slackbuilds.org/repository/14.2/libraries/futures/
http://www.slackbuilds.org/repository/14.2/python/python-certifi/
http://www.slackbuilds.org/repository/14.2/python/singledispatch/
http://www.slackbuilds.org/repository/14.2/python/backports-abc/
http://www.slackbuilds.org/repository/14.2/python/six/
http://www.slackbuilds.org/repository/14.2/system/lilyterm/
http://www.slackbuilds.org/repository/14.2/system/lxterminal/
http://www.slackbuilds.org/repository/14.2/system/mlterm/
http://www.slackbuilds.org/repository/14.2/system/qterminal/
http://www.slackbuilds.org/repository/14.2/libraries/qtermwidget/
http://www.slackbuilds.org/repository/14.2/libraries/lxqt-build-tools/
http://www.slackbuilds.org/repository/14.2/libraries/qt5/
http://www.slackbuilds.org/repository/14.2/libraries/libinput/
http://www.slackbuilds.org/repository/14.2/libraries/libxkbcommon/
http://www.slackbuilds.org/repository/14.2/graphics/graphviz/
http://www.slackbuilds.org/repository/14.2/python/pyudev/
http://www.slackbuilds.org/repository/14.2/python/python-evdev/
http://www.slackbuilds.org/repository/14.2/libraries/libwacom/
http://www.slackbuilds.org/repository/14.2/system/roxterm/
http://www.slackbuilds.org/repository/14.2/system/termenu/
http://www.slackbuilds.org/repository/14.2/system/terminator/
http://www.slackbuilds.org/repository/14.2/python/psutil/
http://www.slackbuilds.org/repository/14.2/system/terminology/
http://www.slackbuilds.org/repository/14.2/development/meson/
http://www.slackbuilds.org/repository/14.2/libraries/efl/
http://www.slackbuilds.org/repository/14.2/python/python3/
http://www.slackbuilds.org/repository/14.2/development/ninja/
http://www.slackbuilds.org/repository/14.2/system/termite/
http://www.slackbuilds.org/repository/14.2/libraries/vte3-ng/
http://www.slackbuilds.org/repository/14.2/development/vala/
http://www.slackbuilds.org/repository/14.2/libraries/pcre2/
http://www.slackbuilds.org/repository/14.2/system/termsaver/
http://www.slackbuilds.org/repository/14.2/system/tinyterm/
http://www.slackbuilds.org/repository/14.2/system/vinterm/
http://www.slackbuilds.org/repository/14.2/libraries/libconfig/
http://www.slackbuilds.org/repository/14.2/system/wterm/
dependency file is created so the order of build is as previous from below to top. As you see it is quite a lot number of dependencies to build these terminals. So the script also creates directory for every package (terminal, not every dependency !) with its own full dependency file and 'run-script' which allows only some packages to be built - say I don't want to build gnome-terminal - to avoid this just chmod -x 'run-script'. By default all run scripts are non-executable. So far these are just empty files.

Here is the script, it is short, clearly no warranties!
Code:
 
     1	#!/bin/bash 
     2	
     3	#
     4	# WMD - wpn of mss dstr
     5	# Create dependencies file for multiple SBo packages
     6	# Command: ./depres.sh file-with-sbo-links
     7	#  
     8	
     9	# TODO: *.run script 
    10	# it just do the thing one is doing by hand:
    11	# grab source tarball,  extract file with slackbuild script and run 
    12	# TODO: manage broken links - they may appear due to behavior of grep \(/.?+\)\{7\}
    13	# simple regexp '(/*){7}' was the cause of broken  links and script went in infinite loop
    14	    
    15	#
    16	# ASS - assertion for dep file: dependencies for name are below the name
    17	#
    18	
    19	W_DIR=`pwd`/                      # working directory 
    20	PACKAGES=$1
    21	
    22	if [ ! -f "$PACKAGES" ]; then 
    23	  exit 1
    24	fi
    25	
    26	if [  -s ${PACKAGES}.dps ]; then 
    27	    rm -f $PACKAGES.dps
    28	fi    
    29	    DEPS=${PACKAGES}.dps; touch $DEPS       # common deps file for all packages in $PACKAGES
    30	    
    31	    HASH_POSTFIX=${HP:=AnuiZWw}  
    32	    LYNX_OPT="-nonumbers -listonly -dump "
    33	    
    34	function get_deps () {
    35		     TMPF=`mktemp`
    36		     TMPF_OUT=`mktemp`
    37		     trap " rm -f $TMPF $TMPF_OUT " 0 1 2 3 15
    38		    # grab all links pointing to dependencies
    39			lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT 
    40			if [  -s $TMPF_OUT ]; then
    41			    while read  _addr; do
    42	               cat $DEPS_PKG | grep -v "$_addr" > $TMPF 
    43	               echo "$_addr" >> $TMPF
    44	               cat $TMPF > $DEPS_PKG
    45	            done < $TMPF_OUT
    46	            _addr=""
    47	        fi 
    48	        rm -f $TMPF $TMPF_OUT  # clear garbage    
    49	}
    50	
    51	# Build hash for dependencies file 
    52	function hash () { 
    53	                  for i in $(seq 1 `cat $DEPS_PKG | wc -l` ) 
    54	                  do 
    55	                   echo ${i}$HASH_POSTFIX 
    56	                  done |  paste - $DEPS_PKG > $DEPSHASH  
    57	}
    58	
    59	function merge () {
    60		  TMPF=`mktemp`
    61		  trap " rm -f $TMPF "  0 1 2 3 15
    62	      if [ -s $DEPS_PKG ]; then
    63	         while read _addr ; do
    64	                   cat $DEPS | grep -v "$_addr" > $TMPF
    65	                   echo "$_addr" >> $TMPF
    66	                   cat $TMPF > $DEPS
    67	         done < $DEPS_PKG
    68	         _addr=""                             
    69	      fi    
    70	      rm -f $TMPF        # clean garbage
    71	}
    72	
    73	#
    74	# Main: loop through all links in $PACKAGES
    75	#
    76	
    77	while read PKG; do                     # pick next package from $PACKAGES
    78	      DEPSHASH=`mktemp`
    79	      DEPS_PKG=`mktemp`                # $DEPS_PKG - deps file for package $PKG
    80	      trap "rm -f $DEPSHASH $DEPS_PKG" 0 1 2 3 15 
    81	      #  
    82	      # build  $DEPS_PKG  
    83	      #
    84	      echo "$PKG" > $DEPS_PKG       
    85	      let PROC=1
    86		  hash
    87		  # Terminate if there is no more links to be processsed
    88	      until  [ $PROC -gt `cat $DEPS_PKG| wc -l` ]; do
    89	          LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"  # get link by its hash  
    90	          # trap: regexp \(/.?+\)\{7} may return broken link 
    91	          # strict policy: exit  at the first error
    92	          # or don't care: go to process next package  
    93	          #if [ `echo "$LNK" | grep -e 'libraries/$' -e 'system/$'` ]; then # strict policy
    94	              #exit 1 
    95	          #fi  
    96	          get_deps                     # get dependencies for $LNK and store them in $DEPS_PKG  
    97	          hash                         # rebuild hash for $DEPS_PKG
    98	          # find next package to process
    99	          PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
   100	          let PROC="$PROC"+1             # point to next link to process in $DEPS_PKG
   101	      done
   102	      
   103	      # make package directory 
   104	      PKGNAM=`echo $PKG | cut -d'/' -f7`      # it should be the same as $PKGNAM in *.info 
   105	      mkdir "${W_DIR}$PKGNAM"
   106	      cd "${W_DIR}$PKGNAM"
   107	
   108	      # find link web page for package with *.info 
   109	      # grab all files and store in package directory 
   110	      PKG_DATA=`echo $PKG | sed 's/repository/slackbuilds/'` 
   111	      curl  "$PKG_DATA"\{README,${PKGNAM}.info,${PKGNAM}.SlackBuild,doinst.sh,slack-desc\} -o \#1 ; 
   112	      
   113	      cat $DEPS_PKG >  ${PKGNAM}.deps   # full deps file for package is also stored 
   114	      
   115	      RBTT="${PKGNAM}.run                # have choice which packages from $PACKAGES to build"
   116	      if [ -s "$RBTT" ]; then
   117	         rm -f "$RBTT"
   118	         touch "$RBTT"
   119	      else
   120	          touch "$RBTT"
   121	      fi
   122	          
   123	      cd ${W_DIR}                       # go back to worling directory
   124	      
   125	      # final loop step: merge $DEPS_PKG  with $DEPS - keeping ASS valid 
   126	      merge 
   127	       
   128	      rm -f $DEPS_PKG $DEPSHASH     # clean garbage                          
   129	done  <  $PACKAGES              
   130	                                
   131	cat $DEPS        
   132	
   133	# marks:
   134	# once there are all *.info ar in place test dependency file running test.sh script
   135	# 
   136	# chmod +x W_DIR/*.run to build all packages
   137	# build just follow dependency file 
   138	#  
   139	
   140	# if all this is correct one can build all SBo which depends only on SBo  in one run
   141	# prerequsities: full Slackware install
I hope at final stage it will be fully functional - so I don't have to any more do these by my own - step by step.

Last edited by igadoter; 09-03-2018 at 04:30 AM. Reason: Added line numbers to the script
 
Old 09-03-2018, 04:39 AM   #10
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
The WMD script as it is now works not quite correct. In the code posted here https://www.linuxquestions.org/quest...0/#post5898172 before the line 100 put this
Code:
PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
so now code reads
Code:
     98      hash                         # rebuild hash for $DEPS_PKG
     99      # find next package to process
    100      PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
    101      let PROC="$PROC"+1             # point to next link to process in $DEPS_PKG
simple
Code:
PROC="$PROC"+1
was not enough to find next link to process.
 
Old 09-04-2018, 07:23 PM   #11
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Hi, I have reached milestone and jut created something really useful, this is new version of script
Code:
     1	#!/bin/bash -x
     2	
     3	#
     4	# Create dependency file for multiple SBo packages
     5	#  
     6	    
     7	#
     8	# assertion for dependency file: dependencies for name are below the name
     9	# keep assertion to be true
    10	#
    11	
    12	PACKAGES=$1
    13	
    14	if [ ! -f "$PACKAGES" ]; then 
    15	  exit 1
    16	fi
    17	
    18	if [  -s ${PACKAGES}.deps ]; then 
    19	    rm -f $PACKAGES.deps
    20	fi    
    21	
    22	DEPS=${PACKAGES}.deps; touch $DEPS       # common deps file for all packages in $PACKAGES
    23	HASH_POSTFIX=${HP:=AnuiZWw}
    24	
    25	# -dump  dump web page
    26	# -listonly list only links 
    27	# -nonumbers links are without numbers   
    28	LYNX_OPT="-nonumbers -listonly -dump "
    29	    
    30	function get_deps () {
    31		     TMPF=`mktemp`
    32		     TMPF_OUT=`mktemp`
    33		     trap " rm -f $TMPF $TMPF_OUT " 0 1 2 3 15
    34		    # grab all links pointing to dependencies
    35			lynx $LYNX_OPT $LNK | grep repository | grep -E \(/.?+\)\{7\} > $TMPF_OUT 
    36			if [  -s $TMPF_OUT ]; then
    37			    while read  _addr; do
    38	               cat $DEPS_PKG | grep -v "$_addr" > $TMPF 
    39	               echo "$_addr" >> $TMPF
    40	               cat $TMPF > $DEPS_PKG
    41	            done < $TMPF_OUT
    42	            _addr=""
    43	        fi 
    44	        rm -f $TMPF $TMPF_OUT  # clear garbage    
    45	}
    46	
    47	# Build hash for dependencies file 
    48	function hash () { 
    49	                  for i in $(seq 1 `cat $DEPS_PKG | wc -l` ) 
    50	                  do 
    51	                   echo ${i}$HASH_POSTFIX 
    52	                  done |  paste - $DEPS_PKG > $DEPSHASH  
    53	}
    54	
    55	function merge () {
    56		  TMPF=`mktemp`
    57		  trap " rm -f $TMPF "  0 1 2 3 15
    58	      if [ -s $DEPS_PKG ]; then
    59	         while read _addr ; do
    60	                   cat $DEPS | grep -v "$_addr" > $TMPF
    61	                   echo "$_addr" >> $TMPF
    62	                   cat $TMPF > $DEPS
    63	         done < $DEPS_PKG
    64	         _addr=""                             
    65	      fi    
    66	      rm -f $TMPF        # clean garbage
    67	}
    68	
    69	#
    70	# main: perform  all links in $PACKAGES
    71	#
    72	while read PKG; do                     
    73	      DEPSHASH=`mktemp`
    74	      DEPS_PKG=`mktemp`                # $DEPS_PKG: dependency file for $PKG
    75	      trap "rm -f $DEPSHASH $DEPS_PKG" 0 1 2 3 15 
    76	      #  
    77	      # perform  $DEPS_PKG  
    78	      #
    79	      echo "$PKG" > $DEPS_PKG       
    80	      let PROC=1
    81		  hash
    82		  # Terminate if there is no more links to be processsed
    83	      until  [ $PROC -gt `cat $DEPS_PKG| wc -l` ]; do
    84	          LNK="$(cat $DEPSHASH | grep ${PROC}$HASH_POSTFIX | cut -f2)"  # get link by its hash  
    85	          # trap: regexp \(/.?+\)\{7} may return broken link 
    86	          # strict policy: exit  at the first error
    87	          # or don't care: go to process next package  
    88	          #if [ `echo "$LNK" | grep -e 'libraries/$' -e 'system/$'` ]; then # strict policy
    89	              #exit 1 
    90	          #fi  
    91	          get_deps                     # get dependencies for $LNK and store them in $DEPS_PKG  
    92	          hash                         # rebuild hash for $DEPS_PKG
    93	          # find next package to process
    94	          PROC=`cat -n $DEPS_PKG | grep $LNK | cut -f1`
    95	          let PROC="$PROC"+1             # point to next link to process in $DEPS_PKG
    96	      done
    97	      
    98	      # merge $DEPS_PKG  with $DEPS - keeping ASS valid 
    99	      merge 
   100	       
   101	      rm -f $DEPS_PKG $DEPSHASH     # clean garbage                          
   102	done  <  $PACKAGES              
   103	
   104	
   105	# main loop for creating directories and downloading package data
   106	while read PKG ; do
   107	      PKGNAME=`basename $PKG`
   108	      PKG_DATA=`echo $PKG | sed 's/repository/slackbuilds/; s/\/$/.tar.gz/'` 
   109	      curl  "$PKG_DATA" | tar -zxvf -
   110	      
   111	      TOP_DIR=`pwd`
   112	      cd "$PKGNAME"
   113	      # script to download sources and additonal data 
   114	      cat <<'EOF' > "${PKGNAME}-run.sh"         
   115	#!/bin/bash                 
   116	
   117	# run basic things 
   118	# download packages verify checksum
   119	
   120	. `ls *.info`               # read contents of *.info file
   121	
   122	if [  "`uname -m`" = "_x86_64" ]; then
   123	    ARCH="_x86_64"
   124	else
   125	    ARCH=""
   126	fi        
   127	
   128	# curl options
   129	# -L means follow redirections
   130	# -O save target with remote name
   131	CURL_OPT=" -L -O "              
   132	
   133	declare URLS=(${DOWNLOAD}$ARCH)   # for multiple downloads
   134	declare CHCKS=(${MD5SUM}$ARCH)
   135	
   136	# build checksums file
   137	echo -n > $PRGNAM.md5
   138	for i in ${!URLS[@]} ; do
   139	     echo ${CHCKS[$i]} ${URLS[$i]##*/} >> $PRGNAM.md5
   140	done
   141	
   142	#grab sources     
   143	for url in  ${URLS[@]} ; do
   144	   curl -L -O "$url"
   145	done   
   146	
   147	# verify checksums   
   148	md5sum -c $PRGNAM.md5
   149	if [ $? -ne 0 ] ; then
   150	     echo Checksum failed!
   151	     exit 1
   152	fi     
   153	EOF
   154	      cd "$TOP_DIR"                      
   155	done < $DEPS
   156	
   157	# flush to stdout                                
   158	cat $DEPS        
   159	
   160  # End of script
It creates dependency file with suffix .deps and sub-directories for all packages in dependency file. Also contents of dependency file is flushed to stdout. In all sub-directories there are *-run.sh scripts. They download sources and and any other data, verify checksums. They have to be run in the same subdirectory as other files - *info -slackbuilds, etc. Here example. I called the script sbo-res.sh. Run
Code:
 
$ ./sbo-res.sh sbo-links
It creates sbo-links.deps. Now make all *-run.sh scripts executable
Code:
$ find . -name \*-run.sh -exec chmod +x {} \;
then run these scripts
Code:
find . -name \*-run.sh  | while read line ; do DIR=`dirname $line` ; RUNS=`basename $line` ;  cd "$DIR" ;   ./"$RUNS"  ;  cd  .. ; done
at the end you obtain all targets. Builds goes along sbo-links.deps from bottom to the top. Enjoy!

TODO: there are the two things: 1) to make better test script 2) handle alternative: 64 versus 32 bit. Some packages don't compile on 32 bits systems and vice versa
 
Old 09-05-2018, 12:57 AM   #12
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,786

Rep: Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848Reputation: 4848
If I can give you some hints (not about the functionality, but quality and speed):
1. please try www.shellcheck.net to check your script and fix the errors/issues reported.
2. useless use of cat
3. instead of basename and dirname you can use ${VAR##*/} and ${VAR%/*}
4. in general avoid backticks or $( ) if possible, use shell built-ins instead
5. do not use awk|grep|cut|cat|sed|wc|whatever chains, usually a single awk (or perl/python/... ) oneliner (script) will do the job

6. use mktemp only once, and all the other tmp dirs should be located inside this temp dir - in this case you can clean up by removing that single dir.
7. there is no error handling.
 
Old 09-05-2018, 02:48 AM   #13
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
1. please try www.shellcheck.net to check your script and fix the errors/issues reported.
tested: looks good.
Quote:
4. in general avoid backticks or $( ) if possible, use shell built-ins instead
I think this just improves readability.
Quote:
5. do not use awk|grep|cut|cat|sed|wc|whatever chains, usually a single awk (or perl/python/... ) oneliner (script)
of course but this is bash script with minimum requirements. Essentially I am using 'mktemp` to mimic local variables.
Quote:
7. there is no error handling
It is one-run script. At any error just exit. So I think killing parent process should be enough. Plus kind of log.


But saying that thanks for your feedback.
 
Old 09-05-2018, 02:36 PM   #14
individual
Member
 
Registered: Jul 2018
Posts: 283
Blog Entries: 1

Rep: Reputation: 218Reputation: 218Reputation: 218
Hey, I like the idea of generating links to dependencies for a SlackBuild, but your script is difficult to follow because of bash's oddities and the use of global variables. In my opinion a traditional programming language would be a better choice for this. I'm not trying to hijack your thread, but I wrote an equivalent script (for your first version) in Perl. I'd love to hear your thoughts on it.
Code:
#!/usr/bin/perl
# Resolve dependencies for a SlackBuild.

use strict;
use warnings;
use HTTP::Tiny;

use constant BASE_URL => 'http://www.slackbuilds.org';

sub resolve_deps {
    my $url = shift || die;

    print "$url\n";

    my $html = HTTP::Tiny->new->get($url)->{content};
    resolve_deps(BASE_URL . $_) for ($html =~ m!(/repository/.*?)'!g);
}

resolve_deps(@ARGV);
EDIT: I forgot to add that this works with Perl 5.8.4 and up, so it is portable.

Last edited by individual; 09-05-2018 at 02:45 PM.
 
Old 09-07-2018, 10:37 AM   #15
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,557

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by individual View Post
Hey, I like the idea of generating links to dependencies for a SlackBuild, but your script is difficult to follow because of bash's oddities and the use of global variables.
This is my fault with these oddities. I am just rewriting the script - it will be looking even more odd - but more like any other programming language. I don't really feel to be someone who can you give advice. I was running for years Linux - I get to use to work with shell. Your script is calling recursively function: It would be good to keep trace of this and limit the number of recursive calls. You may also look at so-called tail-recursion - function with tail-recursion can be easily rewritten as common loop.
 
1 members found this post helpful.
  


Reply

Tags
dependence, resolve, sbo, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sboui: ncurses-based UI for SBo package managers (call for testers) montagdude Slackware 213 05-25-2020 06:50 AM
SBo optional dependencies miev Slackware 3 11-15-2016 09:40 AM
Script to get/set dependencies/required packages in a slackbuild franzen Slackware 22 02-01-2016 03:52 AM
Updating w3af to version 1.6.49 in SBo, need testers mralk3 Slackware 3 12-12-2015 05:54 PM
Recurring failure to resolve openoffice dependencies, possibly due to held packages Yangbo Du Ubuntu 2 02-16-2011 09:49 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 10:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration