Call for testers - script to resolve dependencies for SBo packages

individual · 09-07-2018, 12:14 PM

Quote:

Originally Posted by igadoter

This is my fault with these oddities. I am just rewriting the script - it will be looking even more odd - but more like any other programming language. I don't really feel to be someone who can you give advice. I was running for years Linux - I get to use to work with shell. Your script is calling recursively function: It would be good to keep trace of this and limit the number of recursive calls. You may also look at so-called tail-recursion - function with tail-recursion can be easily rewritten as common loop.

Thanks for the feedback. The reason I wrote a version in Perl is because I saw you mention in your blog post that you wanted to port it to another programming language at some point. I agree on the script being iterative rather than recursive, so I wrote an iterative version.

Code:

#!/usr/bin/perl
# Resolve dependencies for a SlackBuild.

use strict;
use warnings;
use HTTP::Tiny;

use constant BASE_URL => 'http://www.slackbuilds.org';

sub resolve_deps {
    my $url = shift || die;
    my %seen_urls;
    my @url_queue = ($url);

    while (@url_queue) {
        $url = shift @url_queue;
        next if $seen_urls{$url}++;

        print "$url\n";

        my $html = HTTP::Tiny->new->get($url)->{content};
        push @url_queue, BASE_URL . $_ for ( $html =~ m!(/repository/.*?)'!g );
    }
}

resolve_deps(@ARGV);

igadoter · 09-30-2018, 04:57 AM

This is new version. I followed advice of @pan64. I must say the first of all I regret to use www.shellcheck.net.
Trying to fix script along the line proposed by shell check I just broke my script! But as @pan64 suggested script is now almost all-bash - the only issue I was unable to resolve was to remove empty lines from file - with bash only. So I used sed. But the price to be "bash only" is evident: now the script is just difficult to understand. There are much more "bash magic" than in the previous version. I have built with help of this script avfs package - rather easily. Missing part now is simple script to build packages - I am working on it. The trouble is that such script has to be run by root - due to SlackBuild limitations. It is final step. I think there are two on the edge strategies: 1) do everything you can as common user 2) don't care run all as root. I prefer the first. But it is possible to change script to allow to be executed using different strategies. TODO: there is no test for architecture compatibility.
Sorry for such long delay: I was in the place without(!) internet.

Code:

     1	#!/bin/bash -x
     2	
     3	# sbo_res 
     4	#-----------------------------------------------------------------------
     5	# Create dependency file for multiple SBo packages
     6	#    sbo_res FILE
     7	# FILE should contain repository URLS of required packages eg
     8	#    http://www.slackbuilds.org/repositiory/14.2/desktop/alltray/ 
     9	# There can be numbers of such URLS 
    10	# The output FILE.deps is file containing all required  dependencies for 
    11	# every package in FILE - the build order is from bottom to top.
    12	# There is no redundancy - each dependency appears only once - even 
    13	# if shared by several packages. 
    14	# The work is still in progress: use the script solely  by your own risk.
    15	# ----------------------------------------------------------------------
    16	# Author: Piotr Dacko < dackopiotr7 at gmail.com > 
    17	#                     < forum: www.linuxquestions.org forum member: igadoter>
    18	# ----------------------------------------------------------------------
    19	
    20	
    21	# Rule of thumb
    22	# assertion for dependency file: dependencies for name are below the name
    23	
    24	function on_event () {
    25		    # on_event $1 ...  event handler
    26		    # event $1 in context $2, $3, ....
    27		        case "$1" in
    28		            ("ON_ROOT")
    29		              if [ "$2" -eq 0 ]; then 
    30	                      echo "error: root user detected"
    31	                      echo "do not run this script with root privileges...exiting"
    32	                      return 1
    33	                  fi      
    34		              ;;
    35		            ("ON_NOARG")
    36		              if [ "$2" -eq 0 ]; then 
    37	                      echo "error: missing arument"
    38	                      echo "usage: ${0##*/} FILE...exiting"
    39	                      return 1
    40	                  fi
    41		              ;;
    42		            ("ON_NOFILE")
    43		              if [ ! -s "$2" ]; then
    44	                      echo "error: no file $2 or file empty...exiting" 
    45	                      return 1
    46	                  fi
    47	                  ;;
    48	                ("ON_NONETWORK")
    49	                      echo "error: no network connection...exiting"    
    50	                      return 1
    51	                  ;;
    52	                ("ON_NORUNSCRIPT")
    53	                  if [ ! -s "$2" ]; then 
    54	                      echo "error: fail to create run script...exiting "
    55	                      return 1
    56	                  fi
    57	                  ;;  
    58	                ("ON_NOTMP")
    59	                      mktemp -u -p "$TD" -q > /dev/null  ||  \
    60	                      { echo "error: can't create temp file...exiting" ;
    61	                        return  1 ; } 
    62	                   ;;     
    63	                ("ON_FAILURE")
    64	                      echo "error: script failure...exiting"
    65	                      return 1
    66	                  ;;   
    67	                *)
    68	                   return 0 ;;         
    69	             esac
    70	         
    71	}
    72	
    73	
    74	function bcat () {
    75		    # bcat - bash cat, copies input to output per line
    76		    while read ; do echo "$REPLY" ; done < "$1" > "$2"
    77	}
    78	
    79	
    80	function merge () {
    81		    # merge $1 $2
    82		    # glue two files into dependency file keeping assertion
    83	  	    # $2 already should satisfy the assertion
    84	
    85		    local TMPF=`mktemp -u -p "$TD"`
    86		    trap " rm -f $TMPF "  EXIT SIGHUP SIGINT SIGQUIT SIGTERM
    87	
    88		    if [ -s "$1" ]; then
    89		        while read addr; do
    90			        # replace line matching $addr by empty line
    91			        while read ; do 
    92			           echo ${REPLY#$addr} ;             
    93			        done < "$2" > "$TMPF"
    94			        # add $addr at the end 
    95	                echo "$addr" >> "$TMPF"
    96	                # bash 'cat'
    97	                bcat "$TMPF" "$2"
    98	            done < "$1"
    99	            # remove empty lines
   100	            sed  '/^$/d' "$2"   > "$TMPF"     
   101	            bcat "$TMPF" "$2"
   102	        fi      
   103	        rm -f "$TMPF"        # clean garbage
   104	}    
   105	
   106	
   107	function hash () { 
   108	        # hash $1 $2 - rehash file $1 with  hash file  $2
   109	        # clear old hash
   110	        echo -n "" >  "$2"
   111	        local C=0            # numeric 
   112	        while read ; do
   113	             echo "$((++C))${HASH_POSTFIX} $REPLY" 
   114	        done < "$1" > "$2"    
   115	}
   116	
   117	
   118	# preamble: 
   119	
   120	# default settings
   121	VER=${VER:=14.2}             # default version is recent stable version 
   122	SB="http://www.slackbuilds.org/"
   123	SB_REP="${SB}repository/${VER}/"
   124	SB_BUILD="${SB}slackbuilds/${VER}/"
   125	
   126	# lynx options
   127	# -dump dump web page
   128	# -listonly list only links 
   129	# -nonumbers links are without indexes   
   130	LYNX_OPT="-nonumbers -listonly -dump"	
   131	
   132	# auxiliary settings
   133	LOG_FILE="sbo_res.log"
   134	HASH_POSTFIX="AnuiZWw"
   135	TD='/tmp/sbo_res'  
   136	W_DIR=`pwd`  
   137	if [ -d "$TD" ]; then 
   138	   cd "$TD" || exit 1
   139	   rm -rf ./*
   140	   cd ..
   141	   rmdir "$TD"
   142	fi   
   143	mkdir -m 755 "$TD" || exit 1
   144	
   145	cd "$W_DIR"
   146	# basic startup tests
   147	on_event 'ON_ROOT'  `id -u` || exit 1
   148	on_event 'ON_NOARG' "$#" || exit 1
   149	on_event 'ON_NOFILE' "$1" || exit 1 
   150	on_event 'ON_NOTMP' ||  exit 1 
   151	
   152	
   153	# preprocess argument
   154	PACKAGES="$1"
   155	if [  -s "${PACKAGES}.deps" ]; then 
   156	    rm -f "$PACKAGES.deps"
   157	fi    
   158	DEPS="${PACKAGES}.deps" 
   159	
   160	# common deps file for all $PACKAGES
   161	touch "$DEPS"       
   162	
   163	
   164	#
   165	# main loop to perform  all links in $PACKAGES
   166	#
   167	while read PKG; do                     
   168	      DEPSHASH=`mktemp -u -p "$TD"` 
   169	      # dependency file for $PKG              
   170	      PKG_DEPS=`mktemp -u -p "$TD"`
   171	      URLS=`mktemp -u -p "$TD"`
   172	      TOP_DEPS=`mktemp -u -p "$TD"`
   173	      trap "rm -f $DEPSHASH $PKG_DEPS $URLS $TOP_DEPS" EXIT  SIGHUP   SIGINT  SIGQUIT  SIGTERM  
   174	        
   175	      # build dependency file $PKG_DEPS for package $PKG 
   176	      echo "$PKG" > "$PKG_DEPS"
   177	      INDX=1          # numeric 
   178		  hash  "$PKG_DEPS" "$DEPSHASH"         
   179		  # loop to perform $PKG
   180		  # Terminate if there is no more urls' to  process
   181	      until  [ "$INDX" -gt $(wc -l < $PKG_DEPS) ]; do
   182	          
   183	          # get link by its hash  
   184	          while read  HASH PKG_LINK ; do 
   185	              if [ "$HASH" = "${INDX}$HASH_POSTFIX" ]; then  
   186	                 break
   187	              fi
   188	          done < "$DEPSHASH"         
   189	          
   190	          # get top dependencies $TOP_DEPS 
   191	          # for package with repository url $PKG_LINK
   192	          lynx $LYNX_OPT "$PKG_LINK"  > "$URLS"  || \
   193	                  { on_event 'ON_NONETWORK' ; exit 1 ; } 
   194			  grep -E "$SB_REP*/.+/.+" "$URLS" > "$TOP_DEPS"
   195			  
   196			  # and store them in $PKG_DEPS  
   197			  merge "$TOP_DEPS" "$PKG_DEPS"
   198	          
   199			  
   200	          # rebuild hash for $PKG_DEPS          
   201	          hash  "$PKG_DEPS" "$DEPSHASH"                       
   202	
   203	          # find next package to process
   204	          NP=0             # numeric
   205	          while read ; do
   206	                 NP=$((++NP))          
   207	                 if [ "$REPLY" = "$PKG_LINK" ]; then
   208	                     break
   209	                 fi
   210	          done < "$PKG_DEPS"         
   211	          INDX=$((++NP))             
   212	      done
   213	      
   214	      # merge $PKG_DEPS  with $DEPS  keeping assertion valid 
   215	      merge "$PKG_DEPS" "$DEPS"
   216	      
   217	    #	  rm -f "$PKG_DEPS $DEPSHASH $URLS $TOP_DEPS"       # clean garbage    
   218	done < "$PACKAGES"                      
   219	# End main loop 
   220	
   221	#
   222	# main loop to create  build environement 
   223	#
   224	while read PKG ; do
   225	
   226	      # split $PKG url into fields separated by slash
   227	      IFS="/"
   228	      read DEPOSITORY PKGNAME <<EOF
   229	${PKG#$SB_REP}
   230	EOF
   231	      IFS=""
   232	      # get all data for $PKG and extract to $PKGNAME directory 
   233	      # many reasons curl | tar may fail
   234	      curl  "${SB_BUILD}${DEPOSITORY}/${PKGNAME}.tar.gz"  | tar -zxvf -  \
   235	            ||  { on_event 'ON_FAILURE' ; exit 1 ; }        
   236	      
   237	      
   238	      # setup script to download sources and additonal data 
   239	      TOP_DIR=`pwd`                        
   240	      cd "$PKGNAME"  || { on_event 'ON_FAILURE' ; exit 1 ; } 
   241	      
   242	      # run-script for source download we don't want them executable
   243	      RSCPT="${PKGNAME}-run.sh"         
   244	      cat <<'EOF' > "$RSCPT"  ||  { on_event 'ON_NORUNSCRIPT' "$RSCPT" ; exit 1 ; }            
   245	#!/bin/bash                 
   246	
   247	# run basic things 
   248	# download packages verify checksum
   249	
   250	. `ls *.info`               # read contents of *.info file
   251	
   252	if [  "`uname -m`" = "_x86_64" ]; then
   253	    ARCH="_x86_64"
   254	else
   255	    ARCH=""
   256	fi        
   257	
   258	# curl options
   259	# -L  follow redirections
   260	# -O  target with remote name
   261	CURL_OPT=" -L -O "              
   262	
   263	declare URLS=(${DOWNLOAD}$ARCH)   # for multiple downloads
   264	declare CHCKS=(${MD5SUM}$ARCH)
   265	
   266	# build checksums file
   267	echo -n > $PRGNAM.md5
   268	for i in ${!URLS[@]} ; do
   269	     echo ${CHCKS[$i]} ${URLS[$i]##*/} >> $PRGNAM.md5
   270	done
   271	
   272	# grab sources     
   273	for url in  ${URLS[@]} ; do
   274	   curl $CURL_OPT "$url"
   275	done   
   276	
   277	# verify checksums   
   278	md5sum -c $PRGNAM.md5
   279	if [ $? -ne 0 ] ; then
   280	     echo error: checksum failed...exiting
   281	     exit 1
   282	fi     
   283	EOF
   284	      cd "$TOP_DIR"  || { on_event 'ON_FAILURE' ; exit 1 ; }                     
   285	done < "$DEPS"
   286	
   287	# flush to stdout                                
   288	cat "$DEPS"
   289	# remove all tmp -files
   290	rm "$TD"/tmp*
   291	#EOF

Please find little time and test this. I hope it will work. The aim is to create unattended non-root build environment. It is not as difficult as it may seems. At final step one should obtain all binary packages. Without messing with system.