LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 09-07-2018, 12:14 PM   #16
individual
Member
 
Registered: Jul 2018
Posts: 292
Blog Entries: 1

Rep: Reputation: 223Reputation: 223Reputation: 223

Quote:
Originally Posted by igadoter View Post
This is my fault with these oddities. I am just rewriting the script - it will be looking even more odd - but more like any other programming language. I don't really feel to be someone who can you give advice. I was running for years Linux - I get to use to work with shell. Your script is calling recursively function: It would be good to keep trace of this and limit the number of recursive calls. You may also look at so-called tail-recursion - function with tail-recursion can be easily rewritten as common loop.
Thanks for the feedback. The reason I wrote a version in Perl is because I saw you mention in your blog post that you wanted to port it to another programming language at some point. I agree on the script being iterative rather than recursive, so I wrote an iterative version.
Code:
#!/usr/bin/perl
# Resolve dependencies for a SlackBuild.

use strict;
use warnings;
use HTTP::Tiny;

use constant BASE_URL => 'http://www.slackbuilds.org';

sub resolve_deps {
    my $url = shift || die;
    my %seen_urls;
    my @url_queue = ($url);

    while (@url_queue) {
        $url = shift @url_queue;
        next if $seen_urls{$url}++;

        print "$url\n";

        my $html = HTTP::Tiny->new->get($url)->{content};
        push @url_queue, BASE_URL . $_ for ( $html =~ m!(/repository/.*?)'!g );
    }
}

resolve_deps(@ARGV);
 
Old 09-30-2018, 04:57 AM   #17
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 1,592

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
This is new version. I followed advice of @pan64. I must say the first of all I regret to use www.shellcheck.net.
Trying to fix script along the line proposed by shell check I just broke my script! But as @pan64 suggested script is now almost all-bash - the only issue I was unable to resolve was to remove empty lines from file - with bash only. So I used sed. But the price to be "bash only" is evident: now the script is just difficult to understand. There are much more "bash magic" than in the previous version. I have built with help of this script avfs package - rather easily. Missing part now is simple script to build packages - I am working on it. The trouble is that such script has to be run by root - due to SlackBuild limitations. It is final step. I think there are two on the edge strategies: 1) do everything you can as common user 2) don't care run all as root. I prefer the first. But it is possible to change script to allow to be executed using different strategies. TODO: there is no test for architecture compatibility.
Sorry for such long delay: I was in the place without(!) internet.
Code:
     1	#!/bin/bash -x
     2	
     3	# sbo_res 
     4	#-----------------------------------------------------------------------
     5	# Create dependency file for multiple SBo packages
     6	#    sbo_res FILE
     7	# FILE should contain repository URLS of required packages eg
     8	#    http://www.slackbuilds.org/repositiory/14.2/desktop/alltray/ 
     9	# There can be numbers of such URLS 
    10	# The output FILE.deps is file containing all required  dependencies for 
    11	# every package in FILE - the build order is from bottom to top.
    12	# There is no redundancy - each dependency appears only once - even 
    13	# if shared by several packages. 
    14	# The work is still in progress: use the script solely  by your own risk.
    15	# ----------------------------------------------------------------------
    16	# Author: Piotr Dacko < dackopiotr7 at gmail.com > 
    17	#                     < forum: www.linuxquestions.org forum member: igadoter>
    18	# ----------------------------------------------------------------------
    19	
    20	
    21	# Rule of thumb
    22	# assertion for dependency file: dependencies for name are below the name
    23	
    24	function on_event () {
    25		    # on_event $1 ...  event handler
    26		    # event $1 in context $2, $3, ....
    27		        case "$1" in
    28		            ("ON_ROOT")
    29		              if [ "$2" -eq 0 ]; then 
    30	                      echo "error: root user detected"
    31	                      echo "do not run this script with root privileges...exiting"
    32	                      return 1
    33	                  fi      
    34		              ;;
    35		            ("ON_NOARG")
    36		              if [ "$2" -eq 0 ]; then 
    37	                      echo "error: missing arument"
    38	                      echo "usage: ${0##*/} FILE...exiting"
    39	                      return 1
    40	                  fi
    41		              ;;
    42		            ("ON_NOFILE")
    43		              if [ ! -s "$2" ]; then
    44	                      echo "error: no file $2 or file empty...exiting" 
    45	                      return 1
    46	                  fi
    47	                  ;;
    48	                ("ON_NONETWORK")
    49	                      echo "error: no network connection...exiting"    
    50	                      return 1
    51	                  ;;
    52	                ("ON_NORUNSCRIPT")
    53	                  if [ ! -s "$2" ]; then 
    54	                      echo "error: fail to create run script...exiting "
    55	                      return 1
    56	                  fi
    57	                  ;;  
    58	                ("ON_NOTMP")
    59	                      mktemp -u -p "$TD" -q > /dev/null  ||  \
    60	                      { echo "error: can't create temp file...exiting" ;
    61	                        return  1 ; } 
    62	                   ;;     
    63	                ("ON_FAILURE")
    64	                      echo "error: script failure...exiting"
    65	                      return 1
    66	                  ;;   
    67	                *)
    68	                   return 0 ;;         
    69	             esac
    70	         
    71	}
    72	
    73	
    74	function bcat () {
    75		    # bcat - bash cat, copies input to output per line
    76		    while read ; do echo "$REPLY" ; done < "$1" > "$2"
    77	}
    78	
    79	
    80	function merge () {
    81		    # merge $1 $2
    82		    # glue two files into dependency file keeping assertion
    83	  	    # $2 already should satisfy the assertion
    84	
    85		    local TMPF=`mktemp -u -p "$TD"`
    86		    trap " rm -f $TMPF "  EXIT SIGHUP SIGINT SIGQUIT SIGTERM
    87	
    88		    if [ -s "$1" ]; then
    89		        while read addr; do
    90			        # replace line matching $addr by empty line
    91			        while read ; do 
    92			           echo ${REPLY#$addr} ;             
    93			        done < "$2" > "$TMPF"
    94			        # add $addr at the end 
    95	                echo "$addr" >> "$TMPF"
    96	                # bash 'cat'
    97	                bcat "$TMPF" "$2"
    98	            done < "$1"
    99	            # remove empty lines
   100	            sed  '/^$/d' "$2"   > "$TMPF"     
   101	            bcat "$TMPF" "$2"
   102	        fi      
   103	        rm -f "$TMPF"        # clean garbage
   104	}    
   105	
   106	
   107	function hash () { 
   108	        # hash $1 $2 - rehash file $1 with  hash file  $2
   109	        # clear old hash
   110	        echo -n "" >  "$2"
   111	        local C=0            # numeric 
   112	        while read ; do
   113	             echo "$((++C))${HASH_POSTFIX} $REPLY" 
   114	        done < "$1" > "$2"    
   115	}
   116	
   117	
   118	# preamble: 
   119	
   120	# default settings
   121	VER=${VER:=14.2}             # default version is recent stable version 
   122	SB="http://www.slackbuilds.org/"
   123	SB_REP="${SB}repository/${VER}/"
   124	SB_BUILD="${SB}slackbuilds/${VER}/"
   125	
   126	# lynx options
   127	# -dump dump web page
   128	# -listonly list only links 
   129	# -nonumbers links are without indexes   
   130	LYNX_OPT="-nonumbers -listonly -dump"	
   131	
   132	# auxiliary settings
   133	LOG_FILE="sbo_res.log"
   134	HASH_POSTFIX="AnuiZWw"
   135	TD='/tmp/sbo_res'  
   136	W_DIR=`pwd`  
   137	if [ -d "$TD" ]; then 
   138	   cd "$TD" || exit 1
   139	   rm -rf ./*
   140	   cd ..
   141	   rmdir "$TD"
   142	fi   
   143	mkdir -m 755 "$TD" || exit 1
   144	
   145	cd "$W_DIR"
   146	# basic startup tests
   147	on_event 'ON_ROOT'  `id -u` || exit 1
   148	on_event 'ON_NOARG' "$#" || exit 1
   149	on_event 'ON_NOFILE' "$1" || exit 1 
   150	on_event 'ON_NOTMP' ||  exit 1 
   151	
   152	
   153	# preprocess argument
   154	PACKAGES="$1"
   155	if [  -s "${PACKAGES}.deps" ]; then 
   156	    rm -f "$PACKAGES.deps"
   157	fi    
   158	DEPS="${PACKAGES}.deps" 
   159	
   160	# common deps file for all $PACKAGES
   161	touch "$DEPS"       
   162	
   163	
   164	#
   165	# main loop to perform  all links in $PACKAGES
   166	#
   167	while read PKG; do                     
   168	      DEPSHASH=`mktemp -u -p "$TD"` 
   169	      # dependency file for $PKG              
   170	      PKG_DEPS=`mktemp -u -p "$TD"`
   171	      URLS=`mktemp -u -p "$TD"`
   172	      TOP_DEPS=`mktemp -u -p "$TD"`
   173	      trap "rm -f $DEPSHASH $PKG_DEPS $URLS $TOP_DEPS" EXIT  SIGHUP   SIGINT  SIGQUIT  SIGTERM  
   174	        
   175	      # build dependency file $PKG_DEPS for package $PKG 
   176	      echo "$PKG" > "$PKG_DEPS"
   177	      INDX=1          # numeric 
   178		  hash  "$PKG_DEPS" "$DEPSHASH"         
   179		  # loop to perform $PKG
   180		  # Terminate if there is no more urls' to  process
   181	      until  [ "$INDX" -gt $(wc -l < $PKG_DEPS) ]; do
   182	          
   183	          # get link by its hash  
   184	          while read  HASH PKG_LINK ; do 
   185	              if [ "$HASH" = "${INDX}$HASH_POSTFIX" ]; then  
   186	                 break
   187	              fi
   188	          done < "$DEPSHASH"         
   189	          
   190	          # get top dependencies $TOP_DEPS 
   191	          # for package with repository url $PKG_LINK
   192	          lynx $LYNX_OPT "$PKG_LINK"  > "$URLS"  || \
   193	                  { on_event 'ON_NONETWORK' ; exit 1 ; } 
   194			  grep -E "$SB_REP*/.+/.+" "$URLS" > "$TOP_DEPS"
   195			  
   196			  # and store them in $PKG_DEPS  
   197			  merge "$TOP_DEPS" "$PKG_DEPS"
   198	          
   199			  
   200	          # rebuild hash for $PKG_DEPS          
   201	          hash  "$PKG_DEPS" "$DEPSHASH"                       
   202	
   203	          # find next package to process
   204	          NP=0             # numeric
   205	          while read ; do
   206	                 NP=$((++NP))          
   207	                 if [ "$REPLY" = "$PKG_LINK" ]; then
   208	                     break
   209	                 fi
   210	          done < "$PKG_DEPS"         
   211	          INDX=$((++NP))             
   212	      done
   213	      
   214	      # merge $PKG_DEPS  with $DEPS  keeping assertion valid 
   215	      merge "$PKG_DEPS" "$DEPS"
   216	      
   217	    #	  rm -f "$PKG_DEPS $DEPSHASH $URLS $TOP_DEPS"       # clean garbage    
   218	done < "$PACKAGES"                      
   219	# End main loop 
   220	
   221	#
   222	# main loop to create  build environement 
   223	#
   224	while read PKG ; do
   225	
   226	      # split $PKG url into fields separated by slash
   227	      IFS="/"
   228	      read DEPOSITORY PKGNAME <<EOF
   229	${PKG#$SB_REP}
   230	EOF
   231	      IFS=""
   232	      # get all data for $PKG and extract to $PKGNAME directory 
   233	      # many reasons curl | tar may fail
   234	      curl  "${SB_BUILD}${DEPOSITORY}/${PKGNAME}.tar.gz"  | tar -zxvf -  \
   235	            ||  { on_event 'ON_FAILURE' ; exit 1 ; }        
   236	      
   237	      
   238	      # setup script to download sources and additonal data 
   239	      TOP_DIR=`pwd`                        
   240	      cd "$PKGNAME"  || { on_event 'ON_FAILURE' ; exit 1 ; } 
   241	      
   242	      # run-script for source download we don't want them executable
   243	      RSCPT="${PKGNAME}-run.sh"         
   244	      cat <<'EOF' > "$RSCPT"  ||  { on_event 'ON_NORUNSCRIPT' "$RSCPT" ; exit 1 ; }            
   245	#!/bin/bash                 
   246	
   247	# run basic things 
   248	# download packages verify checksum
   249	
   250	. `ls *.info`               # read contents of *.info file
   251	
   252	if [  "`uname -m`" = "_x86_64" ]; then
   253	    ARCH="_x86_64"
   254	else
   255	    ARCH=""
   256	fi        
   257	
   258	# curl options
   259	# -L  follow redirections
   260	# -O  target with remote name
   261	CURL_OPT=" -L -O "              
   262	
   263	declare URLS=(${DOWNLOAD}$ARCH)   # for multiple downloads
   264	declare CHCKS=(${MD5SUM}$ARCH)
   265	
   266	# build checksums file
   267	echo -n > $PRGNAM.md5
   268	for i in ${!URLS[@]} ; do
   269	     echo ${CHCKS[$i]} ${URLS[$i]##*/} >> $PRGNAM.md5
   270	done
   271	
   272	# grab sources     
   273	for url in  ${URLS[@]} ; do
   274	   curl $CURL_OPT "$url"
   275	done   
   276	
   277	# verify checksums   
   278	md5sum -c $PRGNAM.md5
   279	if [ $? -ne 0 ] ; then
   280	     echo error: checksum failed...exiting
   281	     exit 1
   282	fi     
   283	EOF
   284	      cd "$TOP_DIR"  || { on_event 'ON_FAILURE' ; exit 1 ; }                     
   285	done < "$DEPS"
   286	
   287	# flush to stdout                                
   288	cat "$DEPS"
   289	# remove all tmp -files
   290	rm "$TD"/tmp*
   291	#EOF
Please find little time and test this. I hope it will work. The aim is to create unattended non-root build environment. It is not as difficult as it may seems. At final step one should obtain all binary packages. Without messing with system.
 
  


Reply

Tags
dependence, resolve, sbo, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sboui: ncurses-based UI for SBo package managers (call for testers) montagdude Slackware 213 05-25-2020 06:50 AM
SBo optional dependencies miev Slackware 3 11-15-2016 09:40 AM
Script to get/set dependencies/required packages in a slackbuild franzen Slackware 22 02-01-2016 03:52 AM
Updating w3af to version 1.6.49 in SBo, need testers mralk3 Slackware 3 12-12-2015 05:54 PM
Recurring failure to resolve openoffice dependencies, possibly due to held packages Yangbo Du Ubuntu 2 02-16-2011 09:49 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 02:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration