LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (https://www.linuxquestions.org/questions/slackware-14/)
-   -   slackpkg vs. third-party package repository (https://www.linuxquestions.org/questions/slackware-14/slackpkg-vs-third-party-package-repository-4175427364/)

TobiSGD 04-25-2013 09:44 AM

My local mirror is served via Apache from my home-server to my machines, so the missing support for file://-URLs was not the problem.
I just forgot to make slackpkgplus.sh executable, now it is working as it should. :doh:
Thanks for your help and the great work.

TobiSGD 04-25-2013 09:51 AM

Shouldn't have been so fast, it is not working as it should, I am unable to install any package from the third party repositories:
Code:

root ~ ☺ # slackpkg info vlc

Added external repository alienbob :
                URL=http://192.168.178.201/alien/x86_64/
                KEY=Eric Hameleers <alien@slackware.com>

Added external repository restricted :
                URL=http://192.168.178.201/restricted/x86_64/
                KEY=Eric Hameleers <alien@slackware.com>

PACKAGE NAME:  vlc-2.0.6-x86_64-2alien.txz
PACKAGE MIRROR:  http://www.slackware.com/~alien/slackbuilds/
PACKAGE LOCATION:  ./vlc
PACKAGE SIZE (compressed):  23288 K
PACKAGE SIZE (uncompressed):  91750 K
PACKAGE DESCRIPTION:
vlc: vlc (multimedia player for various audio and video formats)
vlc:
vlc: VLC media player is a highly portable multimedia player for various
vlc: audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, mp3, ogg, ...)
vlc: as well as DVDs, VCDs, and various streaming protocols.
vlc: It can also be used as a server to stream in unicast or multicast in
vlc: IPv4 or IPv6 on a high-bandwidth network.
vlc:
vlc:
vlc: vlc home: http://www.videolan.org/vlc/
vlc:

PACKAGE NAME:  vlc-2.0.6-x86_64-2alien.txz
PACKAGE MIRROR:  http://taper.alienbase.nl/mirrors/people/alien/restricted_slackbuilds/
PACKAGE LOCATION:  ./vlc
PACKAGE SIZE (compressed):  24012 K
PACKAGE SIZE (uncompressed):  94120 K
PACKAGE DESCRIPTION:
vlc: vlc (multimedia player for various audio and video formats)
vlc:
vlc: VLC media player is a highly portable multimedia player for various
vlc: audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, mp3, ogg, ...)
vlc: as well as DVDs, VCDs, and various streaming protocols.
vlc: It can also be used as a server to stream in unicast or multicast in
vlc: IPv4 or IPv6 on a high-bandwidth network.
vlc:
vlc:
vlc: vlc home: http://www.videolan.org/vlc/
vlc:


root ~ ☺ # slackpkg install restricted:vlc

Added external repository alienbob :
                URL=http://192.168.178.201/alien/x86_64/
                KEY=Eric Hameleers <alien@slackware.com>

Added external repository restricted :
                URL=http://192.168.178.201/restricted/x86_64/
                KEY=Eric Hameleers <alien@slackware.com>

Looking for vlc in package list. Please wait... DONE

No packages match the pattern for install. Try:

        /usr/sbin/slackpkg reinstall|upgrade


root ~ ☺ # slackpkg install alienbob:vlc 

Added external repository alienbob :
                URL=http://192.168.178.201/alien/x86_64/
                KEY=Eric Hameleers <alien@slackware.com>

Added external repository restricted :
                URL=http://192.168.178.201/restricted/x86_64/
                KEY=Eric Hameleers <alien@slackware.com>

Looking for vlc in package list. Please wait... DONE

No packages match the pattern for install. Try:

        /usr/sbin/slackpkg reinstall|upgrade

So slackpkg knows that there are two VLC packages, but it doesn't install any of them. Tried the same with wine, same result.

phenixia2003 04-25-2013 11:15 AM

Hello,

Quote:

Originally Posted by Alien Bob (Post 4938669)
Slackpkg uses a "file://" syntax for that, please try to follow the original syntax where possible. Your solution only uses one slash.

You're right. I will fix this.



Quote:

Originally Posted by Alien Bob (Post 4938669)
I will re-generate the multilib repository files with my new script, so that the CHECKSUMS.md5 will have entries for the .asc files.

Thanks for that.


Quote:

Originally Posted by TobiSGD (Post 4938688)
Shouldn't have been so fast, it is not working as it should, I am unable to install any package from the third party repositories

That's what I get when a package is already installed. Try with "slackpkg upgrade" instead. Anyway, there's a problem when using restricted and alien repositories at the same time: it always selects packages from "alien" repository and never from "restricted". That's strange because I didn't encounter that kind of problem when using slacky and alien repositories at the same time. I will look at this asap.

Greetings.

--
SeB

TobiSGD 04-25-2013 12:45 PM

Quote:

Originally Posted by phenixia2003 (Post 4938749)
That's what I get when a package is already installed. Try with "slackpkg upgrade" instead.

I uninstalled VLC before testing this. I have the same problem when I try to install Wine, which definitely never was installed on this machine.

Quote:

Anyway, there's a problem when using restricted and alien repositories at the same time: it always selects packages from "alien" repository and never from "restricted".
That would be a progress for me, for now it refuses to install any of the packages, even if I disable the restricted repository (of course with running slackpkg update afterwards).

zerouno 04-26-2013 02:40 AM

ops I'd not see that there was other updates from my last visits.
I developed the following to add gpg supports.

Now I see your code.

Code:

# Thanks to AlienBob and phenixia2003 (on LQ) for contributing


declare -A MIRRORPLUS
if [ -e /etc/slackpkg/slackpkgplus.conf ];then
  . /etc/slackpkg/slackpkgplus.conf
fi

if [ "$SLACKPKGPLUS" = "on" ];then
  # If CHECKGPG is "on", the system will FAIL the GPG signature of extra repository
  # Use MD5 check instead
#  CHECKGPG=off

  REPOPLUS=${!MIRRORPLUS[*]}
  PRIORITY=( ${PRIORITY[*]} slackpkgplus_$(echo $REPOPLUS|sed 's/ / slackpkgplus_/g') )
 

    # -- merge priorities from PKGS_PRIORITY with PRIORITY, as needed ...
 
  if [ ! -z "$PKGS_PRIORITY" ] ; then
    PREFIX=""
   
    for pp in ${PKGS_PRIORITY[*]} ; do
      repository=$(echo "$pp" | cut -f1 -d":")
      package=$(echo "$pp" | cut -f2- -d":")
   
      if [ ! -z "$repository" ] && [ ! -z "$package" ] ; then
        if [ -z "$PREFIX" ] ; then
          PREFIX=( slackpkgplus_${repository}:$package )
        else
          PREFIX=( ${PREFIX[*]} slackpkgplus_${repository}:$package )
        fi
      fi
    done
   
    [ ! -z "$PREFIX" ] && PRIORITY=( ${PREFIX[*]} ${PRIORITY[*]} )
  fi

 
  function checkgpg() {
  set -x
    gpg --verify ${1}.asc ${1} 2>/dev/null && echo "1" || echo "0"
    if [ "$(basename $1)" == "CHECKSUMS.md5" ];then
      for PREPO in $REPOPLUS;do
        egrep -e ^[a-f0-9]{32} ${TMPDIR}/CHECKSUMS.md5-$PREPO|sed -r "s# \./# ./slackpkgplus_$PREPO/#" >> ${TMPDIR}/CHECKSUMS.md5
      done
    fi
    set +x
  }
  function getfile(){
    local URLFILE
    URLFILE=$1

    if echo $URLFILE|grep -q /slackpkgplus_;then
      PREPO=$(echo $URLFILE|sed -r 's#^.*/slackpkgplus_([^/]+)/.*$#\1#')
      URLFILE=$(echo $URLFILE|sed "s#^.*/slackpkgplus_$PREPO/#${MIRRORPLUS[$PREPO]}#")
    fi
   
    $DOWNLOADER $2 $URLFILE
    if [ $(basename $1) = "MANIFEST.bz2" ];then
      if [ ! -s $2 ];then
        echo -n|bzip2 -c >$2
      fi
    fi
    if [ $(basename $1) = "CHECKSUMS.md5" ];then
      for PREPO in $REPOPLUS;do
        $DOWNLOADER $2-$PREPO ${MIRRORPLUS[${PREPO/slackpkgplus_}]}CHECKSUMS.md5
      done
    fi
    if [ $(basename $1) = "CHECKSUMS.md5.asc" ];then
      for PREPO in $REPOPLUS;do
        $DOWNLOADER ${TMPDIR}/CHECKSUMS.md5-$PREPO.asc ${MIRRORPLUS[${PREPO/slackpkgplus_}]}CHECKSUMS.md5.asc
        if [ $? -eq 0 ];then
          if [ $(checkgpg ${TMPDIR}/CHECKSUMS.md5-$PREPO) -ne 1 ];then
            echo
            echo "                        !!! F A T A L !!!"
            echo "    Repository '$PREPO' FAILS to check CHECKSUMS.md5 signature"
            echo "    The file may be corrupted or the gpg key may be not valid."
            echo "    Remember to import keys by launching 'slackpkg update gpg'."
            echo
            sleep 5
            echo > ${TMPDIR}/CHECKSUMS.md5
          fi
        else
          echo
          echo "                  !!! W A R N I N G !!!"
          echo "    Repository '$PREPO' does NOT supports signature checking"
          echo "    You SHOULD to disable GPG check by setting 'CHECKGPG=off'"
          echo "    in /etc/slackpkg/slackpkg.conf"
          echo
          sleep 5
        fi
      done
    fi
    if [ $(basename $1) = "ChangeLog.txt" ];then
      for PREPO in $REPOPLUS;do
        $DOWNLOADER $2-tmp ${MIRRORPLUS[${PREPO/slackpkgplus_}]}PACKAGES.TXT
        echo $PREPO $(md5sum $2-tmp|awk '{print $1}') >>$2
        rm $2-tmp
      done
    fi
    if [ $(basename $1) = "GPG-KEY" ];then
      for PREPO in $REPOPLUS;do
        $DOWNLOADER $2-tmp ${MIRRORPLUS[${PREPO/slackpkgplus_}]}GPG-KEY
        if [ $? -eq 0 ];then
          gpg --import $2-tmp
        else
          echo
          echo "                  !!! W A R N I N G !!!"
          echo "    Repository '$PREPO' does NOT contains the GPG-KEY"
          echo "    You SHOULD to disable GPG check by setting 'CHECKGPG=off'"
          echo "    in /etc/slackpkg/slackpkg.conf"
          echo
          sleep 5
        fi
        rm $2-tmp
      done
    fi


  }


    # Found packages in repository.
    # This function selects the package from the higher priority
    # repository directories.
    #
    # This Modified version supports enhanced priority rule (priority
    # given to package(s) from a given repository). This kind of priority
    # uses the following syntax :
    #
    #  repository_name:pattern
    #
    #
  function givepriority {
          local DIR
          local ARGUMENT=$1
          local PKGDATA
          local CPRIORITY
          local DIR
          local PKG

          unset NAME
          unset FULLNAME
          unset PKGDATA
         
          for CPRIORITY in ${PRIORITY[@]} ; do
                  [ "$PKGDATA" ] && break
                 
                  if echo "$CPRIORITY " | grep -q "[a-zA-Z0-9]\+[:]" ; then
                    DIR=$(echo "$CPRIORITY" | cut -f1 -d":")
                    PAT=$(echo "$CPRIORITY" | cut -f2- -d":")
                 

                    if echo "$ARGUMENT" | grep -q "$PAT" ; then
                      PKGDATA=( $(grep "^${DIR} ${ARGUMENT} " ${TMPDIR}/pkglist) )

                    fi
                  else
                    PKGDATA=( $(grep "^${CPRIORITY} ${ARGUMENT} " ${TMPDIR}/pkglist) )
                  fi
                   
                  if [ "$PKGDATA" ]; then
                    NAME=${PKGDATA[1]}
                    FULLNAME=$(echo "${PKGDATA[5]}.${PKGDATA[7]}")
                  fi
          done
  }

  if [ "$CMD" == "install" ] || [ "$CMD" == "upgrade" ] ; then

          NEWINPUTLIST=""

          for pref in $INPUTLIST ; do
                  if echo "$pref" | grep -q "[a-zA-Z0-9]\+[:][a-zA-Z0-9]\+" ; then
                          repository=$(echo "$pref" | cut -f1 -d":")
                          package=$(echo "$pref" | cut -f2- -d":")

                          PRIORITY=( slackpkgplus_${repository}:$package ${PRIORITY[*]} )
                  else
                          package=$pref
                  fi
                 
                  NEWINPUTLIST="$NEWINPUTLIST $package"
          done

          INPUTLIST=$NEWINPUTLIST
         
  fi

  if [ "$CMD" == "install-new" ] ; then
    ls -1 /var/log/packages/*compat32 2>/dev/null | rev | cut -f1 -d/ | cut -f4- -d- | rev | sort > $TMPDIR/installed-compat32-packages.lst
   
    grep "[[:digit:]]\+compat32[ ]" $WORKDIR/pkglist | cut -f2 -d" " | sort > $TMPDIR/available-compat32-packages.lst

    NEWCOMPAT32PKGS=$(comm -3 $TMPDIR/installed-compat32-packages.lst  $TMPDIR/available-compat32-packages.lst)
   
    if [ ! -z "$NEWCOMPAT32PKGS" ] ; then
      LIST=""
     
      for pkg in $NEWCOMPAT32PKGS ; do
        LIST="$LIST $(grep " ${pkg} " $WORKDIR/pkglist | cut -f6,8 -d" " --output-delimiter=".")"
      done
    fi
  fi
 
fi

Code:

# Enable (on) / Disable (off) slackpkg+
SLACKPKGPLUS=on

# If two or more repositories contains some same packages, you can specify
# from which repository you prefer to search it.
# The syntax is "<repository_name>:<package_name>"
# Accepts regular expressions. To give order to an entire repository use "<repository_name>:.*"

# Examples:
#PKGS_PRIORITY=( slacky:apache-ant alienbob:openjdk )
#
# If you want a multilib system, uncomment the multilib repository and set:
#PKGS_PRIORITY=( multilib:.* )


# Define mirrors (uncomment one or more mirror)
# Note: You MUST disable gpg check if you select the 'salixext' or 'zerouno' !!!
#

# Slackware 14.0 - x86_64
#MIRRORPLUS['multilib']=http://taper.alienbase.nl/mirrors/people/alien/multilib/14.0/
MIRRORPLUS['alienbob']=http://taper.alienbase.nl/mirrors/people/alien/sbrepos/14.0/x86_64/
MIRRORPLUS['slacky']=http://repository.slacky.eu/slackware64-14.0/
MIRRORPLUS['salixos']=http://download.salixos.org/x86_64/14.0/
MIRRORPLUS['salixext']=http://people.salixos.org/ralvex/packages/x86_64/14.0/

# Slackware 14.0 - i386
#MIRRORPLUS['alienbob']=http://taper.alienbase.nl/mirrors/people/alien/sbrepos/14.0/x86/
#MIRRORPLUS['slacky']=http://repository.slacky.eu/slackware-14.0/
#MIRRORPLUS['salixos']=http://download.salixos.org/i486/14.0/
#MIRRORPLUS['salixext']=http://people.salixos.org/ralvex/packages/i486/14.0/

# Slackware current - x86_64
#MIRRORPLUS['multilib']=http://taper.alienbase.nl/mirrors/people/alien/multilib/current/
MIRRORPLUS['zerouno']=http://www.z01.eu/repo-slack/slackware64-current/
MIRRORPLUS['slackel']=http://www.slackel.gr/repo/x86_64/current/

# Slackware current - i386
#MIRRORPLUS['slackel']=http://www.slackel.gr/repo/i486/current/

For now I know only alien and slacky repositories that fully supports gpg.

phenixia2003 04-26-2013 02:41 AM

Hello,

Quote:

Originally Posted by TobiSGD (Post 4938783)
I uninstalled VLC before testing this. I have the same problem when I try to install Wine, which definitely never was installed on this machine.

I looked at this and, I think I have found the reason of this issue. Indeed "slackpkg info" gets data about packages from the file PACKAGES.TXT, while "slackpkg search/install/..." uses an internal file built from the file CHECKSUMS.md5.

Is your local mirrors include all these files :
Code:

CHECKSUMS.md5
MANIFEST.bz2
PACKAGES.TXT
FILELIST.TXT

if this is not the case, you should add the missing ones, and run "slackpkg update", then, redo "slackpkg install restricted:vlc". This should give you a list with "vlc". Note that if all is ok, and you install "vlc", you will get the version from the repository "alienbob" and not from "restricted". This is because both of these file have exactly the same version and build number. I have an idea to fix this and I'm going to implement it.

Otherwise, If you have all those files above, or if "slackpkg install restricted:vlc" throws the message "No packages match the pattern", could you post the output of "slackpkg update" ?


Greetings.

--
SeB

zerouno 04-26-2013 03:23 AM

It's a good code, but my original idea was to overwrite less code possible (to not rewrite).

So instead to overwrite the entire updatefilelists(), I solved in getfile() all problems you solved in red and green points

1) repositories that does not contains ChangeLog.txt
2) repositories that does not contains MANIFEST.bz2
3) repositories that does not contains GPG-KEY or CHECKSUMS.md5.asc
4) loading third-party CHECKSUMS.md5.asc


p.s.: now slacky repository fully supports GPG

TobiSGD 04-26-2013 03:41 AM

1 Attachment(s)
Quote:

Originally Posted by phenixia2003 (Post 4939128)
Hello,



I looked at this and, I think I have found the reason of this issue. Indeed "slackpkg info" gets data about packages from the file PACKAGES.TXT, while "slackpkg search/install/..." uses an internal file built from the file CHECKSUMS.md5.

Is your local mirrors include all these files :
Code:

CHECKSUMS.md5
MANIFEST.bz2
PACKAGES.TXT
FILELIST.TXT

if this is not the case, you should add the missing ones, and run "slackpkg update", then, redo "slackpkg install restricted:vlc". This should give you a list with "vlc". Note that if all is ok, and you install "vlc", you will get the version from the repository "alienbob" and not from "restricted". This is because both of these file have exactly the same version and build number. I have an idea to fix this and I'm going to implement it.

Otherwise, If you have all those files above, or if "slackpkg install restricted:vlc" throws the message "No packages match the pattern", could you post the output of "slackpkg update" ?


Greetings.

--
SeB

My repositories are 1:1 copies of AlienBob's, I rsync them once a day, but I have checked and those files exist in both repositories.
Output of
Code:

slackpkg update
is attached to this post.

zerouno 04-26-2013 03:46 AM

This should to solve the Workman repository, but I've not tested it.
Code:

      X86_64=$(ls /var/log/packages/aaa_base*x86_64*|head -1 2>/dev/null)
      for PREPO in $REPOPLUS;do
        if [ -z "$X86_64" ];then
          egrep -e ^[a-f0-9]{32} ${TMPDIR}/CHECKSUMS.md5-$PREPO|egrep -- "-(x86_64|noarch)-" |sed -r "s# \./# ./slackpkgplus_$PREPO/#" >> ${TMPDIR}/CHECKSUMS.md5
        else
          egrep -e ^[a-f0-9]{32} ${TMPDIR}/CHECKSUMS.md5-$PREPO|egrep -v -- "-(x86_64|arm)-" |sed -r "s# \./# ./slackpkgplus_$PREPO/#" >> ${TMPDIR}/CHECKSUMS.md5
        fi
      done


zerouno 04-26-2013 07:38 AM

Found a BUG in slackpkg! (non plus)

The bug i tested in slackpkg search but may be extended in other sections.

This may the reason becouse slackpkg+ does not work with vlc.


The bug:

when you search a package with following requirements, the package is NOT listed.
1) exists a package called 'something-packagesearched' (note the minus sign)
2) the packages 'something-packagesearched' is listed prior than 'packagesearched' in pkglist

A package affected is 'glib':

Code:

# grep -w glib /var/lib/slackpkg/pkglist
slackware64 dbus-glib 0.100.2 x86_64 1 dbus-glib-0.100.2-x86_64-1 ./slackware64/l txz
slackware64 glib 1.2.10 x86_64 3 glib-1.2.10-x86_64-3 ./slackware64/l txz
slackware64 glib-networking 2.34.2 x86_64 1 glib-networking-2.34.2-x86_64-1 ./slackware64/l txz
# slackpkg search glib
The list below shows all packages with name matching "glib".

[ installed ] - glibc-solibs-2.17-x86_64-2
[ installed ] - glibc-zoneinfo-2013b-noarch-2
[ installed ] - dbus-glib-0.100.2-x86_64-1
[ installed ] - glib-networking-2.34.2-x86_64-1
[ installed ] - glib2-2.34.3-x86_64-1
[ installed ] - glibc-2.17-x86_64-2
[ installed ] - glibc-i18n-2.17-x86_64-2
[ installed ] - glibc-profile-2.17-x86_64-2
[ installed ] - taglib-1.8-x86_64-1
[ installed ] - taglib-extras-1.0.1-x86_64-1

The problem is in core-functions.sh near line 700:

Code:

for i in ${PRIORITY[@]}; do
        PKGS=$(grep "^${i}.*${PATTERN}" \
                ${TMPDIR}/pkglist | cut -f6 -d\ )
        for FULLNAME in $PKGS ; do
          echo -e "\n'$FULLNAME - $LIST'"
                NAME=$(cutpkg ${FULLNAME})
                echo $LIST | \
                        grep -qwe "${NAME}-[^-]\+-\(${ARCH}\|fw\|noarch\)-[^-]\+" && \
                        continue
                LIST="$LIST ${FULLNAME}"
        done
done

The 'grep -qwe' is used to avoid duplicates in $LIST, but the grep search for "-w ${NAME}-..." that match in "something-${NAME}-version-..." if it is already in $LIST, and 'dbus-glib' is already in list when 'glib' is checked.
The solution is grep -qwe "^${NAME}-[^-]\+-\(${ARCH}\|fw\|noarch\)-[^-]\+", but I don't known how many occurs there are in all slackpkg functions.

vlc in slackpkg+ is not listed becouse exists at least 'npapi-vlc'.

phenixia2003 04-26-2013 08:28 AM

Hello,

Quote:

Originally Posted by zerouno (Post 4939252)
Found a BUG in slackpkg! (non plus)

vlc in slackpkg+ is not listed becouse exists at least 'npapi-vlc'.

I have sent this patch for that bug last February.

But I think the issue encountered by TobiSGD is elsewhere because, even without this patch, I have no problem with "slackpkg install alienbob:vlc" (or restricted:vlc) on my side.


--
SeB

zerouno 04-26-2013 08:36 AM

slackpkg search is the only affected from that bug?

phenixia2003 04-26-2013 10:14 AM

Quote:

Originally Posted by zerouno (Post 4939289)
slackpkg search is the only affected from that bug?

When I wrote this patch I looked to see if install/upgrade was affected too, but this was not the case. In fact, as opposed to "install|update|reinstall" the list generated by "search|file-search" cannot contain packages with the same basename.

I guess "search|file-search" would have to be overwritten in slackpkg+, otherwise users will get wrong information.

--
SeB

zerouno 04-26-2013 10:21 AM

Quote:

Originally Posted by phenixia2003 (Post 4939340)
I guess "search|file-search" would have to be overwritten in slackpkg+, otherwise users will get wrong information.

we are rewritting the entire slackpkg :D

zerouno 04-26-2013 10:32 AM

Uploaded beta1:
slackpkg+-0.9beta1-noarch-1.txz

Done some merging code.


All times are GMT -5. The time now is 10:47 AM.