Shel scripting: variables pointing to variables and case

Dark_Helmet · 06-07-2003, 12:01 PM

Ok, I'm working on a script for myself to automate installation of software packages for an LFS system. Here's the problem... I create a number of variables such as:

Code:

BASH_PKG="bash-2.05a"
BINUTILS_PKG="binutils-2.13.2"
...

Then I create a list (to make sure packages are compiled in the right order):

Code:

static_package_order='"${BASH_PKG}" "${BINUTILS_PKG}" ... '

Finally, when the script gets down to business, I use a case that doesn't work the way I'd like. I have a something like this:

Code:

for package in ${static_package_order} ;
  case "${package}" in
    *BASH_PKG*)
               <commands>
               ;;

    *BINUTILS_PKG*)
               <commands>
               ;;
    ...

That works, but in my opinion, it looks nasty.

The package variable contains the text string "${BASH_PKG}" and does not evaluate it any further wheras using "${BASH_PKG})" for the match IS evaluated, and resolves to "bash-2.05a)".

I would like to use ${BASH_PKG} for the matching protion of the case, but I can't figure out any way to get it to work. Like I said, the *'s work, but looks bad. Any suggestions?

Hko · 06-07-2003, 12:52 PM

[/EDIT]
Sorry, this was not a good solution. It doesn't preserve the order.
[/EDIT]

Some other way:

Code:

BASH_PKG="bash-2.05a"
BINUTILS_PKG="binutils-2.13.2"

set | sed -n 's/^\(.*_PKG\)=.*/\1/p' | while read package ; do
   case "${package}" in
      BASH_PKG)
            echo "Package: $package"  # dummy command for demo
            ;;

      BINUTILS_PKG)
            echo "Package: $package"  # dummy command for demo
            ;;
   esac
done

There's two conditions when using this:

1) All your package name variables should end with "_PKG"
2) No other environment variables should end with "_PKG".

Dark_Helmet · 06-08-2003, 12:33 AM

I hadn't noticed that the first time I looked at it. In fact, I probably wouldn't have noticed it unless I inspected the script's output very closely.

Thanks for taking a stab at it.. I did find your use of sed piped into read interesting. That kind of combination has never crossed my mind.

------- Edit -----------

Figured it out... I thought I'd tried this before, but...

Instead of:

Code:

static_package_order='"${BASH_PKG}" "${BINTUILS_PKG}" ... '

It should be:

Code:

static_package_order=" ${BASH_PKG} ${BINUTILS_PKG} ... "

I could have sworn I tried that earlier with the interpreter complaining about an unknown command (as though it were expanding the elements of the list and trying to execute them). That's what led me to create the list using the single-quote, and that in turn was preventing the interpreter from evaluating each individual package name.

Of course, this assumes that evaluation of all the XXXX_PKG variables does not include any whitespace, which in my case, they don't.

Oh well... it works, it looks pretty, and now all I have to do is work up the nerve to test it out...

Hko · 06-08-2003, 07:37 AM

Quote:

Originally posted by Dark_Helmet

Code:

static_package_order=" ${BASH_PKG} ${BINUTILS_PKG} ... "

Sorry, I don't understand why this works for you, but I may overlook something off course. Now the PKG-vars get expanded inside static_package_order, so "$static_package_order" will expand to "bash-2.05a binutils-2.13.2". So now the cases to test for need to be something like:

Code:

case "${package}" in
    bash*)bash-2.05a
        # note that you need to hard-code the actual package name
        # below to handle (untar/compile...) the package.
        tar xzf bash-2.05a.tar.gz
        cd bash-2.05a
        ./configure  && make install    #...... whatever....
        ...
        ;;

If I understand your question correctly, this is not what you wanted. (I may be wrong here...)

Why not seperating the package list from your script to a text file where you list the packages in the right order. Then you'll not have to change your script when a package version changes, or even if "binutils" is renamed to "binairyutilities-2.13-2", just change the actual name in the text file that contains the package list.

I mean, for example, you could make a text file "packages.txt" which reads:

Code:

# package.txt
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# List of symbolic packages names (like a variable) and their 
# actual names. List in correct order of installing.
#
# Syntax: <sym. package name>=<actual package name>
#
# Line beginning with '#' are comments.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

BASH_PKG=bash-2.05a
BINUTILS_PKG=binutils-2.13.2
# ....

And then have the script read the file 'packages.txt' line by line, put the string before the '=' in one variable, and the string after the '=' in another.

This way the order of the packages is maintained, and you have both corresponding package names (the symbolic, general name and the actual name) in two different var's inside the handling loop.

Code:

#!/bin/bash

PKGLIST="packages.txt"

cat "${PKGLIST}" | while read LINE ; do

    # Skip comments (lines starting with '#') in packages.txt
    LINE=${LINE##\#*}
    if [ "${LINE}" == "" ] ; then
	continue
    fi

    # split line read from text file into a package name
    # and its corresponding actual name.
    GENERAL_PACKAGE_NAME=${LINE%%=*}
    ACTUAL_PACKAGE_NAME=${LINE##*=}

    # Handle packages:
    case "${GENERAL_PACKAGE_NAME}" in

	BASH_PKG)
	    echo "[DEMO] Package BASH --> ${ACTUAL_PACKAGE_NAME}"

	    # Now we can use a *variable* here to refer to what
	    # actually is "bash-2.05a". When a new version comes
	    # available and we run this script again, we don't have
	    # to change anything in the script!  Just rename the
	    # actual package name in 'packages.txt' to read, say, 
	    # "bash-9.87-12".

	    # Example:
	    # ~~~~~~~~
	    # tar xzf "${ACTUAL_PACKAGE_NAME}.tar.gz"
	    # cd "${ACTUAL_PACKAGE_NAME}"
	    # ./configure && make install

	    ;;
      
	BINUTILS_PKG)
	    echo "[DEMO] Package: BINUTILS --> ${ACTUAL_PACKAGE_NAME}" 
	    ;;
    esac
done
exit 0

Also, changing the order of installation is easier now, because the script does not have to be changed, and it's more comfortable to change the order of the lines in 'package.txt'.

Hko · 06-08-2003, 08:11 AM

Myself I've never tried LFS, but I suppose many packages need the same handling, let's say: un-tar, change to it's directory and do a ./configure && make install.

If so, the script from my previous post can be changed so it doesn't even need a seperate case test for each different package; only for packages that need special handling need a seperate case-test.

Code:

#!/bin/bash

PKGLIST="packages.txt"

cat "${PKGLIST}" | while read LINE ; do

    # Skip comments (lines starting with '#') in packages.txt
    LINE=${LINE##\#*}
    if [ "${LINE}" == "" ] ; then
	continue
    fi

    # split line read from text file into a package name
    # and its corresponding actual name.
    GENERAL_PACKAGE_NAME=${LINE%%=*}
    ACTUAL_PACKAGE_NAME=${LINE##*=}

    # Handle packages:
    case "${GENERAL_PACKAGE_NAME}" in

	# Only define cases for packages that need some
	# special treatment to install.  Say, for example the package
	# "emacs" is compressed with bzip2 instead of gzip, so tar needs
	# the 'j' option instead of 'z'. And let's also say emacs  needs
	# some byte-compiling of emacs-lisp code:

	EMACS_PKG)
	    tar xjf "${ACTUAL_PACKAGE_NAME}.tar.bz2"
	    cd "${ACTUAL_PACKAGE_NAME}"
	    ./configure && make install
	    #
	    # Add commands here to byte-compile elisp files
	    #
	    ;;

	*)
	    # Default handling.
	    # Handle all packages that only require the general "recipe"
	    # to compile and install.

	    tar xzf "${ACTUAL_PACKAGE_NAME}.tar.gz"
	    cd "${ACTUAL_PACKAGE_NAME}"
	    ./configure && make install
	    ;;
    esac
done
exit 0

Dark_Helmet · 06-08-2003, 11:07 AM

Below I post some of the meaningful code in the script. You'll see that I I've implemented a variant on some of what you mentioned (each package is broken into a generic sequence of commands for instance).

The way I've set up the variable names is extremely predictable on purpose. THe idea was that if a package changed, all that's needed are a handful of sed commands to change the variable definitions, and you're ready to go. Similarly, the static_packkage_order variable can be updated with a similar sequence of sed commands.

Anyway, the case matching now works by referencing the package name. Such as:
${BASH_PKG})

I'm assuming that, in the context of a case, the variable is expanded as a string (and not a command) when it is followed immediately by a close-parenthesis. As long as I each package name is guaranteed not to have a space in the filename, this should work.

I've run the code as I develop it for debug, and it's more than happy with the way it's coded up below. When I originially posted the message, it was ALWAYS falling through to the default clause of my case until I started using the asterisks, and eventually changed the static_package_order variable.

If you have the time to look over what I posted, and can offer some suggestions, I would appreciate it.

Code:

#========================================================================
# bash package information and commands
#========================================================================
BASH_PKG=bash-2.05a
BASH_STATIC_PRE="${EMPTY_CMD}"
BASH_STATIC_CFG="./configure --enable-static-link --prefix=$LFS/static --with-curses"
BASH_STATIC_MAKE="make"
BASH_STATIC_INSTALL="make install"
BASH_STATIC_POST="${EMPTY_CMD}"
BASH_STATIC_PATCHES="${EMPTY_CMD}"

#========================================================================
# binutils package information and commands
#========================================================================
BINUTILS_PKG="binutils-2.13.2"
BINUTILS_STATIC_PRE="mkdir ../binutils-build && cd ../binutils-build"
BINUTILS_STATIC_CFG="../binutils-2.13.2/configure --prefix=$LFS/static --disable-nls"
BINUTILS_STATIC_MAKE="make LDFLAGS='-all-static'"
BINUTILS_STATIC_INSTALL="make install"
BINUTILS_STATIC_POST="${EMPTY_CMD}"
BINUTILS_STATIC_PATCHES="${EMPTY_CMD}"

...
[Package-specific variables continue for a while]
...

#========================================================================
# Static packages and their compilation order
#========================================================================
static_package_order=" ${BASH_PKG}
                       ${BINUTILS_PKG}
                       ${BZIP2_PKG}
                       ${DIFFUTILS_PKG}
                       ${FILEUTILS_PKG}
                       ${FINDUTILS_PKG}
                       ${GAWK_PKG}
                       ${GCC_PKG}
                       ${GREP_PKG}
                       ${GZIP_PKG}
                       ${MAKE_PKG}
                       ${PATCH_PKG}
                       ${SED_PKG}
                       ${SHUTILS_PKG}
                       ${TAR_PKG}
                       ${TEXINFO_PKG}
                       ${TEXTUTILS_PKG}
                       ${UTIL_LINUX_PKG} "


#========================================================================
# Function definitions
#========================================================================

# Function: gen_cmd()
# GENeral COMmand is just that, it reduces the number of keystrokes needed
# when implementing the package case statement found below by creating a
# general structure for executing compilation commands. Since error chacking
# is a must, that's what this function's main purpose is: to remove error
# checking from the main body of the script and to implement it in a uniform
# manner.
function gen_cmd ()
{
  local logging_enabled
  local package_name
  local package_command
  local log_file
  local command_status

  # Yes, I'm anal. Put each argument passed into a MEANINGFUL name rather than
  # using $1, $2, etc. within the functional script commands later.
  logging_enabled=$1
  package_name=$2
  package_command=$3
  log_file=$4
  command_status=0            # default to a good exit status

  # The commands for each package are sent to this function regardless of
  # whether there's any meaningful work to be done. So we check to see if
  # we have to do anything here...
  if [ "${package_command}" != "${EMPTY_CMD}" ] ; then

    # Ok, we have something to do. Do we need to log the output?
    if [ ${logging_enabled} -eq 1 ] ; then
      echo "[Command] time ${package_command} > ${log_file}"

      # Using time is not necessary, but I thought it would be cool to add its
      # output if logging is enabled. Then I can claim how bad-ass my computer
      # is by quoting compile times... until someone else's computer smokes
      # my times...
      #-#time ${package_command} > ${log_file}
      #-#command_status=$?
    else
      echo "[Command] ${package_command}"

      # Plain vanilla version. Just execute the command.
      #-#${package_command}
      #-#command_status=$?
    fi

    # Alright, the command has run, and we put the exit status into
    # command_status. Check that the command executed successfully. If not
    # display a message and exit
    # NOTE: This assumes an exit status of 0 is always good. For most cases
    # this is correct, but it might be better to pass in an expected exit
    # status per command
    if [ ${command_status} -ne 0 ] ; then
      echo "\!\!\!Failure working on package ${package_name}"
      echo "Command returned a status of: ${command_status}"
      echo "Script is exiting with same status..."
      exit ${command_status}
    fi
  fi

}

...
[Other code to help "normalize" the environment]
...

# Now we start compiling the packages...
# Go through the static package list in order
for package in ${static_package_order} ;
do
  # Inform the user what package is being worked on at the moment and
  # provide them a nice, warm, and fuzzy feeling by displaying the command
  # we plan to use to extract the archive, what directory we move to, patches
  # we apply, etc.
  echo " "
  echo "--- Step 3 ---"
  echo "Extracting ${package}"
  #-#package_filename=`find ${STATIC_PKG} -name "${package}.bz2"`

  # Check to make sure the find command returned a valid file. Otherwise,
  # the package does not exist.
  #-#if [ -e "${package_filename}" ] ; then
  #-#  echo "\!\!\!Error: Could not find ${package} archive file"
  #-#  echo "Expected archive path and filename: ${package_filename}"
  #-#  echo "The script cannot reliably continue."
  #-#  echo "Exiting: ${PACKAGE_NOT_FOUND_ERROR}"

  echo "[Command] ${UNTAR_CMD} ${package_filename}"

  # An assumption is being made here. The assumption being that the tar
  # archive was created without using absolute paths. That should be a safe
  # assumption since the idea is to create LFS from a distributions and not
  # install LFS on TOP OF an existing distribution
  #-#package_dir=`${UNTAR_CMD} ${package_filename} | cut -f 1 -d "/"`

  echo "Changing directory to ${package_dir}"
  #-#cd ${package_dir}

  echo " "
  echo "--- Step 4 ---"
  # Not much information here, and that's intentional. The very nature of the
  # problem with compiling packages is that they're all pretty much unique.
  # So I reduced the steps into five groups. For a description of what each
  # group represents, head back to the top of the file.
  case "${package}" in

    #########################################################################

    ${BASH_PKG})
      if [ ${KEEP_LOGS} -eq 1 ] ; then
        if [ ! -d ${STATIC_LOG}/${BASH_PKG} ] ; then
          gen_cmd 0 "${BASH_PKG}" "mkdir ${STATIC_LOG}/${BASH_PKG}" "/dev/null"
        fi
        LOG_FILE="${STATIC_LOG}/${BASH_PKG}/${BASH_PKG}"
      fi

      if [ "${BASH_STATIC_PATCHES}" != "${EMPTY_CMD}" ] ; then
        for pkg_patch in ${BASH_STATIC_PATCHES} ;
        do
          patch_file="${STATIC_PKG}/${pkg_patch}"
          gen_cmd ${KEEP_LOGS} "${BASH_PKG}" "${PATCH_CMD} ${patch_file}" "${LOG_FILE}.patches"
        done
      fi

      gen_cmd ${KEEP_LOGS} "${BASH_PKG}" "${BASH_STATIC_PRE}" "${LOG_FILE}.pre"
      gen_cmd ${KEEP_LOGS} "${BASH_PKG}" "${BASH_STATIC_CFG}" "${LOG_FILE}.cfg"
      gen_cmd ${KEEP_LOGS} "${BASH_PKG}" "${BASH_STATIC_MAKE}" "${LOG_FILE}.make"
      gen_cmd ${KEEP_LOGS} "${BASH_PKG}" "${BASH_STATIC_INSTALL}" "${LOG_FILE}.install"
      gen_cmd ${KEEP_LOGS} "${BASH_PKG}" "${BASH_STATIC_POST}" "${LOG_FILE}.post"
      ;;

    #########################################################################

    ${BINUTILS_PKG})
      if [ ${KEEP_LOGS} -eq 1 ] ; then
        if [ ! -d ${STATIC_LOG}/${BINUTILS_PKG} ] ; then
          gen_cmd 0 "${BINUTILS_PKG}" "mkdir ${STATIC_LOG}/${BINUTILS_PKG}" "/dev/null"
        fi
        LOG_FILE="${STATIC_LOG}/${BINUTILS_PKG}/${BINUTILS_PKG}"
      fi

      if [ "${BINUTILS_STATIC_PATCHES}" != "${EMPTY_CMD}" ] ; then
        for pkg_patch in ${BINUTILS_STATIC_PATCHES} ;
        do
          patch_file="${STATIC_PKG}/${pkg_patch}"
          gen_cmd ${KEEP_LOGS} "${BINUTILS_PKG}" "${PATCH_CMD} ${patch_file}" "${LOG_FILE}.patches"
        done
      fi

      gen_cmd ${KEEP_LOGS} "${BINUTILS_PKG}" "${BINUTILS_STATIC_PRE}" "${LOG_FILE}.pre"
      gen_cmd ${KEEP_LOGS} "${BINUTILS_PKG}" "${BINUTILS_STATIC_CFG}" "${LOG_FILE}.cfg"
      gen_cmd ${KEEP_LOGS} "${BINUTILS_PKG}" "${BINUTILS_STATIC_MAKE}" "${LOG_FILE}.make"
      gen_cmd ${KEEP_LOGS} "${BINUTILS_PKG}" "${BINUTILS_STATIC_INSTALL}" "${LOG_FILE}.install"
      gen_cmd ${KEEP_LOGS} "${BINUTILS_PKG}" "${BINUTILS_STATIC_POST}" "${LOG_FILE}.post"
      ;;

    #########################################################################

...
[Many other packages listed]
...