LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Articles > Technical
User Name
Password

Notices


By louigi600 at 2016-09-09 13:46
I use bash scripts to do all sorts of things, both in my spare time and for my job, and many of them require creating and reading a configuration file or parsing some options or both.
I used to end up writing very similar code over and over again and dealing with creating config files, parsing options and usage message used to take up a big part in the script. I wanted to do something about this to make my life simpler and minimize the amount of code I'd need to write for dealing with such issues and possibly stop rewriting each time nearly the same thing.

Let's deal with the first issue: minimizing the amount of code to get the script to prompt you with the values to put into it's configuration file. Yes I like my scripts to prompt me for their configuration if the is not one yet and I like to keep it in a separate file so I can update the script without messing up the configuration.
For me the config files would generally contain the declaration of some variables that I would then source into the script itself. Something like this:

Code:
VM_PATH='/VM'
VM_DB='/VM/vm.conf'
ISO_PATH='/ISO'
MONITOR_START_PORT='1030'
SHUTDOWN_TIMEOUT='300'
OS_RESERVED_MEM='512'
Ok the single quotes aren't necessary here but this was automatically created by a script so that's why they all have single quotes in case the variable has blank spaces in them.

Let's have a look at how I get the script to prompt me for each one of those variables, only if no config file is found, without writing specific code for every single configuration option.

First of all I declare a variable that contains all the names of the variables that will go in the configuration file

Code:
SCRIPT_PARAMS="VM_PATH VM_DB ISO_PATH MONITOR_START_PORT SHUTDOWN_TIMEOUT OS_RESERVED_MEM"
Optionally I declare some sensible default values for those variables (used for prompting a default)
Code:
VM_PATH=/VM
VM_DB=${VM_PATH}/vm.conf
ISO_PATH=/ISO
MONITOR_START_PORT=1030
SHUTDOWN_TIMEOUT=300
OS_RESERVED_MEM=512
Then early in the script execution I would issue something like this (which implies that the scripts config file has the same name as the the script with ".conf" appended to it)
Code:
NAME=$(basename $0)
CWD=$(dirname $0)
[ -r ${CWD}/${NAME}.conf ] && source ${CWD}/${NAME}.conf || make_config
All the magic is done in the make_config function, along with some other functions used in there. Let's have a look at that code:

Code:
make_config ()
{ ACCEPT=0
  while [ $ACCEPT -eq 0 ]
  do
    form_dialog "Basic config for $NAME" $SCRIPT_PARAMS && IFS=';' read -r -d '\;' $SCRIPT_PARAMS < $DIALOG_OUTPUT
    accept_dialog "Accept Configuration ?" $SCRIPT_PARAMS 
    ACCEPT=$?
  done
   
  ( for PARAM in $SCRIPT_PARAMS
    do
      echo "${PARAM}='$(eval echo "\$$PARAM")'"
    done
  ) > $CWD/${NAME}.conf
  chmod 600 $CWD/${NAME}.conf
  sleep 1
}
I used to have make_config to prompt for each variable by writing text on the terminal and subsequently print all of them back asking if it was ok to write the to the config file but I now prefer to have dialog do that for me all in one go with the form_dialog function and then prompt me again with accept_dialog function to acknowlege that it's all ok to go into the config file:

Code:
form_dialog ()
{ TITLE=$1
  shift
  unset STRING
  MAX=0
  for A in $*
  do
    [ $((${#A} +3))  -gt $MAX ] && MAX=$((${#A} +3))
  done 
  i=0
  for A in $*
  do
    STRING[$i]="${A}: $(($i + 1))  1  \"$(eval echo "\$$A")\" $(($i + 1)) $MAX 40 0"
    ((i++))
  done 
  eval $(echo dialog --title \"$TITLE\" --separator \'\;\' --form \"\" 0 0 0 ${STRING[*]} \2\>$DIALOG_OUTPUT )
}
accept_dialog ()
{ TITLE=$1
  shift
  unset STRING
  MAX=0
  for A in $*
  do
    [ $((${#A} +3))  -gt $MAX ] && MAX=$((${#A} +3))
  done
  i=0
  for A in $*
  do
    STRING[$i]="${A} = $(eval echo "\$$A") \n"
    ((i++))
  done
  eval $(echo dialog --title \"$TITLE\" --yesno  \" ${STRING[*]}\" 0 0 ) && return 1 || return 0
}
You may be thinking that's a lot of code for just 6 options in the config file ... but if you had 30 options in the config file all that would change is the SCRIPT_PARAMS variable, no extra code would need to be written: I love that !


Now let's look at how I deal with minimizing the code for the script parameter parsing and usage message generation. I use getopt to parse the script flags and options which in turn does reduce code but still the GETOPT_STRING variable needs to be created according to the flags and options you intend to parse and the code needs to be written to output a usage help message. Like the config file thing I work around the problem by having a variable with all the information required ... but wait a second here it's a little more complicated because each needs to know if it's a flag or parameter and the help message describing what it does: I use an array for this:

Code:
MYOPTIONS=(
"a,,\t\t:Do not require association to AP ,$ASSOCIATE"
"c,:,\t:Wpa supplicant config (default $WCFG),$WCFG"
"d,,\t\t:Do not require IP via DHCP,$DHCP"
"f,,\t\t:Do not activate basic Firewallroles,$FIREWALL"
"h,,\t\t:Show this help message,0"
"i,:,\t:Use wireless specific interface,$NIC_NAME" 
"m,:,\t:Set the interface to a specific mode (default $MODE),$MODE"
"M,:,\t:Use interface with this MAC (eg -M $MAC_EG),$MAC"
"r,:,\t:Set REGION (default $REG),$REG"
"s,:,\t:Register to AP with this essid,$ESSID"
"w,,\t\t:Do not Start wpa_supplicant for AP association,$WPA_SUPP"
)
where each element contains all the information for each flag/option separated by colons:
  1. the flag/parameter (I don't use long options in my scripts or things get too complicated)
  2. ":" means it requires a parameter id empty it's a flag
  3. the text message describing what it does for usege (the extra tabs help allign properly)
  4. the value stored for the option

Modern bash has now associative arrays (that would make it easier to access an array element named after it's flag name) but I thaught this up prior to the existence of bash4 (nothing against ksh but did not want to deal with making sure it was installed on all the systems). With bash4 or ksh the MYOPTIONS could look like this:

Code:
MYOPTIONS=(
[a]=",\t\t:Do not require association to AP ,$ASSOCIATE"
[c]=":,\t:Wpa supplicant config (default $WCFG),$WCFG"
[d]=",\t\t:Do not require IP via DHCP,$DHCP"
[f]=",\t\t:Do not activate basic Firewallroles,$FIREWALL"
[h]=",\t\t:Show this help message,0"
[i]=":,\t:Use wireless specific interface,$NIC_NAME" 
[m]=":,\t:Set the interface to a specific mode (default $MODE),$MODE"
[M]=":,\t:Use interface with this MAC (eg -M $MAC_EG),$MAC"
[r]=":,\t:Set REGION (default $REG),$REG"
[s]=":,\t:Register to AP with this essid,$ESSID"
[w]=",\t\t:Do not Start wpa_supplicant for AP association,$WPA_SUPP"
)
And you could do without the part of the code required to find the element of the array for the relevant flag/parameter.

Here's how the usage function gets reduced to containing no specific code:
Code:
usage ()
{ cat <<EOF 

usage: $NAME [OPTIONS]

Wireless Card Management Helper

OPTIONS
EOF
  I=0
  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    PARM=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 2)
    HTXT=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 3)
    [ "$PARM" = ":" ] && echo -e "-$OPTION <value> $HTXT" || echo -e "-$OPTION $HTXT"
    I=$(expr $I + 1)
  done
  
  echo
  if [ "$*" != "" ] 
  then
    echo "Error: $*"
    exit 1
  else
    exit 0
  fi
}

Now let's see how I produce the GETOPT_STRING variable and how I use getopt itself:

Code:
GETOPT_STRING=$(I=0
  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    PARM=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 2)
    echo -en "${OPTION}$PARM" 
    I=$(expr $I + 1)
  done
)
getopt $GETOPT_STRING $* >/dev/null 2>/dev/null || \
  usage unknown option or insufficient parameters: $*

GPO=$(getopt $GETOPT_STRING $* 2>/dev/null |tr -d "'")
set -- $GPO
while [ $# -ge 1 ]
do
  case $1 in
    -a) set_myoptions_value a 0; shift ;;
    -c) set_myoptions_value c $2; shift 2;;
    -d) set_myoptions_value d 0; shift ;;
    -f) set_myoptions_value f 0; shift ;;
    -i) set_myoptions_value i $2; shift 2 ;;
    -m) set_myoptions_value m $2; shift 2;;
    -M) set_myoptions_value M $2; shift 2;;
    -r) set_myoptions_value r $2; shift 2;;
    -s) set_myoptions_value s $2; shift 2;;
    -w) set_myoptions_value w 0; shift ;;
    -h) usage ;;
    --) shift ; break ;;
    *) usage $1 unknown option ;;
  esac 
done
As you can see the only specific code is in the case where I use a function (set_myoptions_value) to store the values in the correct MYOPTIONS array element: that's not much code for handling 11 flags/parameters and the good thing that it's not going to grow much even if I had many more.
There is however some other functions I have to make it easy for me to set and recall values for each flag/parameter but this remains the same and I just copy it into each script that uses MYOPTIONS array:
Code:
get_myoptions_array_element ()
{ I=0
  SEARCH_OPTION=$(echo "$1" |tr -d "-")

  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    [ "$OPTION" = "$SEARCH_OPTION" ] && break
    I=$(expr $I + 1)
  done
  echo $I
}

get_myoptions_value ()
{ I=0
  SEARCH_OPTION=$(echo "$1" |tr -d "-")

  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    VALUE=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 4)
    [ "$OPTION" = "$SEARCH_OPTION" ] && (echo $VALUE ; break)
    I=$(expr $I + 1)
  done
}

get_myoptions_entry ()
{ I=0
  SEARCH_OPTION=$(echo "$1" |tr -d "-")

  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    VALUE=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f -3)
    [ "$OPTION" = "$SEARCH_OPTION" ] && (echo $VALUE ; break)
    I=$(expr $I + 1)
  done
}

set_myoptions_value ()
{ AE=$(get_myoptions_array_element $1)
  VAL=$(get_myoptions_entry $1)
  NEWE="${VAL},$2"
  MYOPTIONS[$AE]=$NEWE
  export MYOPTIONS
}
With the associative array MYOPTIONS such functions can be reduced and simplified:
Code:
get_myoptions_value ()
{ cut -d "," -f 3 <<< ${MYOPTIONS[$1]}
}

set_myoptions_value ()
{ MYOPTIONS[$1]="$(cut -d "," -f -2 <<< ${MYOPTIONS[$1]}),$2"
}
Similarly, by using associative array instead of a list, for the SCRIPT_PARAMS one could do without the fiddly recall the contents of a variable whose name is in another variable and code would be shorter and simpler. I'll leave that as an exercise for the reader.

It is also possible to replace the case where all the values for the flags/options are parsed with a loop on the MYOPTIONS array wich in turn would require even less specific code for the options ... but you can't get rid of it all because at some point there will be some specific action to be taken for each flag/option.

by pan64 on Thu, 2016-09-22 04:28
I'm sorry, but I need to say the code you presented is inefficient. You need to eliminate $( .. ), because that will invoke a new shell and usually can be solved without that (which will need less resources, run faster...)

by louigi600 on Thu, 2016-09-22 04:47
Yes surely but the article is on minimizing code not making it more efficient. It's focus is on reducing the amount of code necessary to deal with input flags/parameters and configuration file creation.
I'll see if I can review the code so that it's not a bad example of bash coding ... can I edit my article ?

Help me find the contents of a variable whose name is in another variable ... I know what I did is a mess but I still can't think of an alternative.

by pan64 on Thu, 2016-09-22 06:09
Quote:
Help me find the contents of a variable whose name is in another variable
Code:
A=B
B=12
echo ${A}
echo ${!A}

by louigi600 on Thu, 2016-09-22 07:02
Quote:
Originally Posted by pan64 View Post
Code:
A=B
B=12
echo ${A}
echo ${!A}
Yes I guess I should have read better the bash man page ... No how do I edit the article to make it better ?

by pan64 on Thu, 2016-09-22 07:06
probably there is an edit button, but actually you can post an improved version too.

by louigi600 on Thu, 2016-09-22 07:13
Quote:
Originally Posted by pan64 View Post
probably there is an edit button, but actually you can post an improved version too.
For posts there is but I can't find it for article, I wanted to avoid rewriting it just make the current version better.
But if it's not possible I guess a rewrite will be scheduled.

by louigi600 on Thu, 2016-09-22 09:42
Quote:
Originally Posted by pan64 View Post
You need to eliminate $( .. ), because that will invoke a new shell ...
No $(command) and/or `command` is command substitution and does not invoke a shell unless the command is a shell.
It's the plain ( list ) that invokes a subshell to run the list.

Ok but still you have a point: there are some places where I can do without command substitution and I can declare som variables as integer and do without some of the ((expression)). It will be an exercise for me to write cleaner bash scripts

by MadeInGermany on Fri, 2017-05-05 15:59
The $( ) or ( ) is a sub shell.
That is another process, or at least it behaves like one.
An example should demonstrate this
Code:
x=1
echo $(x="2 $x"; echo "$x")
echo "$x"
( x="2 $x"; echo "$x" )
echo "$x"
Nothing is returned to the main shell.
My attempt for an efficient coding
Code:
get_myoptions_value ()
{ I=0
  #SEARCH_OPTION=$(echo "$1" |tr -d "-")
  SEARCH_OPTION=${1//-/}

  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    #OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    OPTION=${MYOPTIONS[$I]%%,*}" # the first comma-separated value

    #VALUE=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 4)
    VALUE=${MYOPTIONS[$I]##*,} # the last one (this might not be 4)
    #[ "$OPTION" = "$SEARCH_OPTION" ] && (echo $VALUE ; break)
    [ "$OPTION" = "$SEARCH_OPTION" ] && { echo $VALUE ; break ; }
    #I=$(expr $I + 1)
    (( I++ ))
  done
}
The { ;} is efficient because it stays in the current shell, just like a simple if-then-else.
BTW nothing speaks against
Code:
    if [ "$OPTION" = "$SEARCH_OPTION" ] ; then echo $VALUE ; break ; fi

by louigi600 on Sun, 2017-05-07 04:22
To my understanding, which may be wrong, $(command) forks the command so if it's a shell it will run a subshell if it's an executable it will execute the command obviously in another process in any case.
$(command) and `command` do the same thing I just prefer the newer $(command) syntax.
As I said before this not about optimizing code for performance but all about minimizing the amount of code necessary to deal with scripts with several options and configuration parameters.
Although I appreciate comments regarding attempting to get petter performance I do thing that they're missing the whole point of the article.

In any case a reedit or improved version is pending

MYOPTIONS is an associative array you don't need to walk the whole looking for the looking for the right element.
Code:
get_myoptions_value ()
{ I=0
  #SEARCH_OPTION=$(echo "$1" |tr -d "-")
  SEARCH_OPTION=${1//-/}

  while [ $I -lt ${#MYOPTIONS[*]} ]
  do
    #OPTION=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 1)
    OPTION=${MYOPTIONS[$I]%%,*}" # the first comma-separated value

    #VALUE=$(echo "${MYOPTIONS[$I]}" |cut -d "," -f 4)
    VALUE=${MYOPTIONS[$I]##*,} # the last one (this might not be 4)
    #[ "$OPTION" = "$SEARCH_OPTION" ] && (echo $VALUE ; break)
    [ "$OPTION" = "$SEARCH_OPTION" ] && { echo $VALUE ; break ; }
    #I=$(expr $I + 1)
    (( I++ ))
  done
}
That's not more efficient it's doing a ueless array walk and it's less readable, but yes instead of forking cut I could have removed the longest matching prefix ... that might go in the revised version.

by pan64 on Sun, 2017-05-07 04:31
for example:
Code:
eval $(echo dialog --title \"$TITLE\" --yesno  \" ${STRING[*]}\" 0 0 ) && return 1 || return 0
# can be simplified by:
dialog --title "$TITLE" --yesno  " ${STRING[*]}" 0 0
# you can put a return $? if you want at the end, but not required
in general you should eliminate eval, which will not only speed up the execution, but as you can see makes the code more readable (and also shorter)


  



All times are GMT -5. The time now is 01:40 PM.

Main Menu
Advertisement
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration