LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Blogs > rtmistler
User Name
Password

Notices

Rate this Entry

Bash Scripting for Dummies and Geniuses

Posted 11-20-2013 at 09:18 AM by rtmistler
Updated 06-17-2014 at 01:25 PM by rtmistler

One only need look at LinuxQuestions.org to notice the great amount of script (and code) questions and varieties of comments to realize that we all have varying opinions about bash programming.

Many questions are very complicated, as are the resulting answers and script recommendations. I've noticed that two fundamental points are invariably overlooked by the questioners, and many of the comments:

Quote:
"All programming is the use of simple operations to solve complex problems through iteration or combination of those simple steps."
Quote:
"Whatever you can type on a command line, you can write in a script."
Fundamentals for bash scripting should not be forgotten by script authors. I preach to programmers, "It's not about being cool and writing something rapidly and unique that bedazzles readers; it's about writing good solid code so that you, and others can use it, debug it, benefit by it, maintain it, and re-use it."

Intended Audience

This is intended for the very many persons who try to write scripts and end up with very difficult to debug problems that halt their progress so that they have to ask questions. It is also intended for experienced programmers who sometimes forget that modularizing code, stepping back from a problem, and drawing upon the vast resources within their own experience will really help them to avoid similar periods of frustration.

My Fundamental Bash Toolkit

And these are not mine, they're everyone's.
  1. set -xv
  2. #
  3. echo
  4. $?
  5. var=
  6. functions
  7. command line
  8. test operators

Likely you know these. So use them. For those who have some passing familiarity or wish to view my perspectives on them, please read on:

set -xv

This will enable debug so that you'll literally see the line by line passage of your script in stdout. Use it on a simple script, here's one:
Code:
#!/bin/bash
# Very simple sample script to perform a grep command
#set -xv
grep $1 $2
exit 0
Here's the output with no arguments:
Code:
./sample.sh 

grep $1 $2
+ grep
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.

exit 0
+ exit 0
This shows me the grep error; I would see it anyway, but it also reminds me that I coded a script to use arguments, and that I neglected to use arguments. This method is not always the best, as the script grows and gets more complex, you may be inundated with this debug and choose to employ other methods, but it is a helpful tool to detect your mistakes at a macro level. Here it reminds me to check my argument list for the count of arguments as well as the syntax of those arguments.

Don't Forget That You Can Revert: set +xv

Another cool thing you can do with this is to enable/disable verbose debug.

Code:
echo "123abc" > a.txt
set -xv
cp -f a.txt b.txt
mv b.txt c.txt
grep a "*.txt"
set +xv
<script-continues>
This way if I've already debugged certain parts of my script, I can selectively choose to enable verbose output and therefore not see tons of data for things which I know have been tested and concentrate on a section I can't quite get correct yet.

#

The comment. This is useful because it allows you to add comments to describe your actions in your script, and it also allows you to add debug, and then later remove the debug from the execution path, but not erase the debug in case you can benefit from using it in the future. A method to comment out a block of code could use the if-statement:
Code:
if [ ]; then
    code which will not be executed;
fi
The point here is to not just get something working and then erase it, but instead to comment debug steps out so that you can resurrect them in case you need to modify your script in the future.

echo

This is also a well known script capability, as are the input/output redirectors ('>', '&', '1', '2). You can echo information to a file, it can be diagnostic data showing the scripts progression, and you can echo to stdout to see output as you run the script. Using this capability will tell you information like; what variables defined as, outcomes of system calls, other useful information.

Whenever I code a script to do something and I'm unsure about what's been done to a variable, I use echo to validate what things look like to aid me in debugging. For example, I wrote a script to take in arguments and needed to use those arguments to be modified; I created local variables with the arguments, but also with added information.
Code:
   A=$1
   B=$1
#   echo "Un-modified arguments: A=$A   B=$B" > setup-disk.log;
   A+=1
   B+=2
#   echo "Modified arguments: A=$A   B=$B" > setup-disk.log;
   sudo umount $A>&setup-disk.log
   sudo umount $B>&setup-disk.log
Yes, it is simple, the point is that people assume because it's simple it neither needs comments nor debug, but when stuff doesn't work, those same people amazingly forget the fundamental things and start searching for answers to a different problem; such as the part of the script which will use $A and $B and wonder why that part doesn't work, while forgetting something simple like they didn't prepare their variables correctly.

Additional Word on Redirectors

Myself, I constantly have to search and try things before I know the correct answer. I certainly suggest that you also look things up and experiment to attain the results which work best for you. These are the most common forms of "output" redirection which I use:

Code:
# > - direct stdout to a file, always create the file as new

echo "This is the start of my script" > /home/myuser/logs/startup-script.log;

# >> - direct stdout to a file, either create or append

echo "This is an additional log entry in my file" >> /home/myuser/logs/startup-script.log;

# 2>&1 - THIS I always have to look up.  Redirect stderr to stdout.  And FURTHER I
# always have to look up or experiment to ensure I got it right.  I really should include
# this in a standard script I always reference so I can remember it ...

grep "abc" "*.c" >> /home/myuser/logs/search.log 2>&1

# The resulting information here would be placed into the search.log file whether it be
# stdout, stderr, or both.
A main reason why I always have to look up the 2>&1 notation is because if I were to do something like that grep statement; me wondering what the output would be is a case where I'd check the return from grep $?, or assign the return to a variable and check that variable. Therefore whatever goes to stderr might be irrelevant; because using set -xv and echoing out the results likely in that case gets me sufficient information to discern how better to code that particular line. I'm not averse to multiple techniques; however my point here is the most powerful ones to me are ">>" and ">".


$?

This is the return value from a function or system call. Testing or echoing this is highly recommended. Part of the comment in the manual page for grep where it discusses EXIT STATUS.
Quote:
Normally, the exit status is 0 if selected lines are found and 1 otherwise. But the exit status is 2 if an error occurred
This implies that successful grep calls will result in exit status of 0; however one can call grep successfully, find nothing and receive an exit status of 1. In the case of error inputs to grep, you would receive an exit status greater than 1. Note that the paragraph goes further (I didn't quote it all) to cite that there are some POSIX recommendations which state the return be greater than 1, therefore an exact check for 2 may not cover all implementations.

The summary here is "For any programming environment, if there are return values, they likely are designed to be helpful and you are better off checking them, versus ignoring them."

var=

Assignment of a variable. Also linked to this is the value or content in that variable. It is pretty fundamental; however many times I have avoided this fundamental step because I've written a single line which does a combined set of steps all at once. And then had to go back, break up a combined statement and do fundamental steps like:
  1. Set a result to a variable
  2. Echo that variable to make sure I got step 1 correct
  3. Use that variable in my next step
  4. Continue this process until I had accomplished the original intention of my combined command line.
It does little good to code a killer line if the sole purpose of it is to be seen as this great killer line of code. If it can be done just as efficiently with simple instructions then I argue that it is more maintainable for you and others if it has been done incrementally with fundamental steps versus a complicated one-line which even you; the author, may forget over time how you achieved it.

functions

You can have functions in most (if not all) programming languages, you can also have them in bash and you can pass arguments as well as return results using functions. This has the same results in a script that you get from a typical program:
  1. It modularizes functionality
  2. It makes code re-usable
  3. It increases maintainability of the code
Here's an example of the main portion of a script, followed by one of the functions called in the script:
Code:
# Main section

    # Set up to run the script, make sure the starting conditions are ready
    setup $@

    # Parse arguments
    # Format is: setup-disk.sh [device] <-f|-norfs|nil>
    #   [device] - mandatory argument, device for the compact flash
    #   <> - optional arguments, mutually exclusive
    #   -f - "format disk", install grub, kernel, and root file system (RFS)
    #   -norfs - "no RFS", assume format, grub, and RFS O.K. just copy kernel
    #   nil - no second argument; assume format and grub O.K., copy kernel and RFS
    parse_arguments $@

    # Allow the user to view what they've chosen and give them a chance to cancel the script or continue
    present_options $@

    # If we're going to format the disk, then we need to set it up fully by:
    #  - deleting all existing partitions
    #  - creating the partitions for this product
    #  - installing grub
    if [ $FORMAT -eq 1 ]; then
        echo "Deleting all disk partitions . . ."
        delete_partitions $1
        echo "Creating product partitions . . ."
        create_partitions $1
        echo "Creating product file systems . . ."
        create_file_systems $1
        echo "Installing grub . . ."
        install_grub $1
    fi

    # We always copy the kernel to the disk
    echo "Copying kernel . . ."
    copy_kernel $1

    # If they chose to install the RFS, then do that
    if [ $RFS == 1 ]; then
        echo "Copying RFS . . ."
        copy_rfs $1
    fi

    echo "Done"

    exit 0
As you can see, the main part of this script calls a series of functions. As coded, it does not check the return value of any of the functions; at the time I wrote that script I chose to exit the script entirely from within the functions, for instance from within present_options. Next shown is that function where you can see that it either returns to the main part of the script a return value, or it causes the script to exit completely. This could've been coded differently by returning always to the main and then checking in the main for the desire to exit or continue. The difference being the use of exit to exit the script and return to return back to the main part of the script.
Code:
# Present Options
# Inform the user of the script's actions and
# what it thinks they've chosen to do and then
# give them an option to proceed or cancel.
present_options()
{
    echo ""
    echo ""
    echo "This script prepares a disk with a release."
    echo "Options are to:"
    echo "     - Format and set up the entire disk (use -f)"
    echo "     - Copy the kernel only (use -norfs)"
    echo "     - Copy the kernel and Root File System (RFS) (default)"
    echo ""
    echo "Choices cannot be combined.  Format (-f) option assumes full disk overwrite."
    echo ""
    echo "Options chosen:"
    echo "     Device: $1"
    echo "       Kernel will be copied"
    if [ $FORMAT -eq 1 ]; then
        echo "       Disk will be formatted"
    else
        echo "       Disk will NOT be formatted"
        if [ $RFS -eq 1 ]; then
            echo "       Root File System will be copied"
        else
            echo "       Root File System will NOT be copied"
        fi
    fi
    echo ""
    echo "Do you wish to proceed? (y/[n]) "
    read ANSWER
    if [ -z $ANSWER ]; then
        echo "Aborting(1) . . ."
        echo ""
        exit -1
    elif [ $ANSWER == "Y" ]; then
        echo "Proceeding with disk setup . . ."
        echo ""
        return 0
    elif [ $ANSWER == "y" ]; then
        echo "Proceeding with disk setup . . ."
        echo ""
        return 0
    else
        echo "Aborting . . ."
        echo ""
        exit -1
    fi
}
To reiterate; the use of functions is just good programming practice. In my case here there was no re-use of my functions within the script; however years later I did make use of this script as well as it's "sibling", a script known as "copy-rfs.sh". The script partly shown above is called "setup-disk.sh" and it originally made a full disk for an embedded distribution. A main part of it, to copy the kernel and put the RFS in place was able to be used on another project, in unmodified form because the argument for formatting the disk was an option; and the circumstances where I had a /boot partition and a separate partition for my RFS was similar such that the script was re-usable in it's entirety. The copy-rfs.sh script was also invaluable because that takes my development disk and copies my RFS entirely into a file, resizes that file to be the minimum size required for the file system; allowing me to place that file system on any product release disk. Having developed those scripts about 4 years prior to reusing them was very beneficial. My next step was to make two "setup-disk-<product-name>.sh" scripts where my original was now named to match that very old project name and an edited new one; which does not do any disk formatting, was make to match the new project. The differences were:
  1. Slight modifications to functions parse_arguments and present_options
  2. Omission of the variable $FORMAT and all associated functions called based on that variable's use

command line

As stated above, command line is "bash" and therefore what you have in that script should run from the command line, given similar environment and variables that your script has set up. Therefore if you're writing a script and have tried a variety of debugging methods and can't quite understand why a certain system call always fails; how about trying that call on the command line and then coding solely that call in a brief script to validate that you know how to set up to provide that system call with the appropriate arguments from within your script. This way when you are deep within the execution path of a script you will have set up properly to perform that system call and it will be one less item to debug in the totality of your script.

test operators

Here is a list of some useful test operators compiled from various sources and docs. I find it helpful to refer to it when I'm unsure of the exact test I need, or even if such a test condition exists. This is not all encompassing.

Files

-e file exists
-f file is a regular file
-s file is not empty
-d file is a directory
-h,-L file is a symbolic link
-r file has read permission
-w file has write permission
-x file has execute permission
-O you are owner of file
-G group-id matches yours
-N file modified since it was last read

Numbers

-eq is equal to
-ne is not equal to
-gt is greater than
-ge is greater than or equal to
-lt is less than
-le is less than or equal to

Strings

= is equal to
== is equal to
!= is not equal to
-n string is not empty
-z string is empty

You Learn New Things Everyday

In the course of my continued participation in LQ, I had the occasion to offer an answer to a thread question. This actually led me to research and find the subject of BASH Internal Variables. Specifically for This LQ Thread I did my search and discovered not only the $PIPESTATUS internal variable, but details about it which were important. I should add that the initial discovery of PIPESTATUS was really encountered via a stack exchange question, to give due credit. Not in the direct answer, but rather in the following comments, people recommended PIPESTATUS. Eventually this led to the knowledge that this status applied immediately and the actions of checking that variable would then change that variable. Therefore one would have to use the full list/array result at once for example by assigning this to a local variable and then parsing that local variable.

In the meantime, perusing the Internal Variables for BASH is a great idea. Yet one more item to add to one's reference toolkit.

Cross Referencing Back

User habitual has created a very good list of useful links. They were kind enough to add this blog entry to their list; and I recommend you visit their list of links to see what other helpful information you can obtain. There are clearly quite a few links for BASH as well as Linux in general.

Conclusions

Thank you for taking the time to read my comments. Similar to many others, I may tend to draw out points a bit much. The real intention here is to provide the insights that:
  1. Simple and direct are better
  2. On very complicated coding problems, the one thing that always re-centers me is to revisit my whole method and break it down into steps. It really does help.

Happy Coding and Scripting!
Posted in Bash Scripting
Views 222 Comments 3
« Prev     Main     Next »
Total Comments 3

Comments

  1. Old Comment
    Great article.
    Posted 11-21-2013 at 10:24 AM by vmccord vmccord is offline
  2. Old Comment
    Thank you for this post, both delightfully plain and very insightful.
    Posted 11-22-2013 at 05:12 PM by xri xri is offline
  3. Old Comment
    Added to my "Reference List"
    Posted 01-30-2014 at 03:34 PM by Habitual Habitual is offline
 

  



All times are GMT -5. The time now is 03:40 AM.

Main Menu
Advertisement

My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration