Bash Scripting for Dummies and Geniuses

One only need look at LinuxQuestions.org to notice the great amount of script (and code) questions and varieties of comments to realize that we all have varying opinions about bash programming.

Many questions are very complicated, as are the resulting answers and script recommendations. I've noticed that two fundamental points are invariably overlooked by the questioners, and many of the comments:

Fundamentals for bash scripting should not be forgotten by script authors. I preach to programmers, "It's not about being cool and writing something rapidly and unique that bedazzles readers; it's about writing good solid code so that you, and others can use it, debug it, benefit by it, maintain it, and re-use it."

Intended Audience

This is intended for the very many persons who try to write scripts and end up with very difficult to debug problems that halt their progress so that they have to ask questions. It is also intended for experienced programmers who sometimes forget that modularizing code, stepping back from a problem, and drawing upon the vast resources within their own experience will really help them to avoid similar periods of frustration.


Additional Reading

The following links are general guidelines for BASH. A beginners guide and the advanced BASH scripting guide. Both very useful to review:

Bash Guide for Beginners
Advanced BASH Scripting Guide
Online Regex Checker tool
Online shellcheck tool
Shellcheck code on GitHub

My Fundamental Bash Toolkit

And these are not mine, they're everyone's.

1. set -xv
2. #
3. echo
4. $?
5. var=
6. functions
7. command line
8. test operators

Likely you know these. So use them. For those who have some passing familiarity or wish to view my perspectives on them, please read on:

set -xv

This will enable debug so that you'll literally see the line by line passage of your script in stdout. Use it on a simple script, here's one:Here's the output with no arguments:This shows me the grep error; I would see it anyway, but it also reminds me that I coded a script to use arguments, and that I neglected to use arguments. This method is not always the best, as the script grows and gets more complex, you may be inundated with this debug and choose to employ other methods, but it is a helpful tool to detect your mistakes at a macro level. Here it reminds me to check my argument list for the count of arguments as well as the syntax of those arguments.

Don't Forget That You Can Revert: set +xv

Another cool thing you can do with this is to enable/disable verbose debug.

This way if I've already debugged certain parts of my script, I can selectively choose to enable verbose output and therefore not see tons of data for things which I know have been tested and concentrate on a section I can't quite get correct yet.

#

The comment. This is useful because it allows you to add comments to describe your actions in your script, and it also allows you to add debug, and then later remove the debug from the execution path, but not erase the debug in case you can benefit from using it in the future. A method to comment out a block of code could use the if-statement:The point here is to not just get something working and then erase it, but instead to comment debug steps out so that you can resurrect them in case you need to modify your script in the future.

echo

This is also a well known script capability, as are the input/output redirectors ('>', '&', '1', '2). You can echo information to a file, it can be diagnostic data showing the scripts progression, and you can echo to stdout to see output as you run the script. Using this capability will tell you information like; what variables defined as, outcomes of system calls, other useful information.

Whenever I code a script to do something and I'm unsure about what's been done to a variable, I use echo to validate what things look like to aid me in debugging. For example, I wrote a script to take in arguments and needed to use those arguments to be modified; I created local variables with the arguments, but also with added information.Yes, it is simple, the point is that people assume because it's simple it neither needs comments nor debug, but when stuff doesn't work, those same people amazingly forget the fundamental things and start searching for answers to a different problem; such as the part of the script which will use $A and $B and wonder why that part doesn't work, while forgetting something simple like they didn't prepare their variables correctly.

Additional Word on Redirectors

Myself, I constantly have to search and try things before I know the correct answer. I certainly suggest that you also look things up and experiment to attain the results which work best for you. These are the most common forms of "output" redirection which I use:

A main reason why I always have to look up the 2>&1 notation is because if I were to do something like that grep statement; me wondering what the output would be is a case where I'd check the return from grep $?, or assign the return to a variable and check that variable. Therefore whatever goes to stderr might be irrelevant; because using set -xv and echoing out the results likely in that case gets me sufficient information to discern how better to code that particular line. I'm not averse to multiple techniques; however my point here is the most powerful ones to me are ">>" and ">".


$?

This is the return value from a function or system call. Testing or echoing this is highly recommended. Part of the comment in the manual page for grep where it discusses EXIT STATUS.This implies that successful grep calls will result in exit status of 0; however one can call grep successfully, find nothing and receive an exit status of 1. In the case of error inputs to grep, you would receive an exit status greater than 1. Note that the paragraph goes further (I didn't quote it all) to cite that there are some POSIX recommendations which state the return be greater than 1, therefore an exact check for 2 may not cover all implementations.

The summary here is "For any programming environment, if there are return values, they likely are designed to be helpful and you are better off checking them, versus ignoring them."

var=

Assignment of a variable. Also linked to this is the value or content in that variable. It is pretty fundamental; however many times I have avoided this fundamental step because I've written a single line which does a combined set of steps all at once. And then had to go back, break up a combined statement and do fundamental steps like:

1. Set a result to a variable
2. Echo that variable to make sure I got step 1 correct
3. Use that variable in my next step
4. Continue this process until I had accomplished the original intention of my combined command line.

It does little good to code a killer line if the sole purpose of it is to be seen as this great killer line of code. If it can be done just as efficiently with simple instructions then I argue that it is more maintainable for you and others if it has been done incrementally with fundamental steps versus a complicated one-line which even you; the author, may forget over time how you achieved it.

functions

You can have functions in most (if not all) programming languages, you can also have them in bash and you can pass arguments as well as return results using functions. This has the same results in a script that you get from a typical program:

1. It modularizes functionality
2. It makes code re-usable
3. It increases maintainability of the code

Here's an example of the main portion of a script, followed by one of the functions called in the script:As you can see, the main part of this script calls a series of functions. As coded, it does not check the return value of any of the functions; at the time I wrote that script I chose to exit the script entirely from within the functions, for instance from within present_options. Next shown is that function where you can see that it either returns to the main part of the script a return value, or it causes the script to exit completely. This could've been coded differently by returning always to the main and then checking in the main for the desire to exit or continue. The difference being the use of exit to exit the script and return to return back to the main part of the script.To reiterate; the use of functions is just good programming practice. In my case here there was no re-use of my functions within the script; however years later I did make use of this script as well as it's "sibling", a script known as "copy-rfs.sh". The script partly shown above is called "setup-disk.sh" and it originally made a full disk for an embedded distribution. A main part of it, to copy the kernel and put the RFS in place was able to be used on another project, in unmodified form because the argument for formatting the disk was an option; and the circumstances where I had a /boot partition and a separate partition for my RFS was similar such that the script was re-usable in it's entirety. The copy-rfs.sh script was also invaluable because that takes my development disk and copies my RFS entirely into a file, resizes that file to be the minimum size required for the file system; allowing me to place that file system on any product release disk. Having developed those scripts about 4 years prior to reusing them was very beneficial. My next step was to make two "setup-disk-<product-name>.sh" scripts where my original was now named to match that very old project name and an edited new one; which does not do any disk formatting, was make to match the new project. The differences were:

1. Slight modifications to functions parse_arguments and present_options
2. Omission of the variable $FORMAT and all associated functions called based on that variable's use

command line

As stated above, command line is "bash" and therefore what you have in that script should run from the command line, given similar environment and variables that your script has set up. Therefore if you're writing a script and have tried a variety of debugging methods and can't quite understand why a certain system call always fails; how about trying that call on the command line and then coding solely that call in a brief script to validate that you know how to set up to provide that system call with the appropriate arguments from within your script. This way when you are deep within the execution path of a script you will have set up properly to perform that system call and it will be one less item to debug in the totality of your script.

test operators

Here is a list of some useful test operators compiled from various sources and docs. I find it helpful to refer to it when I'm unsure of the exact test I need, or even if such a test condition exists. This is not all encompassing.

Files

-e file exists
-f file is a regular file
-s file is not empty
-d file is a directory
-h,-L file is a symbolic link
-r file has read permission
-w file has write permission
-x file has execute permission
-O you are owner of file
-G group-id matches yours
-N file modified since it was last read

Numbers

-eq is equal to
-ne is not equal to
-gt is greater than
-ge is greater than or equal to
-lt is less than
-le is less than or equal to

Strings

= is equal to
== is equal to
!= is not equal to
-n string is not empty
-z string is empty

You Learn New Things Everyday

In the course of my continued participation in LQ, I had the occasion to offer an answer to a thread question. This actually led me to research and find the subject of BASH Internal Variables. Specifically for This LQ Thread I did my search and discovered not only the $PIPESTATUS internal variable, but details about it which were important. I should add that the initial discovery of PIPESTATUS was really encountered via a stack exchange question, to give due credit. Not in the direct answer, but rather in the following comments, people recommended PIPESTATUS. Eventually this led to the knowledge that this status applied immediately and the actions of checking that variable would then change that variable. Therefore one would have to use the full list/array result at once for example by assigning this to a local variable and then parsing that local variable.

In the meantime, perusing the Internal Variables for BASH is a great idea. Yet one more item to add to one's reference toolkit.

Cross Referencing Back

User Habitual has created a very good list of useful links. They were kind enough to add this blog entry to their list; and I recommend you visit their list of links to see what other helpful information you can obtain. There are clearly quite a few links for BASH as well as Linux in general.

Conclusions

Thank you for taking the time to read my comments. Similar to many others, I may tend to draw out points a bit much. The real intention here is to provide the insights that:

1. Simple and direct are better
2. On very complicated coding problems, the one thing that always re-centers me is to revisit my whole method and break it down into steps. It really does help.

Happy Coding and Scripting!