terminal: ways of executing binaries and scripts

carwe · 07-29-2012, 01:11 PM

I've been using Linux for some time now and I've been searching for a complete overview of this but not found any. There are so many ways of executing binaries and scripts from the terminal and I'm trying to write a complete guide/overview to get every detail right for myself. What I have below is the best I've come up with so far - could you tell me if it's right, state any errors, help me improve it?

Here it goes:

At first, two points to state:

Linux makes a distinction between a command and a path. A command is only typed as-is on the prompt, and will execute a built-in or will cause Linux to look for a corresponding binary or a script on the $PATH.
For Linux to interpret something as a path, it needs to contain at least one slash (/). E.g. in ./myScript, ./ can seem pretty redundant - it's there only to make Linux interpret it as a path rather than a command.

So, the options for executing a binary or a script:

Executing a binary 'binary':

Code:

$ binary          # when 'binary' is on the PATH, or is a built-in
$ ./binary        # when 'binary' is not on the path but in the current directory
$ /home/me/binary # when 'binary' is not on the PATH, and not in the current dir

Executing a script 'script':

The file will have to have execute permissions unless stated otherwise.

Code:

$ script        # execute a script that is on PATH. Will be executed in a new shell.
                # The interpreter to use is determined by the she-bang in the file.
$ ./script      # execute a script that is in the current dir. Otherwise as above.
$ /a/dir/script # when the script is not on the PATH and not in current dir. 
                # Otherwise as above.
$ . script      # execute a script in the current dir. Will be executed in the
                # current shell environment.
$ source script # equivalent to the above *1
$ sh script     # executes 'script' in a new shell *2 (the same goes for 'bash ...',
                # 'zsh ...' etc.). Execute permission not neccessary.

About she-bangs:

Scripts with a she-bang (e.g. #!/bin/sh) on the first line tells which interpreter to use.

This interpreter will be used when executed by ./script or using a command: 'script' (script must be on the PATH)
Using 'sh script' will ignore the she-bang and use, in this case, sh as the interpreter
Using '. script' or 'source' will ignore the she-bang and use the current interpreter (since . or source is equivalent to just executing each line of the script in the current shell)

Footnotes

*1: This is only almost true. In bash they are indeed the same command, but when using source, script will be searched for in PATH before the current dir. That's bash, but in POSIX-only shells, source doesn't work, but . does. So rather use the latter for portability.

*2: what actually happens is that we run the binary sh with 'script' as argument, which will make 'sh' execute 'script' in its new shell

theNbomr · 07-29-2012, 05:45 PM

You're in the right neighborhood, but still missing a couple of concepts.

There is a distinction between the name of the executable and the way a shell finds it. An executable can be specified with a fully qualified, unambiguous filespec. The path on which the executable is found is not left to interpretation and does not use the $PATH variable to find it.

In any other case, the $PATH variable is used as an ordered list of places to look for the specified executable.

The paths used in $PATH or in the filespec which specifies any file may be an absolute, or relative paths. A relative path does not have a leading '/', and is interpreted to mean 'relative to $PWD'. A path in $PATH (or anywhere else) does not have to contain a '/', and can be relative or absolute. Relative paths in $PATH are seldom used.

Examples (executable is '/some/absolute/dir/myBinary'):

Code:

PATH=/some/absolute/dir:some/relative/dir:.

#
# Execute myBinary by letting the shell find it
#
myBinary

#
# Go somewhere in the path to my binary
#
cd /some/absolute
dir/myBinary

#
# Go to the directory where myBinary lives
#
cd dir
#  or...
cd /some/absolute/dir

# invoke 'myBinary'
./myBinary

# or...  because '.' is in $PATH (which is generally not done)
myBinary

#
#  Finally, invoke 'myBinary' from anywhere, unambiguously
#
cd /any/other/directory/tree
/some/absolute/dir/myBinary

Note how paths given as arguments to cd can also be absolute or relative.

--- rod.

carwe · 07-30-2012, 03:51 AM

Thanks for the reply.

However I'm fully aware of everything you write so maybe you mis-understood some intent of mine in my original instructions.

I just wanted to state the different ways of invoking scripts and binaries in the three cases (1. file is in current dir, 2. file is in another dir, 3. file is on PATH) for completeness. It would be natural to interpret these commands as equivalent:

Code:

$ myBinary     #1
$ ./myBinary   #2

...but they are usually not because of #1 being interpreted as a command (binary/executable needs to be on PATH) and #2 as a path, since it contains at least one slash /.

That's the only reason I was so detailed about giving the different cases.

Is this making sense? If you still think I'm getting something wrong, I suggest you quote that and suggest a correct alternative, then I can understand exactly what you mean and other way around.

I'm still interested in any corrections or additions to my original post.

pixellany · 07-30-2012, 05:43 AM

Quote:

Executing a binary 'binary':

Code:

$ binary          # when 'binary' is on the PATH, or is a built-in
$ ./binary        # when 'binary' is not on the path but in the current directory
$ /home/me/binary # when 'binary' is not on the PATH, and not in the current dir

The second 2 forms work regardless of whether "binary" exists in the PATH.

In the first form, "binary" does not have to be in the PATH----it just has to be in the current directory. If binary IS in the PATH, then I think that one gets run first---which is typically why we need #2.

It is quite possible to have several versions of "binary" on the system---the different addressing methods would determine which one gets run.

theNbomr · 07-30-2012, 08:09 AM

The location of the target binary in the current working directory is irrelevant. There is no automatic search for binaries in $PWD unless it is so specified in $PATH. The '.' notation is not used differently than any other path specifier, therefore your examples:

Quote:

Code:

$ ./binary        # when 'binary' is not on the path but in the current directory
$ /home/me/binary # when 'binary' is not on the PATH, and not in the current dir

do not demonstrate anything distinct from each other. Each is simply an example of a fully qualified, unambiguous filespec.
--- rod.

carwe · 07-30-2012, 08:14 AM

Quote:

Originally Posted by theNbomr

The location of the target binary in the current working directory is irrelevant. There is no automatic search for binaries in $PWD unless it is so specified in $PATH. The '.' notation is not used differently than any other path specifier, therefore your examples:
do not demonstrate anything distinct from each other. Each is simply an example of a fully qualified, unambiguous filespec.
--- rod.

I know. But '$ ./binary' is kind of unexpected (for a novice), one would think '$ binary' works just as well, in order to execute a binary in the current dir. It doesn't however, since Linux needs at least one slash to interpret it as a path rather than a command. I wanted to make that clear, that's why I also wanted to include the example of how to execute a binary in the current dir.

theNbomr · 07-30-2012, 08:30 AM

I guess if you want us to guess what you think is unexpected or odd, then we are likely to see things differently. Do you want the facts or do you want an affirmation of your expectations? What makes the requirement to specify the current working directory 'unexpected'? The inconsistent behavior of MS-DOS, perhaps?
I still think your understanding is flawed, based on your insistence that 'Linux needs at least one slash to interpret it as a path rather than a command'. '.' and '..' are both paths. No slash in either of them.
--- rod.

carwe · 07-30-2012, 08:48 AM

Thanks for replys and the help on this.

Quote:

Originally Posted by theNbomr

I guess if you want us to guess what you think is unexpected or odd, then we are likely to see things differently.

I'm just motivating why I included both examples. This is not important, but since that was questioned I just wanted to give the motivation.

Quote:

Originally Posted by theNbomr

Do you want the facts or do you want an affirmation of your expectations? What makes the requirement to specify the current working directory 'unexpected'?

What is unexpected is subjective, as you say. But this makes it somewhat unexpected:

Code:

$ ls ./
$ ls              # equivalent to above
$ ls aSubDir/
$ ls ./aSubDir/   # equivalent to above
$ ./myBinary
$ myBinary        # not equivalent to above

But only of you don't already know about this. If you know about it, it's expected, as with everything.

Quote:

The inconsistent behavior of MS-DOS, perhaps?

Inconsistent behavior to MS-DOS would be a very good reason to clearly point out the difference, yes.

Quote:

I still think your understanding is flawed, based on your insistence that 'Linux needs at least one slash to interpret it as a path rather than a command'. '.' and '..' are both paths. No slash in either of them.
--- rod.

It might very well be that I still don't understand this throroughly, yes. 'Linux needs at least one slash to interpret it as a path rather than a command' is what I have been taught, yes. It might not be right, or it might not be the best way to express it. Instead of phrasing it like that, maybe it would be better just to say 'There is no automatic search for binaries in $PWD unless it is so specified in $PATH.', as you wrote in a post.

theNbomr · 07-30-2012, 09:12 AM

Quote:

Originally Posted by carwe

What is unexpected is subjective, as you say. But this makes it somewhat unexpected:

Code:

$ ls ./
$ ls              # equivalent to above
$ ls aSubDir/
$ ls ./aSubDir/   # equivalent to above
$ ./myBinary
$ myBinary        # not equivalent to above

The use of paths as arguments to other programs does not have to follow any particular rules.
Your example demonstrates this, but it is a distinct category from the use of paths to find executable programs.

--- rod.

David the H. · 07-30-2012, 01:53 PM

I don't think you have the difference between sh and bash/ksh/etc. quite clear either.

"sh" is your POSIX-compliant script interpreter. Any commands executed with sh will be run according to the requirements of the POSIX standard. The actual interpreter (executable) that your system uses for sh may be bash, or ksh, or dash, or zsh. That is configurable by the system operator. But whatever the interpreter, when it sees "sh", it will go into POSIX mode to execute the commands.

If the script run with sh has been written with syntax intended for bash, for example, a few different things could happen.

1) If a command conflicts with the posix specification, it will be rejected, and most likely spit out an error.

2) Some shells have modified the default behavior of certain commands that are also defined by POSIX. In this case the POSIX behavior will be used, which may be different from what the script expects.

3) If the command encountered is undefined by POSIX, then the response is left up to the actual shell used as the interpreter. In most cases this means that the shell will execute any syntax it recognizes, according to its own internal behavior, and error on anything it doesn't know how to handle. bash will continue to accept arrays, for example, even though they aren't POSIX, but dash will show an error, as it doesn't support them internally.

In short, sh guarantees that scripts written according to the POSIX standard will run as expected, at least syntactically. Whether non-POSIX scripts will run or not depends on the interpreter the system uses for sh.

carwe · 07-30-2012, 04:30 PM

Thanks for the points. No, this wasn't completely clear to me. I thought bash, dash, and sh were just different executables and that sh was the one following the POSIX standard. But as I understand you, if bash, dash, etc. encounters #!/bin/sh it will still be basn/dash/etc. that executes ths script but it will be done in "POSIX-mode".

So, then a script with a she-bang '#!/bin/sh' could be invoked in any of the ways: 'script', or state its path, or use ., source, or e.g. 'bash scrip' since the she-bang will be respected and whatever interpreter is used will use "POSIX mode"?

That would mean that invoking with . or 'source' does have an important difference to executing each line in the current shell: if the current shell is bash, executing the lines of the file manually (say, copy-paste) wouldn't use "POSIX mode", but all the others would, if there's a she-bang?

What if the she-bang reads "#!/bin/zsh"? What will '. script' or 'bash script' do then?

chrism01 · 07-30-2012, 11:46 PM

Re paths:

1. if a path starts with '/', then it is an Absolute Path eg
home/me/prog
/bin/dd

2. if it starts with a '.', then it is a Relative Path eg

./prog
../otherdir/prog

NB: this is general ie applies to any file, not just programs.

If you only 'name' the file with no path at all, it must be in your $PATH somewhere, which often catches newbies, as '.' ie pwd ie your current path (aka cwd) is NOT normally in your path, and neither is /home/you, so trying to call it direct from /home/you without a path will not work.

theNbomr · 07-31-2012, 08:40 AM

Quote:

Originally Posted by chrism01

2. if it starts with a '.', then it is a Relative Path eg

While that is quite correct, I would add that that any './' prefix is generally redundant. I would use the definition that a relative path is one that does not have a leading '/' prefix.

It is probably worth noting that many shells interpret the '~' shorthand notation as equivalent to $HOME, or the user's home directory. So, it is often used as in:

Code:

~/some/directory/someExecutable

While this seems to violate my statement above, it actually expands, at some point, to something like:

Code:

 /home/joeuser/some/directory/someExecutable

--- rod.

David the H. · 07-31-2012, 12:53 PM

Quote:

Originally Posted by carwe

That would mean that invoking with . or 'source' does have an important difference to executing each line in the current shell: if the current shell is bash, executing the lines of the file manually (say, copy-paste) wouldn't use "POSIX mode", but all the others would, if there's a she-bang?

What if the she-bang reads "#!/bin/zsh"? What will '. script' or 'bash script' do then?

The first character in the shebang is "#", which makes it a comment. If the file isn't being executed directly, the shebang is ignored entirely and the shell doing the interpreting attempts to run everything according to its own syntax.

carwe · 07-31-2012, 01:44 PM

Quote:

Originally Posted by David the H.

The first character in the shebang is "#", which makes it a comment. If the file isn't being executed directly, the shebang is ignored entirely and the shell doing the interpreting attempts to run everything according to its own syntax.

Right, but what's your definition of "executed directly" in you answer above? Any of: argument to 'sh'/'bash'/etc., '.', './script'? Exactly when is the she-bang not interpreted as a comment and hence not ignored?