How to run shell scripts wrapped in Python.

narnie · 07-11-2010, 01:27 AM

Hello,

I'm a new programmer (a life-long desire, but at 40 just getting to it) and cutting my teeth with Python. I just found the popen2 module today. I was so excited because I could do this:

Code:

>>> from popen2 import popen3
>>> cmd ="""TEXT="Hello, World!"
... echo $TEXT"""
>>> std_out, std_in, std_err = popen3(cmd)
>>> stdout = std_out.read()
>>> stderr = std_err.read()
>>> print stdout
Hello, World!

>>> if not stderr:
...     stderr = False
...     
>>> print(stderr)
False

Unfortunately, I also get this:

Quote:

/usr/lib/pymodules/python2.6/IPython/Extensions/ipy_completers.py:119: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module.

I started to explore the subprocess module, and found that it can be used, and is funner to do so.

You use Popen and PIPE from the module.

I have written a function that does the grunt-work for you, if you just want to use it, but in the source, you can see how it is implemented.

I'm sure more experienced Python-ers can do better than me, but this is what I cooked up in my newbie-ness. It wouldn't be too hard to abstract further and turn it into a class, and I'll probably do so just to get some more class experience and it could make it easier to handle the outputs rather than spitting out a tuple. Anyhoo, here 'tis:

Code:

def runInShell (cmd, input=None):
    '''Runs shell commands
    
    syntax:
    
        runInShell(cmd[, input])
    
    This function takes cmd, which is a command line or command lines to be 
    run in a bash shell.
    
    The argument input can also be specified to pass info via stdin to the 
    shell.

    The function returns a tuple which is output from standard output, 
    standard error, any standard input passed, and the return code giving:
    
        (stdout, stderr, stdin, returncode)


    examples:
    
        >>> text = 'Hello World!'
        >>> cmd = \'\'\'TEXT=`cat </dev/stdin`
        ... echo $TEXT\'\'\'
        >>> (stdout, stderr, stdin, returncode) = runInShell(cmd, text)
        >>> print stdout
        Hello World!
        >>> print stderr
        None
        >>> print stdin
        Hello World!
        >>> print returncode 
        0
        
        >>> cmd = \'\'\'echo "Hello"
        echos "World"\'\'\'
        >>> stdout, stderr, stdin, returncode = runInShell(cmd)
        >>> print stdout 
        Hello

        >>> print stderr 
        /bin/sh: line 1: echos: command not found

        >>> print stdin 
        None
        >>> print returncode 
        127
    '''
    
    from subprocess import Popen, PIPE
    
    shell = Popen(cmd, shell=True, executable='/bin/bash', stdin=PIPE, stdout=PIPE, stderr=PIPE)

    stdout, stderr = shell.communicate(input)

    if not stdout:
        stdout = None

    stdin = input
    
    if not stderr:
        stderr = None

    return stdout, stderr, stdin, shell.returncode

If you want to use another shell, substitute that shell for /bin/bash (e.g., /bin/sh for the shell portability purists out there).

It is easy to modify what is returned as well. Don't like the tuple? Then just change it to return or even print stdout if you like.

I just hope this helps someone understand quickly what it took me a few hours to play around with and learn, but hey, I enjoyed the experience.

Yours,
Narnie

narnie · 07-11-2010, 01:49 AM

In rereading the above, I see that it is in fact running it in /bin/sh. Not sure why because the executable variable is set as '/bin/bash' and according to the documentation for subprocess, that should take care of it.

I'll post back if I find a solution, as this will be a big problem for me as I often do c-style math calls as in $(( x++ )) and that won't work in /bin/sh (I only use linux, so don't have to worry about shell portability myself).

In the meantime if anyone who knows why it is doing this, I'd love to hear from you.

Yours,
Narnie

bgeddy · 07-11-2010, 01:18 PM

Yes this behavoir is odd. Looking a the module subprocess.py it has this line:

Code:

if shell:
        args = ["/bin/sh", "-c"] + args

which will put "/bin/sh -c" in front of the argument list if shell is set to True. Effectively this means /bin/sh is ran whenever you have shell=True as can be seen by running this addition to your script:-

Code:

if __name__=="__main__":
    retval=runInShell("echo $BASH")
    print retval

If you really want bash commands to run you could wrap them up in a bash script such as this:-

Code:

#!/bin/bash
echo $BASH

if you save that to bash.sh and run :

Code:

if __name__=="__main__":
    retval=runInShell("./bash.sh")

you'll see that bash is obviously being forced to run.

bgeddy · 07-11-2010, 05:58 PM

A further idea - sticking with your module why not just force bash to be ran like so:

Code:

     print runInShell("/bin/bash -c set")

you'll see from the inbuilt set response that bash is being ran as bash and not sh. It's easy to amend your module to tag this to the front of any bash commands needing running. I'm not familiar with the subprocess module but I must admit I find the behavior confusing with shell commands when an executable is set to /bin/bash as in your code. One would expect the command arguments to be passed to bash running as bash and not running as the sh link enforces.
Well spotted anyway. Good luck with your python scripts.

narnie · 07-12-2010, 12:43 AM

Quote:

Originally Posted by bgeddy

Yes this behavoir is odd. Looking a the module subprocess.py it has this line:

Code:

if shell:
        args = ["/bin/sh", "-c"] + args

which will put "/bin/sh -c" in front of the argument list if shell is set to True. Effectively this means /bin/sh is ran whenever you have shell=True as can be seen by running this addition to your script:-

Code:

if __name__=="__main__":
    retval=runInShell("echo $BASH")
    print retval

If you really want bash commands to run you could wrap them up in a bash script such as this:-

Code:

#!/bin/bash
echo $BASH

if you save that to bash.sh and run :

Code:

if __name__=="__main__":
    retval=runInShell("./bash.sh")

you'll see that bash is obviously being forced to run.

Brilliant piece of deduction looking at the "source" of the problem.

Sadly, prefacing with crunch-bang didn't seem to work in my trials of the code:

Code:

>>> cmd = '''#!/bin/bash
... echo $BASH'''
>>> runInShell (cmd)
('/bin/sh\n', None, None, 0)
>>>

I'm also afraid this doesn't work, and I see that bash does some strange things from the command line as well requiring quoting of what is after the -c argment:

Code:

>>> cmd = '''/bin/bash -c \
... echo $BASH'''
>>> runInShell (cmd)
('\n', None, None, 0)

Amazingly, when properly quoted though, it still shows it is in /bin/sh, though:

Code:

>>> cmd = '/bin/bash -c "echo \\"Hello, World!\nI\'m in:\n$BASH\\""'
>>> print runInShell (cmd)[0]
Hello, World!
I'm in:
/bin/sh

Somehow, even with run as /bin/bash -c it is running in /bin/sh. How looly is that?

On a proper command line, it works:

Code:

narnie@laptop $ sh
$ /bin/bash -c 'echo "Hello, World! I am in $BASH"'
Hello, World! I am in /bin/bash
$

Something quite strange is going on here.

Puzzled,
Narnie

narnie · 07-12-2010, 01:37 AM

Quote:

Originally Posted by bgeddy

Yes this behavoir is odd. Looking a the module subprocess.py it has this line:

Code:

if shell:
        args = ["/bin/sh", "-c"] + args

which will put "/bin/sh -c" in front of the argument list if shell is set to True. Effectively this means /bin/sh is ran whenever you have shell=True as can be seen by running this addition to your script:-

Code:

if __name__=="__main__":
    retval=runInShell("echo $BASH")
    print retval

If you really want bash commands to run you could wrap them up in a bash script such as this:-

Code:

#!/bin/bash
echo $BASH

if you save that to bash.sh and run :

Code:

if __name__=="__main__":
    retval=runInShell("./bash.sh")

you'll see that bash is obviously being forced to run.

I'm so new to Python and programming in general, I didn't think I could even begin to tackle the source, so checking subprocess.py didn't even occur to me.

After looking a bit at what it was doing, I believe I have the fix for the bug in the source.

Original Source (as bgeddy noted above) has this plus the few more lines to note:

Code:

            if shell:
                args = ["/bin/sh", "-c"] + args

            if executable is None:
                executable = args[0]

We want to change it to:

Code:

            if shell:
                if executable is None:
                    args = ["/bin/sh", "-c"] + args
                    executable = args[0]
                else:
                    args = [executable, "-c"] + args

#            if executable is None:
#                executable = args[0]

Or even just delete out the last two lines.

Rerunning everything gives:

Code:

>>> print runInShell ('echos')
--> print(runInShell ('echos'))
(None, '/bin/bash: echos: command not found\n', None, 127)
>>>

As it should noting that it is a /bin/bash error.

If I modify the code for runInShell to remove the executable parameter, then as expected, I get this:

Code:

>>> print runInShell ('echos')
--> print(runInShell ('echos'))
(None, '/bin/sh: echos: not found\n', None, 127)
>>>

Showing a /bin/sh error.

I love this stuff!!! I'M GOING TO GIVE MY FIRST EVER BUG FIX FOR SOURCE CODE!!! I just can't believe it !!!! (now if I only new how to do a patch and submit it, it would be even better. Baby steps. Baby steps).

Python is brilliant and is really making learning programming great fun.

Cheerio,
Narnie

narnie · 07-12-2010, 02:04 AM

Modified my runInShell code (which I've renamed to shellCommands because I use ipython and I like to be able to hit ru-tab to get %run [whatever].py to run a script) to include a shelltype argument that defaults to /bin/bash. It can be changed to /bin/sh or whatever one's favorite flavor.

One problem in my code is that if there is a command line request from the script (as in a "read" statement), then it doesn't work as it doesn't grab the input from python. I don't know if there is a way around this or not. I tried using the file descripter 0 for the stdin pipe, but that didn't seem to work. I'll try some other things and post back if I'm successful. Help form the more brilliant out there would be appreciated in this problem.

Anyway, here's the new code.

Code:

def shellCommands (cmd='echo "Hello, World!"', input=None, shelltype=None):
    '''Runs shell commands
    
    syntax:
    
        shellCommands(cmd[, input, shelltype])
    
    This function takes cmd, which is a command line or command lines to be 
    run in a bash shell.
    
    The argument input can also be specified to pass info via stdin to the 
    shell.
    
    The argument shelltype is for the shell that is desired, defaulting to
    /bin/bash. Using shelltype='/bin/sh' would change it to that shell.
    
    NOTE: THE subprocess MODULE HAS AN ERROR IN THE SOURCE. SEE THE BOTTOM
    OF THE DOCSTRING FOR CORRECTIONS THAT I HAVE MADE THAT FIXES IT

    The function returns a tuple which is output from standard output, 
    standard error, any standard input passed, and the return code giving:
    
        (stdout, stderr, stdin, returncode)


    examples:
    
        >>> text = 'Hello World!'
        >>> cmd = \'\'\'TEXT=`cat </dev/stdin`
        ... echo $TEXT\'\'\'
        >>> (stdout, stderr, stdin, returncode) = shellCommands(cmd, text)
        >>> print stdout
        Hello World!
        >>> print stderr
        None
        >>> print stdin
        Hello World!
        >>> print returncode 
        0
        
        >>> cmd = \'\'\'echo "Hello"
        echos "World"\'\'\'
        >>> stdout, stderr, stdin, returncode = shellCommands(cmd)
        >>> print stdout 
        Hello
        >>> print stderr 
        /bin/bash: line 1: echos: command not found
        >>> print stdin 
        None
        >>> print returncode 
        127
        >>> print shellCommands ('echos')
        (None, '/bin/bash: echos: command not found\n', None, 127)
        >>> print shellCommands ('echos', shelltype='/bin/sh')
        (None, '/bin/sh: echos: not found\n', None, 127)
        >>>
        
        
        
    FIXING ERROR IN subprocess MODULE:
    
        change:
        lines 1018-1022 for python v 2.6
        lines 1092-1096 for python v 2.7
        lines 1046-1050 for python v 3.1
        from this:
        
            if shell:
                args = ["/bin/sh", "-c"] + args

            if executable is None:
                executable = args[0]

        to this:
        
            if shell:
            if executable is None:
                args = ["/bin/sh", "-c"] + args
                executable = args[0]
            else:
                args = [executable, "-c"] + args

#            if executable is None:
#                executable = args[0]

    '''
    
    from subprocess2 import Popen, PIPE
    
    if shelltype is None:
        shelltype = '/bin/bash'
    
    shell = Popen(cmd, shell=True, executable=shelltype, stdin=PIPE, stdout=PIPE, stderr=PIPE)

    stdout, stderr = shell.communicate(input)

    if not stdout:
        stdout = None

    stdin = input
    
    if not stderr:
        stderr = None

    return stdout, stderr, stdin, shell.returncode

Yours,
Narnie

PS, the error exists in at least versions 2.6 thru 3.1 and the line numbers to correct are included in my shellCommands's docstring for each version of python. Can't wait to submit my bug report and fix.

bgeddy · 07-12-2010, 12:06 PM

Thanks for keeping us posted with your progress. If you are interested the IRC channel on freenode #python has some top python guys available for help and discussion. There's also #python-dev which is for python language developers (although it can be pretty intense)! Also the main python website has lots of very useful information and links. There you can look at PEP 324 all about the subprocess module as does the standard library reference which states :

Quote:

The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh.

As you have seen - this doesn't appear to be the module's behavior. It look like it's using /bin/sh and passing this the executable which is slightly different.

ghostdog74 · 07-13-2010, 01:01 AM

@OP, what shell commands do you think you have to absolutely call from Python, instead of using Python's own modules?

narnie · 07-13-2010, 02:30 AM

Ok, for anyone following this thread, I'm going to post a class I have made to do what my function above did, with some twists. I did this mainly to get some more "classy" experience, but it might be useful to some. I feel it might be a little sloppy, but it works for me.

Here is the code, and how it works should be found to be well documented in the docstring.

Code:

class ShellTools(object):
    '''Runs shell commands
    
    Usage:
    
        shell = RunInshell()
        shell.cmd = 'echo "Hey there!"'
        stdout, stderr, stdin, returncode = shell.execute()
    
    This class is set up to define a command, any input to the shell that
    is needed, define what shell is desired to run it in if the default 
    /bin/bash shell is not desired , then call the execute() method which
    returns the results (as a tuple of four values as noted above).
    
    WARNING:
    
        You will not be able to run code in /bin/bash with the stock
        subprocess module in Python 2.6, 2.7, nor 3.1 as there is a bug
        that makes it always run in /bin/sh almost no matter what. I will
        be submitting a bug report for this in hopes it will be fixed up-
        stream.
        
        However, it is easy to fix the source. One can either do this 
        directly with the subprocess module, or save it in your PYTHONPATH 
        and run that version of the subprocess module (this is what I am
        doing but is not reflected in this code for ShellTools).
        
        To see how to correct this error, please see the bottom of this help
        for all current versions of Python.
    
    This class is designed to be quite flexible in its utilization.
    
    In its simplest form:
        
        >>> shell = ShellTools()
        >>> shell.execute()
        ('Hello, World!\\n', None, None, 0)

    It the command and input can be given at first binding of the class
    like so:
        
        >>> shell = ShellTools('echo -n "My name is: " ; cat /dev/stdin', "Fred")
        >>> shell.execute()
        ('My name is: Fred', None, 'Fred', 0)
        >>>

    It can also be bound first, then set up and run in this way:
    
        >>> shell = ShellTools()
        >>> shell.execute('echo "How are you today?" ; echos', whichShell="/bin/sh")
        ('How are you today?\\n', '/bin/sh: echos: not found\\n', None, 127)
        >>>
    
    Finally, each variable that can be set explicitly can be used in the following way
    allowing versility in calling the execute method several times:
    
        >>> shell = ShellTools()
        >>> shell.cmd = 'ls s* ; cat </dev/stdin'
        >>> shell.input = 'These are the files that start with "s"'
        >>> for i in shell.execute():
        ...     print i
        ...     
        shelltool.py
        shelve.test
        subprocess2.py
        subprocess2.pyc
        These are the files that start with "s"
        None
        These are the files that start with "s"
        0
        >>>
        >>> shell.whichShell = '/bin/sh'
        >>> shell.cmd = 'echos'
        >>> shell.execute()
        ------------------------------------------------------------
        Traceback (most recent call last):
          File "<ipython console>", line 1, in <module>
          File "shelltool.py", line 82, in execute
            input, stdin, stdout, stderr, returncode = None, None, None, None, None
          File "subprocess2.py", line 689, in communicate
            return self._communicate(input)
          File "subprocess2.py", line 1201, in _communicate
            bytes_written = os.write(self.stdin.fileno(), chunk)
        OSError: [Errno 32] Broken pipe

        >>> shell.input 
        'These are the files that start with "s"'
        >>> shell.input = None 
        >>> shell.execute()
        (None, '/bin/sh: echos: not found\\n', None, 127)
        >>>
        >>> shell.whichShell = '/bin/bash'
        >>> shell.execute()
        (None, '/bin/bash: echos: command not found\\n', None, 127)
        >>>
        
    Here you can also see a caveat. It is best to have the ShellTools.input set
    to None if not needed, or it may generate a Broken pipe error.
    
    Also, the stdin, stdout, stderr, and return code is available after running
    execute by accessing them in this way:
        
        >>> shell.execute('echo "Here are my variables:"')
        ('Here are my variables:\\n', None, None, 0)
        >>> print shell.stdout 
        Here are my variables:
        >>> print shell.stdin 
        None
        >>> print shell.stderr 
        None
        >>> print shell.returncode 
        0
        >>>
    
    FIXING ERROR IN subprocess MODULE:

    change:
    lines 1018-1022 for python v 2.6
    lines 1092-1096 for python v 2.7
    lines 1046-1050 for python v 3.1
    from this:
    
        if shell:
            args = ["/bin/sh", "-c"] + args

        if executable is None:
            executable = args[0]

    to this:
    
        if shell:
        if executable is None:
            args = ["/bin/sh", "-c"] + args
            executable = args[0]
        else:
            args = [executable, "-c"] + args

#        if executable is None:
#            executable = args[0]

    '''

    cmd = 'echo "Hello, World!"'
    whichShell = '/bin/bash'
    input, stdin, stdout, stderr, returncode = None, None, None, None, None
    
    def __init__ (self, command=None, input=None, whichShell=None):
    
        if command is not None:
            self.cmd = command
            run = True
            
        if input is not None:
            self.input = input
            
        if whichShell is not None:
            self.whichShell = whichShell
        
    def execute(self, command=None, input=None, whichShell=None):

        if command is None:
            command = self.cmd
            
        if input is None:
            input = self.input
            
        if whichShell is None:
            whichShell = self.whichShell
            
        from subprocess import Popen, PIPE
        
        shell = Popen(command, shell=True, executable=whichShell, stdin=PIPE, stdout=PIPE, stderr=PIPE)

        self.stdout, self.stderr = shell.communicate(input)

        if not self.stdout:
            self.stdout = None

        self.stdin = input
        
        if not self.stderr:
            self.stderr = None
        
        self.returncode = shell.returncode

        return self.stdout, self.stderr, self.stdin, shell.returncode

Cheerio,
Narnie

narnie · 07-13-2010, 02:31 AM

Quote:

Originally Posted by bgeddy

Thanks for keeping us posted with your progress. If you are interested the IRC channel on freenode #python has some top python guys available for help and discussion. There's also #python-dev which is for python language developers (although it can be pretty intense)! Also the main python website has lots of very useful information and links. There you can look at PEP 324 all about the subprocess module as does the standard library reference which states :
As you have seen - this doesn't appear to be the module's behavior. It look like it's using /bin/sh and passing this the executable which is slightly different.

My pleasure.

Narnie

narnie · 07-13-2010, 02:40 AM

Quote:

Originally Posted by ghostdog74

@OP, what shell commands do you think you have to absolutely call from Python, instead of using Python's own modules?

Well, I'm just learning Python, but am quit good at bash scripting.

I must admit I haven't even begun to explore all of the modules for Python.

Any pointers to modules that cover the same things that the shell does would be greatly appreciated, because in Python, I would like to lessen my bash dependency.

An example might be the mv command in the bash shell. If the python equivalent tries to move across filesystems, it will fail. I read about another module that can be used for this, but it is much easier for me to just call mv from bash which will actually perform a copy and then delete the original when the destination crosses the filesystem (this is the same behavior for the "better" Python module, but at this point, there is so much to learn, it is easier for me to just drop back to the shell).

Another reason might be bottlenecks. I might be pushing on this one, but since most of the shell commands are written in C, they are bound to be faster. Of course, if the hard drive is the bottleneck, this won't matter.

Again, pointers to as many modules as possible for me to review from the standard library (and otherwise) that cover similar jobs to shell programming would be very much greatly appreciated.

Cheerio,
Narnie

bgeddy · 07-13-2010, 07:16 AM

Quote:

Well, I'm just learning Python, but am quit good at bash scripting.

Knowing what you are setting out to do (shell scipting in python ) this book - Python for Unix and Linux System Administration may be quite useful. I've had a copy for some time but can't really comment as I haven't used it much. It makes mention of the subprocess module (it covers a lot more) but doesn't really go into detail. Looks quite useful anyway. Given that you are wanting to replace shell scripts with python it may be just what you are looking for. Worth a look anyway.

bgeddy · 07-13-2010, 09:32 AM

Quote:

An example might be the mv command in the bash shell. If the python equivalent tries to move across filesystems, it will fail. I read about another module that can be used for this, but it is much easier for me to just call mv from bash which will actually perform a copy and then delete the original when the destination crosses the filesystem (this is the same behavior for the "better" Python module, but at this point, there is so much to learn, it is easier for me to just drop back to the shell).

Ok so the os.rename might fail here but look at the shutil module and its's abilities. The command here will move across filesystems no problem:

Code:

import shutil
shutil.move("/jfs/file1","/ext3/")

Names are made up for illustration. There are lots of python modules to assist with almost anything and often the hard part is finding out about them! I have to agree with ghostdog74 on this point - running a shell from python is often not the best way to go about stuff.

ghostdog74 · 07-13-2010, 07:35 PM

[QUOTE=narnie;4031582]

Quote:

Any pointers to modules that cover the same things that the shell does would be greatly appreciated, because in Python, I would like to lessen my bash dependency.

read the Python docs, especially the libraries. Its all there.

Quote:

An example might be the mv command in the bash shell. If the python equivalent tries to move across filesystems, it will fail.

shutil

Quote:

Another reason might be bottlenecks. I might be pushing on this one, but since most of the shell commands are written in C, they are bound to be faster. Of course, if the hard drive is the bottleneck, this won't matter.

well, then i don't see a need to use Python. Just use the shell...

Quote:

Again, pointers to as many modules as possible for me to review from the standard library (and otherwise) that cover similar jobs to shell programming would be very much greatly appreciated.

Look here, especially under file and directory access..