LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
LinkBack Search this Thread
Old 01-11-2008, 09:07 PM   #1
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Rep: Reputation: 51
python: how do you capture stdout stream from external program?


I have a command line program that takes a lot of options and does a lot of things.

I'm making a python wrapper (GUI) to make the command line options easier. Without re-writing the whole other program (which isn't python), I thought I'd just have the python gui formulate the long command line & run the program in a separate (sub)process using spawnl() (or spawne()).

My question: The program spits percentage done back to standard out. If I can capture that, i can use it to update a progress bar in the python GUI.

I know how to redirect stdout from the python to, for instance, update a GUI field, but I haven't figured out how to redirect stdout from an external program launched from within python.

I can get the output from an external program using os.system(), but that only gives me the output when the external program is finished - I need something that polls more often (or can detect when it sees a newline or something of the sort).

ideas?
 
Old 01-11-2008, 09:30 PM   #2
Uncle_Theodore
Member
 
Registered: Dec 2007
Location: Charleston WV, USA
Distribution: Slackware 12.2, Arch Linux Amd64
Posts: 896

Rep: Reputation: 60
You mean, you want something like a popen call? Python has its own
http://docs.python.org/lib/module-popen2.html
 
Old 01-11-2008, 10:07 PM   #3
angrybanana
Member
 
Registered: Oct 2003
Distribution: Archlinux
Posts: 147

Rep: Reputation: 21
http://docs.python.org/lib/module-subprocess.html
Is also another way to do it.
 
Old 01-14-2008, 06:25 PM   #4
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Both of those solutions use some form of popen.

It appears to do what I asked, but it seems to load up a variable/buffer for one single output. For instance, using this example:

python script:
Code:
import subprocess as sub
p = sub.Popen('./script',stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output
./script:
Code:
#!/bin/tcsh -f
@ i = 0
while ($i <= 20)
    echo $i
    sleep 1
    @ i += 5
end
So, in this example, python only gives one output when the script exits, i.e.:

0
5
10
15
20

... all at once. What I'm looking to do is read what comes from stdout as it's coming from stdout, so instead of the number being printed all at once, they are bring printed one per second.

Is there some nice way of doing this?

... or am I just doing the above example incorrectly?

Last edited by BrianK; 01-14-2008 at 06:27 PM.
 
Old 01-14-2008, 07:17 PM   #5
Uncle_Theodore
Member
 
Registered: Dec 2007
Location: Charleston WV, USA
Distribution: Slackware 12.2, Arch Linux Amd64
Posts: 896

Rep: Reputation: 60
Actually, I was thinking more along the lines of

Code:
import os

p = os.popen('./script',"r")
while 1:
    line = p.readline()
    if not line: break
    print line
Your example looks a little overcomplicated...
 
Old 01-14-2008, 07:36 PM   #6
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 239Reputation: 239Reputation: 239
Quote:
Originally Posted by BrianK View Post
Both of those solutions use some form of popen.

It appears to do what I asked, but it seems to load up a variable/buffer for one single output. For instance, using this example:

python script:
Code:
import subprocess as sub
p = sub.Popen('./script',stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output
./script:
Code:
#!/bin/tcsh -f
@ i = 0
while ($i <= 20)
    echo $i
    sleep 1
    @ i += 5
end
So, in this example, python only gives one output when the script exits, i.e.:

0
5
10
15
20

... all at once. What I'm looking to do is read what comes from stdout as it's coming from stdout, so instead of the number being printed all at once, they are bring printed one per second.

Is there some nice way of doing this?

... or am I just doing the above example incorrectly?
Code:
#!/usr/bin/env python
import os,time
fin,fout = os.popen4(cmd) #cmd is your script
for result in fout.readlines():
    print result
    time.sleep(4)
of course, if this is REALLY what you are doing, then do everything in your Python script.
Code:
for i in range(0,20,5):
    print i
 
Old 01-14-2008, 07:52 PM   #7
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Quote:
Originally Posted by Uncle_Theodore View Post
Actually, I was thinking more along the lines of

Code:
import os

p = os.popen('./script',"r")
while 1:
    line = p.readline()
    if not line: break
    print line
Your example looks a little overcomplicated...
hrm... while a bit cleaner than mine (), that still blocks. So, for instance, if I put the above code in a button event in the GUI, the interface is completely blocked until the script completes, at which time, everything updates.

Do I have to manually thread this? Oh bother.
 
Old 01-24-2008, 04:30 PM   #8
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
In the interest of completeness, here's the result - not the cleanest, I'm sure, but work as an exercise:

feedback from a shell script that updates a progress bar:

Code:
import wx,os,subprocess as sub
        dlg = wx.ProgressDialog("Computation progress",
                               "Doing stuff...",
                               maximum = 100,
                               parent=self,
                               style = wx.PD_CAN_ABORT
                                | wx.PD_APP_MODAL
                                | wx.PD_ELAPSED_TIME
                                | wx.PD_ESTIMATED_TIME
                                | wx.PD_REMAINING_TIME)

        command = "some_program"

        p = sub.Popen(command,shell=True, stdout=sub.PIPE).stdout
        while 1:
            line = p.readline()
            if not line: break
            if line.strip().isdigit():
                msg = "Compositing... %d%%" % int(line)
                (keepGoing) = dlg.Update(int(line),msg)
                lastline = line
                if int(line) == 100:
                    dlg.Update(100,newmsg='Finishing up... (this may take a sec)')
            else:
                (keepGoing) = dlg.Update(int(lastline))
            if line == "ALL DONE":
                keepGoing = False
                break
            if not keepGoing:
                os.system("killall -9 some_program")
                break
... linux only as it uses killall

the shell script (which in this case would be "some_program") needs to spit out at least a number representing the percent done. It can spit back other things as well, but some of the output needs to be just a percentage (without percent sign) on a line by itself, i.e.:
Code:
#!/bin/tcsh -f
@ i = 1
while ($i <= 100)
 echo $i
 @ i += 1
 sleep 1
end

Last edited by BrianK; 01-24-2008 at 04:38 PM.
 
Old 01-24-2008, 08:32 PM   #9
angrybanana
Member
 
Registered: Oct 2003
Distribution: Archlinux
Posts: 147

Rep: Reputation: 21
Just throwing it out there pexpect might support this.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
demultiplexing bash stdout stream pobbz Linux - Software 3 06-21-2007 09:14 AM
How to capture both stdin and stdout of a child process? jineshkj Linux - Software 2 04-21-2006 06:34 AM
how to redirect stdout [binary stream] to multiple processes vtaminh Linux - General 2 08-19-2004 01:05 PM
stdout contents capture arvind_tyche Programming 1 06-24-2004 06:08 AM
stdout capture anb bring to file sqn Programming 2 07-10-2003 07:42 AM


All times are GMT -5. The time now is 08:10 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration