Needing help with forking??? a task in Python

narnie · 08-28-2010, 07:48 PM

Hello,

I'm trying to figure out how to continue execution of a Python program while a background task is going on.

In bash, it is adding "&" to the end as in:

Code:

#! /bin/bash
find / -iname "*.py" > /tmp/all_pys &
echo "looking for all python scripts and putting them in /tmp/all_pys"

(yes, I know I could just put the echo BEFORE the find but I just use it for example)

Say I do that in python (not necessarily running the shell, but perhaps using scriptutil.ffind or something similar), but want to allow the user to be doing something else while running that search in the background, how would I thread, fork, subprocess (not to be confused with the module and Popen) [or whatever I should call it] that?

Thanks in advance,
Narnie

ghostdog74 · 08-28-2010, 09:49 PM

what is it you are trying to do? Do you want to run Python scripts using the bash shell? Describe more clearly and show your Python code.

Kenny_Strawn · 08-29-2010, 12:35 AM

Code:

import os

os.fork()

ghostdog74 · 08-29-2010, 12:51 AM

Quote:

Originally Posted by Kenny_Strawn

Code:

import os

while True:
   os.Fork()

its os.fork() and why are you putting it inside an infinity loop?

narnie · 08-29-2010, 04:37 PM

Quote:

Originally Posted by ghostdog74

what is it you are trying to do? Do you want to run Python scripts using the bash shell? Describe more clearly and show your Python code.

This is just an example. I don't have any particular task in mind (yet). I just want to know how it is done for the future. Here is something I just quickly threw together:

Code:

#! /usr/bin/python

from sys import exit
from time import sleep

def calcPI():
    #calculates pi to 20 million decimal places
    #do math magic here
    pass

def fileSearch():
    #find file from root down
    from subprocess import Popen, PIPE
    command = 'find / -iname "*myfile"*'
    shell = Popen(command, shell=True, stderr=PIPE, stdin=PIPE, stdout=PIPE)
    stdout, stderr = shell.communicate('')
    if not stderr:
        print stdout
    else:
        print stderr
        if stdout.strip():
            print stdout

def longTask3():
    #this will take at least 30 mins
    sleep(60*30)

def longTask4():
    #this will take at least 3 hours
    sleep(60*30*3)

def doWhat():
    options = {
        1: calcPI,
        2: fileSearch,
        3: longTask3,
        4: longTask4,
        5: exit
        }
    text = '''
What would you like to do:

1) Find pi to 20 million decimals
2) Find a file
3) do longTask3
4) do longTask4
5) exit

? '''
    while True:
        num = raw_input(text)
        try: num = int(num)
        except ValueError:
            print '\nChoose only these options:'
            continue
        if not num in options:
            print '\nTry again with these options:'
            continue
        doWhateverToForkorThreadorWhatever(options, num)

def doWhateverToForkorThreadorWhatever(options, num):
    print '\nrunning option number', num
    options[num]()

doWhat()

Greatfully,
Narnie

narnie · 08-29-2010, 04:37 PM

Quote:

Originally Posted by Kenny_Strawn

Code:

import os

while True:
   os.Fork()

will check this out, thanks!

Narnie

narnie · 08-29-2010, 04:40 PM

Quote:

Originally Posted by narnie

will check this out, thanks!

Narnie

I think he meant os.fork() not os.Fork() for those who might find this thread in the furture.

ghostdog74 · 08-29-2010, 07:30 PM

Quote:

Originally Posted by narnie

This is just an example. I don't have any particular task in mind (yet). I just want to know how it is done for the future. Here is something I just quickly threw together:

if what have in mind is to "fork" or call external shell process, do it if you really have no choice otherwise, use Python's own modules to do it.

Quote:

Code:

def fileSearch():
    #find file from root down
    from subprocess import Popen, PIPE
    command = 'find / -iname "*myfile"*'
    shell = Popen(command, shell=True, stderr=PIPE, stdin=PIPE, stdout=PIPE)
    stdout, stderr = shell.communicate('')
    if not stderr:
        print stdout
    else:
        print stderr
        if stdout.strip():
            print stdout

Use os.walk(). See the Python docs for more. General usage:

Code:

import os
for r,d,f in os.walk(/path):
    for file in f:
         print os.path.join(r,file)

another related module you might be interested in in future is threading.

narnie · 08-29-2010, 11:24 PM

Quote:

Originally Posted by ghostdog74

if what have in mind is to "fork" or call external shell process, do it if you really have no choice otherwise, use Python's own modules to do it.

Use os.walk(). See the Python docs for more. General usage:

Code:

import os
for r,d,f in os.walk(/path):
    for file in f:
         print os.path.join(r,file)

another related module you might be interested in in future is threading.

Not necessarily shell scripting processes or tasks. I just used the find file as an example.

The finding pi process would strictly be a Python math process (ah the calculus of it). Also, just as placeholders, my longProcess3 and 4 are Python-only processes (in this case, using time.sleep() to keep it pure Python).

I'll look into the threading module with high hopes. I would like to start a function and have it working in the background as the program continues to run.

What I didn't really point out and should have is that in the flow of the program, I want to run, say, option 1. It will start the function and return to the list allowing me to run another option immediately after the first option was started. Thus, I'm not waiting for the called function to finish before being able to choose another.

With thanks,
Narnie

narnie · 08-29-2010, 11:37 PM

Quote:

Originally Posted by ghostdog74

. . .another related module you might be interested in in future is threading.

This looks like what I'm looking for. I was hoping implementation was a little easier and quicker (it's hard to beat the efficiency of "(sleep 5 ; echo "done sleeping) &" but I'm sure it adds more flexibility).

I'll do some searching on actual code examples for this module so I can get an idea of it's implementation in the "real world." Any pointers, here, too would be appreciated.

With grateful thanks,
Narnie

narnie · 08-29-2010, 11:42 PM

Quote:

Originally Posted by ghostdog74

. . .
Use os.walk(). See the Python docs for more. General usage:

Code:

import os
for r,d,f in os.walk(/path):
    for file in f:
         print os.path.join(r,file)

Also, thanks for this example. I haven't gotten to the os.walk() yet, so it is nice to see it in action. Many thanks!

Narnie

MTK358 · 08-30-2010, 05:05 AM

Quote:

Originally Posted by narnie

will check this out, thanks!

NO!

That's a malicious program called a "fork bomb". It will fork so many processes that it completely blocks up your computer.

I don't know why Kenny_Strawn is so obsessed about them, probably once we caught his failed attempt to get Windows users to run one in another thread.

crabboy · 08-31-2010, 11:50 PM

Kenny, your post is as dumb as the old rm slash trick that 13 year old kids love to post. Very immature and not at all helpful.

Kenny_Strawn · 09-01-2010, 01:46 AM

Okay, as an update, I was only posting an example of how to use the os.fork() function. Yes, it can be used in other ways, but to the OP: Just use the function, not the while loop around it. If you EVER use the fork() function in an infinite loop, you will end up with a program that infinitely forks out of control, causing your system to freeze. USE THE FORK OPERATION SPARINGLY!!!!!

MTK358 · 09-01-2010, 06:55 AM

Then why did you write the while loop around it?