Python: easiest way to wait for whichever child process completes first?

johnsfine · 06-20-2014, 08:25 AM

I'm guiding an intern, who is working in Python. I hardly know any Python myself.

The Python program currently launches child processes and immediately waits for each to complete. But I want it to launch several and then loop:
1) wait for whichever completes first
2) launch another one

They are expected to complete in a very different sequence than launched.

A google search found many people asking the same question and a lot of answers implying this problem is difficult.

The best I found was psutil.wait_procs, which seems to require complicated setup for an imperfect solution to the issue. Before I send the intern down that path, I'd like an opinion on whether it is the best path.

BTW, we need both a Windows solution and a Linux solution and would prefer if those were as similar to each other as practical.

linosaurusroot · 06-20-2014, 09:20 AM

Omitting the python part of your question; you want to use wait() repeatedly to get each child's exit. Contrast with waitpid() where you specify the pid you want.

There is a python discussion at http://bytes.com/topic/python/answer...t-losing-child

linosaurusroot · 06-20-2014, 09:21 AM

You could give each child process a pipe and spot termination with select().

johnsfine · 06-20-2014, 10:06 AM

Quote:

Originally Posted by linosaurusroot

you want to use wait() repeatedly to get each child's exit. Contrast with waitpid() where you specify the pid you want.

There is a python discussion at http://bytes.com/topic/python/answer...t-losing-child

Thanks. That sure looks simple. The working sample program (rather far down in that thread) also helps a lot.

I wonder about all the confusion I found when I did a google search for this. I guess I need to test whether that simple method works on both Windows and Linux or whether it is Linux only.

johnsfine · 06-21-2014, 12:41 PM

Quote:

Originally Posted by linosaurusroot

you want to use wait() repeatedly to get each child's exit. Contrast with waitpid() where you specify the pid you want.

In Linux Python, that seems to be exactly what I wanted. That doesn't work in Windows Python.

I need this Python program to work in both Linux and Windows.

I'm pretty sure the polling method can do the same job in Windows with a bit more complexity and less efficiency, and I'll nail down those details soon. But is there a better way?

turtleli · 06-21-2014, 02:17 PM

Not a Python programmer myself, but perhaps one of the Python modules for concurrent execution is what you're looking for? The multiprocessing and subprocess modules looks quite useful for doing what you described.

johnsfine · 06-22-2014, 05:28 AM

Since I don't know much Python, I'd appreciate comments (or suggested improvement) on the program I now have working in both Linux and Windows. I needed to use polling, because I did not find a better way to wait in Windows.

Code:

import os
import time
from subprocess import Popen

def wait4one(p):
   while True:
      for i,what in enumerate(p):
         if p[i] is not None:
            exitstat = p[i].poll()
            if exitstat is not None:
               print "i=%d exit=%d" % (i, exitstat)
               p[i] = None
               return i
      time.sleep(0.1)

delay = [2,1,3,1,2,3,1,2,1,2,2]
cost =  [1,1,1,2,1,1,1,3,1,6,1]
n_cost = [0, 0, 0, 0]
n_proc = {}

max_out = 4
out = 0
for i in xrange(11):
   while out > 0 and out+cost[i] > max_out:
      j = wait4one(n_proc)
      out = out - n_cost[j]
      n_cost[j] = 0
   for j in xrange(4):
      if n_cost[j] is 0:
         n_proc[j] = Popen('sleep '+str(delay[i]), shell=True)
         n_cost[j] = cost[i]
         out = out + cost[i]
         print "Started child process %d (%d) cost=%d delay=%d out=%d" % (j, n_proc[j].pid, cost[i], delay[i], out)
         break

delay[] and the sleep command are stand ins for something more complicated in the intended program (for now they test completion in a different sequence than start). cost[] will come from an external source, as will the 4 that appears in a few places (including length of n_cost[]). Otherwise, this test program is the same as what the real program should be.

Each process to be launched has a "cost", most are 1, but some larger. Don't worry about what cost is, but if you can't think of it as an abstract, pretend it is GB ram requirement. out is the total cost of all processes that have been launched but not completed. The processes are launched in a predetermined sequence, but may or may not be allowed to start before previous completions. A process with a cost >= max_out can only be started after all previous are finished. Other processes can be started as soon as the cost including that new one would be <= max_out

n_proc[] is the list of processes running now. n_cost[] is the cost of each running process.

sundialsvcs · 06-22-2014, 08:24 AM

I don't believe that "busy waiting" should ever be necessary, not even in Windows. Also, I have serious doubts whether this implementation would prove to be 100% reliable in a production case where processes were taking unpredictable amounts of time and might even terminate at the same time. I'd suggest looking carefully at the source-code to the concurrent-execution modules that turtlei recently spoke of, to see if there are any cross-platform gleanings to be won from it.

If this actually works for you, then of course the proper thing to do might be to "move along," but on a skeptical peer-review I would have to give this alternative a thumbs-down because I would never be convinced that it does not have timing-hole problems that would cause "vexing instability" under load. I think that a truly-reliable cross-platform solution exists, and I'll predict that the writers of the published library modules probably found it.

Now – it might be different such that you really do have to write it two different ways. I don't know; I haven't looked closely. I do know that the Windows threading-model doesn't exactly follow the "father reaps" model. It's more "DEC/PDP-ish" than that.

johnsfine · 06-22-2014, 09:27 AM

Quote:

Originally Posted by sundialsvcs

I'd suggest looking carefully at the source-code to the concurrent-execution modules that turtlei recently spoke of

Unless I seriously missed something, that primarily covers multi-threading within the Python program and I could not easily find a form of "wait for whatever finishes first" even there. In other languages, I am used to the technique of using a thread in the current process for each event outside the current process that you want to wait for. That can give you more flexibility in waiting for events than you get from whatever "wait for multiple events" construct is directly available. Something like that might be derived from the info turtleli linked, but it would be hard for someone who knows as little Python as I do to dig that out and the result would be complicated.

Quote:

I don't believe that "busy waiting" should ever be necessary, not even in Windows.

I wish that were true, but "wait for multiple events" is so messy in Windows that polling is often the better answer.

Quote:

Also, I have serious doubts whether this implementation would prove to be 100% reliable in a production case where processes were taking unpredictable amounts of time and might even terminate at the same time.

The test case does have multiple terminating close enough to the same time to reveal any such bugs. Until you poll for it being done, the info (exitcode and the fact that it is done) is stable.

I don't see anything in this polling design that might not be stable. If you have a specific failure path in mind, I would want to hear it. Or if you have simpler code, I always prefer that, because simpler is easier to trust. But your expression of general mistrust of the simple code I posted really doesn't tell me anything.

I still would like to hear specific suggestions from someone who knows Python better than I do.

ntubski · 06-26-2014, 06:23 AM

I'm not a python expert, but I have written a few small programs in it.

An obvious simplification would be to del a finished process from n_proc instead of setting it to None, this would save having to check for None. Also, it's a bit funny to use enumerate but then only use the index anyway (on the other hand, you have to use the index to reference the place in n_procs, so maybe it's better to use the index everywhere for uniformity?).

Code:

def wait4one(p):
   while True:
      for i, proc in enumerate(p):
            exitstat = proc.poll()
            if exitstat is not None:
               print "i=%d exit=%d" % (i, exitstat)
               del p[i]
               return i
      time.sleep(0.1)

I think the main loop code is bit tricky to follow because it's essentially encoding a list of objects as separate lists of attributes. cost and delay could combined like this:

Code:

from collections import namedtuple
Job = namedtuple('Job', ['delay', 'cost'])
jobs = [Job(d, c) for (d, c) in zip(delay,cost)]

# then you can use
for job in jobs:
    while out > 0 and out+job.cost > max_out:
    ...
    for ...
       Popen('sleep '+str(job.delay), shell=True)
       ...

And probably n_proc and n_cost could also be combined.

Quote:

Unless I seriously missed something, that primarily covers multi-threading within the Python program and I could not easily find a form of "wait for whatever finishes first" even there.

multiprocessing is for multiple processes, but the subprocesses are intended to run python, so on Windows you end up having an extra process creation for each external subprocess you create (although perhaps this overhead would be negligible next to the work your subprocesses perform).

The JoinableQueue or Pool classes could be useful to wait for multiple processes. I think you would need a BoundedSemaphore to implement the cost restriction.

johnsfine · 06-26-2014, 11:26 AM

Quote:

Originally Posted by ntubski

I'm not a python expert, but I have written a few small programs in it.

I barely know python at all. I have corrected bugs and mis-designed features in some really terrible python programs written by others. But never written any real python programs myself.

Code:

def wait4one(p):
   while True:
      for i, proc in enumerate(p):
            exitstat = proc.poll()
            if exitstat is not None:
               print "i=%d exit=%d" % (i, exitstat)
               del p[i]
               return i
      time.sleep(0.1)

Did you test your version of wait4one? It does not work for me.

I don't understand the rules of the container involved, so I don't know what proc gets from enumerate in your code. I would expect it to be p[i] and I tried that before posting my code. It is some other kind of object so proc.poll() does not work.

Still not understanding the container: I don't know what del p[i] does. But anyway it also doesn't work as you seem to intend it.

ntubski · 06-27-2014, 01:05 AM

Quote:

Originally Posted by johnsfine

Did you test your version of wait4one? It does not work for me.

Hmm, it seems I tested a different version than the one I posted. enumerate doesn't do something sensible on dictionaries. For lists (arrays) it gives index, value pairs.

Here is a working version of wait4one:

Code:

def wait4one(p):
   while True:
      for i in p.keys():
            exitstat = p[i].poll()
            if exitstat is not None:
               print "i=%d exit=%d" % (i, exitstat)
               del p[i]
               return i
      time.sleep(0.1)

del p[i] removes the slot i from p.