LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-22-2007, 07:11 AM   #1
paroxsitic
LQ Newbie
 
Registered: Apr 2007
Posts: 10

Rep: Reputation: 0
Trying to create a type of multi threading in bash.


I am trying to get a concept where I can create simultaneous running child processes from a parent and able to read their exit status as soon as they come in. Once the exit code is read it will start a new child to take its place. That way there is alway 10 children running at once.

In the end I am going to be using this for a way to do asynchronous wgets to speed up my wget scripting. So if you have an alternative, let me know. Thats also why I set a randomized timer on the example child process to simulate wgets and so I can check if it doesnt do things in order or not.

Here is the child process I am threading, just for reference:

Code:
number=0
let "number = $RANDOM % 20"
time=$(echo "1+$number*0.1" | bc)
sleep $time
echo "I got $1 ($time)"
exit $1

I first started out with this job script I found. I realized that it didnt get the exit codes fast enough. I wanted the exit code retrieved as soon as the process is exited.

Code:
#!/bin/bash

MAXJOBS=10

function jobidfromstring()
{
        local STRING;
        local RET;
 
        STRING=$1;
        RET="$(echo $STRING | sed 's/^[^0-9]*//' | sed 's/[^0-9].*$//')"
 
        echo $RET;
}
 
function clearToSpawn
{
    local JOBCOUNT="$(jobs -r | grep -c .)"
    if [ $JOBCOUNT -lt $MAXJOBS ] ; then
        echo 1;
        return 1;
    fi
 
    echo 0;
    return 0;
}
 
JOBLIST=""
 
for (( i=0; i<=20; i++ ));
do
        while [ `clearToSpawn` -ne 1 ] ; do
                sleep 0.5
        done   
        ./child $i &
        LASTJOB=`jobidfromstring $(jobs %%)`
       JOBLIST="$JOBLIST $LASTJOB"
done

for JOB in $JOBLIST ; do
        wait %$JOB
        echo "Job $JOB exited with status $?"
done
echo "Done."

Then I decided to tryout the SIGCHLD handler and see how that worked. It turned out that I wasn't able to define the handler in a way that would capture every child for some reason.

Code:
#!/bin/bash

iNext=0

function ChildReturned() {
        wait $1
        echo "$1 was returned. Its exits status was: $?"
        iNext=$((iNext+1))
        StartChild $iNext
}


function StartChild {
        ./child $1 &
        pid=$!
        echo "Started: $1  PID= $pid"
        trap 'ChildReturned $pid' SIGCHLD
}


set -bm

for (( iNext=0; iNext<=10; iNext++ ));
do
        StartChild $iNext
done

wait

Finally I tried just starting the children and looping through a while statement that would check when each one died, retrieve the exit code, and then start a new child. However I realized that a child that terminates (SIGCHLD) still exists as a process until you get the exit code with wait(). Therefore using kill -0 wasnt going to work.

Code:
#!/bin/bash


throttle=10
iNext=0
JobIDArray[0]=""

function jobidfromstring()
{
        local STRING;
        local RET;

        STRING=$1;
        RET="$(echo $STRING | sed 's/^[^0-9]*//' | sed 's/[^0-9].*$//')"
        echo $RET;
}

function AnyAlive {
        for (( i=1; i<=throttle; i++ )); do
                if kill -0 %${JobIDArray[i]} > /dev/null 2>&1; then
                        echo 1
                        return 1
                fi
        done
        echo 0
        return 0
}

for (( iNext=1; iNext<=throttle; iNext++ )); do
        ./child $iNext &
        JobIDArray[iNext]=`jobidfromstring $(jobs %%)`
        echo "          JobIDArray[$iNext] = ${JobIDArray[iNext]}"
done


while [[ `AnyAlive` -eq 1 ]] ; do
        for (( i=1; i<=throttle; i++ )); do
                jID=${JobIDArray[i]}
                if ! kill -0 %$jID; then
                        wait %$jID
                        return=$?
                        iNext=$((iNext+1))
                        ./child $iNext &
                        JobIDArray[i]=`jobidfromstring $(jobs %%)`
                        echo "$jID was returned. Its exits status was: $return   |   JobIDArray[$i] = ${JobIDArray[i]}"
                fi
        done
done


---------------------------

I am pretty much stuck at where to begin next. I am very new to bash and I don't even know if its possible to do what I want. My third script would work fine if I just had a way to identify if a job was terminated or not. I know that if you try to kill -s SIGCHLD the job when its terminated, it will give "no job exists" or something. I can't seem to know how I can get that to work in an IF statement. And if I did, would wait() still get the exit code.
 
Old 09-22-2007, 01:38 PM   #2
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Linux Mint 17, Linux Mint 18
Posts: 7,822

Rep: Reputation: 301Reputation: 301Reputation: 301Reputation: 301
I suggest that you base your logic on wait with the wpid parameter set to -1. That will wait for the termination of any of your 10 child processes.

When you get SIGCHLD terminate the child, start another child, and go into your multiple wait again.

-----------------
Steve Stites
 
Old 09-22-2007, 10:06 PM   #3
paroxsitic
LQ Newbie
 
Registered: Apr 2007
Posts: 10

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jailbait View Post
I suggest that you base your logic on wait with the wpid parameter set to -1. That will wait for the termination of any of your 10 child processes.

When you get SIGCHLD terminate the child, start another child, and go into your multiple wait again.

-----------------
Steve Stites

Only C has a wait function that accepts -1. I don't think bash will allow this :\
 
Old 09-23-2007, 01:29 PM   #4
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Linux Mint 17, Linux Mint 18
Posts: 7,822

Rep: Reputation: 301Reputation: 301Reputation: 301Reputation: 301
Quote:
Originally Posted by paroxsitic View Post
Only C has a wait function that accepts -1. I don't think bash will allow this :\
Take a look at wait3 or wait4. These is variations of wait that allow a -1 pid value.


http://www.logbud.com/man?mode=man&p...ait4&section=2

------------------
Steve Stites
 
Old 09-24-2007, 02:07 AM   #5
paroxsitic
LQ Newbie
 
Registered: Apr 2007
Posts: 10

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jailbait View Post
Take a look at wait3 or wait4. These is variations of wait that allow a -1 pid value.


http://www.logbud.com/man?mode=man&p...ait4&section=2

------------------
Steve Stites
I know, if I am not mistaken that is for C. I assuming that from the syntax of everything and also because of the .h files. I wanted something that could be done in all bash.
 
Old 09-24-2007, 02:51 AM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,329

Rep: Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364
Whe you start a background process, the pid of that process is stored in the special var '$!', so you can check for it.
See eg http://tldp.org/LDP/abs/html/internalvariables.html
 
Old 09-25-2007, 12:32 AM   #7
paroxsitic
LQ Newbie
 
Registered: Apr 2007
Posts: 10

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by chrism01 View Post
Whe you start a background process, the pid of that process is stored in the special var '$!', so you can check for it.
See eg http://tldp.org/LDP/abs/html/internalvariables.html
Yes I know that, please check my code.
My problem is knowing the PID or Job number of the child process that just sent a SIGCHLD. Thanks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
multi threading slower than single threading on dual core. why? nebojsa.andjelkovic Programming 13 01-30-2007 09:56 PM
problems about multi-threading and/or multi-processing with tcp network in java ... alred Programming 5 06-23-2006 03:21 PM
help! perl multi-threading? eph Programming 0 05-03-2004 09:15 PM
Multi-threading rch Programming 3 03-30-2003 10:27 PM
Multi threading Mohsen Programming 5 03-01-2003 11:13 PM


All times are GMT -5. The time now is 02:26 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration