LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-27-2013, 11:09 AM   #1
Ericxx
Member
 
Registered: Feb 2010
Posts: 33

Rep: Reputation: 0
Multi-thread terminated unexpectedly with POSIX threads


Hello,

I am using POSIX threads to create several worker threads and scheduling them to run one by one. However, after some random execution time, all of the worker threads got terminated. In addition, this problem does not happen every time. Could you please kindly suggest how to debug this issue? Thanks very much.

Regards,
Eric
 
Old 06-27-2013, 11:33 AM   #2
JohnGraham
Member
 
Registered: Oct 2009
Posts: 467

Rep: Reputation: 139Reputation: 139
Not wanting to sound patronising, but have you tried literally running the code in a debugger? Also, what do you mean by "all of the worker threads got terminated"? Did the whole program terminate? Was there a message printed to the console? What happened?
 
Old 06-27-2013, 12:03 PM   #3
Ericxx
Member
 
Registered: Feb 2010
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by JohnGraham View Post
Not wanting to sound patronising, but have you tried literally running the code in a debugger? Also, what do you mean by "all of the worker threads got terminated"? Did the whole program terminate? Was there a message printed to the console? What happened?
Thanks, John. Basically, I have four threads. 1st is a data acquisition, 2nd is data processing, 3rd is data communication, 4th is data logging. The four threads are scheduled via condition variables to realize execution from 1 to 4 every 100 ms. The code can be running smoothly without any problems for quite a while with the console printing, but occasionally the whole program got stuck and no more output available in the console. I have tried to run in the debug mode to locate which line of code caused this issue. But the whole OS crashes, changed from GUI to console mode. I am also wondering if the crash of my application caused the OS crash. But as this happened occasionally, I find it hard to grasp the intermediate information of the application to analyse the problem. Let me know if you need any other details. Thank you.
 
Old 06-27-2013, 02:18 PM   #4
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,847
Blog Entries: 1

Rep: Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866Reputation: 1866
> But the whole OS crashes

Perhaps your program has eaten up every single bit of memory?
 
Old 06-27-2013, 02:51 PM   #5
JohnGraham
Member
 
Registered: Oct 2009
Posts: 467

Rep: Reputation: 139Reputation: 139
Quote:
Originally Posted by Ericxx View Post
But the whole OS crashes
Seriously? You mean the *whole* OS? As in you have to reboot your computer? What OS are you using, and on what platform?
 
Old 06-27-2013, 06:37 PM   #6
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
You've got a timing hole somewhere. Sux to debug it, but ...
 
Old 06-28-2013, 09:12 AM   #7
Ericxx
Member
 
Registered: Feb 2010
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by NevemTeve View Post
> But the whole OS crashes

Perhaps your program has eaten up every single bit of memory?
Should not be. I have monitored the resources with the vmstat and memory should be enough. The OS crashed only occasionally. Most of the time my program hanged but the OS still keeped running. Strange point is that when the application hangs, the camera capture thread keeps running, while all the other threads get killed. So this camera capture thread might be the problem?
 
Old 06-28-2013, 09:18 AM   #8
Ericxx
Member
 
Registered: Feb 2010
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by JohnGraham View Post
Seriously? You mean the *whole* OS? As in you have to reboot your computer? What OS are you using, and on what platform?
I am using Ubuntu 12.04 running on an Intel Core2Duo computer stack. Today I tried more times and OS seems to be stable Now just trying to figure out the reason for the crash.
 
Old 06-28-2013, 09:25 AM   #9
2ck
Member
 
Registered: Mar 2010
Location: /home/twock
Distribution: Debian
Posts: 74
Blog Entries: 9

Rep: Reputation: 21
It sounds like your threads are deadlocking. Can you share your code or give a more exacting description of how the threads interact, including all synchronization constructs?
 
Old 07-07-2013, 09:20 PM   #10
Ericxx
Member
 
Registered: Feb 2010
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by 2ck View Post
It sounds like your threads are deadlocking. Can you share your code or give a more exacting description of how the threads interact, including all synchronization constructs?
Sorry for my late update. I should have identified the problem which is a mutex issue. The situation is that one worker thread is capturing images continuously and updating the images which are accessed by other worker threads. One image data structure is defined for storing the updated image from the capture thread and accessed by other worker threads. I tried to use the mutex mechanism, but the application still got crashed randomly once executed. Looking forward to your suggestions on this. Thank you.

Regards,
Eric
 
Old 07-08-2013, 12:41 AM   #11
2ck
Member
 
Registered: Mar 2010
Location: /home/twock
Distribution: Debian
Posts: 74
Blog Entries: 9

Rep: Reputation: 21
I read your description again and it actually doesn't sound like deadlock, so sorry about that. sundialsvcs is right that it's likely some race condition in your code...

If there are multiple ways to access an image as it is being processed, then make sure the same mutex lock surrounds the code in each of them. Also, there is a 'try_acquire' or similar in the POSIX threads API. If you are using that, you should consider whether the thread that tries to acquire the lock does something that assumes the lock is held. That said, you should minimize the places where you can access the shared image, probably to synchronous read/write methods and have all access occur through those -- at least to start with.

That's all I can think of right now. A debugger probably would help since you'll be able to see what the threads were doing when they crashed. You'll still need to do some reasoning about why what they were doing was allowed to happen.

Good luck!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] what's the difference between a multi-cpu , multi-core and a multi-thread system ? entz Linux - Hardware 11 12-20-2011 04:49 PM
pthread: how to monitor if thread has terminated wkhoo Programming 1 08-15-2008 11:19 AM
kdm[****]: X server for display :0 terminated unexpectedly http:// Linux - Newbie 2 07-14-2008 01:20 PM
:0 Display in loop then terminated unexpectedly PB0711 SUSE / openSUSE 1 01-18-2008 08:09 PM
Azureus TERMINATED. unexpectedly. J_7D5 Linux - Software 3 11-11-2005 06:36 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:03 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration