Monitoring the execution of POSIX threads

I work on Red-Hat Enterprice OS.
I run one process which generates many POSIX threads.
How can I monitor/trace their execution.

Best regards,
Yehuda

Please elaborate - so far I would say that you monitor them like any other process/thread.

Dear Pearlseattle
First of all, thanks for your response.

We have a program which creates many threads. I want to monitor the threads. The threads are POSIX using pthread library.
The utilities ps, mpstat, iostat give information on the process level.

Best regards,
Yehuda

Hi
I might be wrong (anybody wants to comment?) but such threads are handled the same way as processes - meaning that if your run e.g. htop (or ps, etc...) and you spawn a thread, you'll see it running with its own PID (different from the parent process).

from man ps:

THREAD DISPLAY
H               Show threads as if they were processes

-L              Show threads, possibly with LWP and NLWP columns

-T              Show threads, possibly with SPID column

m               Show threads after processes

-m              Show threads after processes

Also, practically everything you would ever want to know about a particular thread can be found in files under /proc/[PID]/task/[TID].

Dear Millgates
Thanks for your answer.
Indeed top gives me the threads but the TID is not part of the information. Instead, I get a different PID for each POSIX therad.
Is there a way to get the TID also?

The information related to the thread is very interesting. How to get either he TID or the thread name while execution. We ran the top and ps utilities as a child process to the main process running the threads and record he information.
Thanks,
Yehuda

Originally Posted by YehudaSinger TID is not part of the information. Instead, I get a different PID for each POSIX therad. Is there a way to get the TID also?

The individual threads of a single process should share the PID. If you see different ID's for the threads, then those are TID, not PID.

That said, you can customize the output of ps, for example:

ps -L -o pid,tid,comm
  PID   TID COMMAND
  798   798 bash
 1699  1699 testprog
 1699  1700 testprog
 1699  1701 testprog
 1699  1702 testprog
 1699  1703 testprog
 1699  1704 testprog
 1699  1705 testprog
 1699  1706 testprog
 1709  1709 ps

Dear Millgates
Thanks for your answer. here are my remarks:
1, The TID is not the thread ID which is obtained either during the creation or by the thread_self)_'
2. The TID is obtained by the system call gettid(). This function is not in the libc. Hence, we call it by the following sequence: syscall(SYS_gettid) ;


Thanks.
Yehuda