Download your favorite Linux distribution at LQ ISO.
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 06-04-2021, 07:50 AM   #1
Registered: Dec 2008
Posts: 41

Rep: Reputation: 1
Threads consuming memory - GDB debugging to narrow down the issue

Hi ,

I have one daemon which spawns around 15 threads.

first five threads timedwait threads.
third thread is service thread
other five threads are notified threads.

There are couple of issues , i am facing below.

1. Incrementally i am seeing that process is taking huge memory . I am seeing this process consumption from top command. Where this daemon process is eating 15% of memory when it stops responding.

2. From gdb trace i see the control flow got locked at below stack trace.

Thread 25 (Thread 0x7ff2dabef700 (LWP 11050)):^M
#0 0x00007ff2dd607b6c in __lll_lock_wait_private () from /lib64/^M
#1 0x00007ff2dd586bd6 in _L_lock_12192 () from /lib64/^M
#2 0x00007ff2dd584181 in malloc () from /lib64/^M
#3 0x00007ff2dd58a52a in strdup () from /lib64/

3. I see like around 10 threads waiting on same mutex varaiable as showed below in gdb trace.

#1 0x00007ff2de324d32 in _L_lock_791 () from /lib64/^M
#2 0x00007ff2de324c38 in pthread_mutex_lock () from /lib64/^

I want to debug this issue with gdb . Mainly i want to find which thread is eating memory.

Any steps to check or related articles would be very usefull.

Old 06-04-2021, 01:31 PM   #2
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,316
Blog Entries: 13

Rep: Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372Reputation: 4372
From within GDB, you get information on program threads with a command and you swap to the thread you wish to examine using a command, neither of which are any more difficult than the rest of GDB commands.

Perform a web search for GDB debugging threads, and you'll find lots of references.

Sometimes it's also good to instrument some debug so that you when you break in the debugger you can example status information. For one, I'd have each thread record and store somewhere, their own PID, their parent PID, and if they forked any children, those PIDs as well. This information is still available via GDB sometimes, but sometimes just easier to make your own variable, populate it, and then refer to it in the debugger.

I may be incorrect, but when it says Thread 25, that means it's that numbered thread, implying you have 25 or more threads. Albeit some threads may have terminated, and it may be that you have 10 open at any given time.
Old 06-05-2021, 06:32 AM   #3
Registered: Dec 2010
Location: California, USA
Distribution: I run my own OS
Posts: 635

Rep: Reputation: 300Reputation: 300Reputation: 300Reputation: 300
Find the call to free () that frees the memory allocated by strdup (). Confirm that free () is executed.

Debugging memory leaks is not very different in multithreaded programs. You need to review the malloc/free pairs to ensure that allocated memory is eventually freed.
Old 06-09-2021, 05:36 PM   #4
Senior Member
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Ubuntu, Slackware, Amazon Linux, OpenBSD, LFS (on Sparc_32 and i386)
Posts: 2,260
Blog Entries: 29

Rep: Reputation: 161Reputation: 161
and be sure that each strdup() is called with a valid and reasonable length string.
Old 06-14-2021, 01:58 AM   #5
Registered: Dec 2008
Posts: 41

Original Poster
Rep: Reputation: 1
Thanks for the pointers..

I am searching like .. what kind of system calls or any tools which i can include in debug messages .
This debug messages will be in thread code block .
This debug message will be printed in regular time frame and it should print the dynamic memory stats it's holding currently.

Thought process is like , This will help in narrowing down upto thread level granularity.

Is there already references in /proc filesystem , where i can find these info.

Thanks in advance
Old 06-14-2021, 12:23 PM   #6
Senior Member
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,170
Blog Entries: 1

Rep: Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536Reputation: 1536
Guess you should try valgrind. It should print sg. like this:
$ valgrind pwd
==182== Memcheck, a memory error detector
==182== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==182== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==182== Command: pwd
==182== error calling PR_SET_PTRACER, vgdb might block
==182==     in use at exit: 0 bytes in 0 blocks
==182==   total heap usage: 48 allocs, 48 frees, 13,358 bytes allocated
==182== All heap blocks were freed -- no leaks are possible
==182== For counts of detected and suppressed errors, rerun with: -v
==182== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to narrow down error 'The exec() call failed' ? susja Linux - Newbie 8 07-19-2012 09:53 PM
debugging with multiple threads using GDB. pkpatil424 Linux - Newbie 0 06-27-2012 04:39 AM
DDR Memory: Mixed Manufacturers? Or Straight and Narrow? JazzItSelf Linux - Hardware 2 10-12-2007 11:19 PM
Narrow font for AbiWord Gnute Linux - Software 1 08-21-2005 12:06 AM
narrow lil' boxes acid_kewpie LQ Suggestions & Feedback 8 02-10-2002 12:16 PM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:49 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration