if threads are lefting "hanging" they use up memory.
However, as a program runs it may periodically calls brk() and and if it does it then expands it's actual memory usage.
Programs hold onto memory - this is not under your control - in case more is needed later on.
setrlimit() can control this memory expansion. If you set limits using RLIMIT_AS this will control the max size of the process working set in memory.
If you can run your code outside of the embedded environment, try using valgrind to check for memory leaks.