How do you force Jitter into the Scheduler? Testing multithreaded programs.

cyent · 06-19-2014, 09:18 PM

So somebody has inflicted a multithreaded unit test on us. (Yes, yes, I Know, that's A Very Bad Idea).

Now this damn thing is flaky.

It fails sometimes.

I'd like to run it a thousand times with the linux scheduler making subtly different choices about scheduling the various threads in each run to see if I can get it to fail more often.... and hence find and fix the issue.

Any ideas?

sundialsvcs · 06-19-2014, 11:14 PM

You have a "race condition," therefore a poorly-designed program. A well-designed multiprocess/multithread program must make no assumptions as to timing. It must run correctly, no matter what the Linux scheduling settings might be. You cannot "fix the issue" without redesigning the program. Sorry . . .

cyent · 06-19-2014, 11:42 PM

Yes, yes, I know.

However, the first step to fixing it is to be able to instrument it more closely and trigger the fault again...

Alas, the fault only triggers on in fifty or so times.

Thus the question wasn't "Why is it flaky?" I know why.

The question is, "How can I get the Linux Scheduler to stop behaving like the lovely Best of Breed scheduler it is.... and start behaving like a malicious monster that will deliberately and with intent shuffle the scheduling and timing of the threads to expose the flaw?"

sundialsvcs · 06-20-2014, 07:20 AM

Of course you do.

Thread-sync problems are always flaky, and the only way I know of to deal with them is by desk-checking the application logic. For example, a shared data-structure that isn't protected by a mutex in every case. When you do successfully use a debugger, you often wind up looking at where the flaming wreckage landed, not the point at which the bomb actually went off in the cockpit.