Hello,

I am in college and in one of my classes we are given an assignment in which we are give a binary of a program and a library (Not a binary, we can edit the library). The program deadlocks at some point when it is running. The library is just an implementation of pthreads. We are suppose to make some changes in the kernel (probably add a system call or edit the current system call according to the hints given) to prevent the program from deadlocking. I don't really know where to start. I was think to keep track on the mutexes and write an algorithm to prevent deadlock in the kernel. I know pthread uses sys_clone(). So how can I keep track of the mutexes inside the kernel? Could anyone point me in a right direction?

Thanks

Since you have the library you need to inspect that library and try to understand it.
You also need to find where the binary deadlocks.

Also how do you know it deadlocks? What hints are you given and why don't they help you to go in the right direction?

It is in the project description that the program deadlocks. Prof and TA are not very useful for some reason, whole class is having problems.

I and other classmates examined the library but it is not the problem because it just uses pthreads without applying any logic or algorithms.

So is it possible to monitor mutex by making a kernel module?

Have you tried 'strace'?

It's possible to debug just using printk()

This has to do with threads and therefore multiple process IDs, so output as much information when you start these threads, process ID being the main one.

Using system tools like "top" you can find out what process ID is stuck waiting for semaphore and then backtrack on that.

You can use either printk() or just output data to different log files to print when you obtain and give up semaphores.

That's what I've done in the past. Meanwhile also output what the processes are doing, stuff like "still running", but use a delay for obvious reasons on that one.

I'm half guessing that the library is just a distraction. What kind of library is not binary and that you can edit by the way? Never heard of that. Do you link your code with it, or is it merely a script of some type?

You might not be able to do anything about the deadlock...

Deadlocks are caused by bugs in the application. You can't fix that by changing the kernel.

An additional problem is that adding code to the library can change the timing - and that in turn changes the timing of the deadlock. It may not even happen.