Hi,
I want to debug my Kernel Module using kgdb. But, I am trapped in a Deadlock kind of situation.

When my Test Kernel boots, it waits for some command from Remote GDB connection. Then I press "Continue" and so that Kernel can boot properly. Now, the thing is as Kernel is running until Machine is really powered down and thus there is no way that I get (gdb) prompt back on my Dev machine.

Now, as I want to debug my own module. I do insmod that module on Test machine. But there is no way to do add-symbol-file and/or to set Breakpoints on Dev machine AS GDB prompt is not returned yet.

There is a possible solutions that I can think of after asking Google But yet I am not sure of.

1. Instead of using "Continue", I shoud use "detach". That will allow other machine to boot and it will detach my Dev machine from that Kernel. And later on I can reconnect it. But, when I reconnect using "target remote /dev/ttyS0", it waits for some DEBUG EVENT (or NMI) and I dont know how can I generate that DEBUG EVENT.


Any help would be more than welcome.

I have solved the whole thing and I ll be posting the solution soon

Hi
Here are the steps for configuring Kernel and consequently doing Kernel and Module debugging...

Everything below is related to or with reference to 2.6.26 Kernel.... The main reason is KGDB code is merged into Linux tree from 2.6.26-RC5 kernel

1. Obviously start with downloading the source of 2.6.26.2 kernel, the latest stable version.... from kernel.org
2. Recompile the Kernel -> Go to Kernel Hacking -> Enable two options one is related to SysRq Key, and the other one is KGDB ... Give "Y" to both of them...

3. Build kernel make -j 12 && make modules && make modules_install

4. Transfer the vmlinux and system.map and initrd.img files on Test machine.

5. Now, edit the GRUB entry for that kernel on Test machine... Add options or give Kernel Parameters like kgddbwait and kgdboc=ttyS0,115200

kgdbwait -- > This will make Kernel to wait on boot time and will expect someone to connect to it and give further commands
kgdboc --> This is a KGDB I/O driver and we are supplying two aarguments.... ttyS0 will tell that communication will happen on Serial Port 0 and the baudrate will be 115200

6. Now boot the Kernel with those kernel parameters...

7. On dev machine, start GDB session ... [dev@root]gdb vmlinux
The argument vmlinux file is the file that is created with Debug symbols... It will be of much larger size... and probably in the present directory where you gave "make" command..

8. We assume that on Dev machine you have set serial interface baudrate as 115200.. Then connect to the test machine with target command..
(gdb)target remote /dev/ttyS0

9. This will stop your Kernel booting on Test machine and will give control to your Dev machine..Now, you can do next and step and put breakpoints and etc...

BUT The real problem starts now, that I faced and solved
Once your Dev machine has control and you want to do normal boot.... You can do two things....
1. (gdb)Continue --> This will continue Kernel booting and it will not return GDB prompt back because it is quite understandable that your kernel will be running until it is not really powered off..
2. (gdb)detach --> This will continue kernel booting and gdb prompt will be returned and now GDB is detached from the test machine...

Now, the question is HOW to get to the KERNEL debugging again ? How Dev machine GDB can get control over Test machine again ?
So, the answer is Sending TRAP signal on GDB session doesnt help... You have to send MANUALLY on TEST machine SysRq command... So, on TEST machine press SysRq + g (i.e., press "ALT" key then Press "PrintScreen" Key and then Press "g" key)

This will stop the Kernel booting and give control to GDB... If you had "detach" on GDB session previously.... you will have to do again (gdb)target remote /dev/ttyS0

There are other possibilities where you face problems
1. Your Kerenel is booted and SysRq+g is not working...
Please do [r00t@root] echo 1 > /proc/sys/kernel/sysrq
This will enable sending SysRq commands.....

2. You may face problem that you want to do Kernel Module Debugging and how to stop Kernel...
There are two ways.... 1. You call kgdb_breakpoints() function in your init_module function and it is declared in kgdb.h file....This will send a TRAP signal and control will be passed to GDB session.........

Another way is load your module and then send SysRq + g command...

However, one fundamental thing you need to take care here ....
You had started your GDB session with vmlinux file and now you are trying to debug a module that you have written... Quite obviously your GDB session wont know any of the symbols that you are using in your Module...

To solve that problem.... you have to make GDB aware of those symbols....
(gdb)add-symbol-file <.ko file> <.TEXT section address>
You load module on Test machine and then do [root@test]cat /sys/modules/<module name>/sections/.text
This will give you the second argument that you need to give to add-symbol-file commands.....

It is assumed that every time you load the module, it will be loaded at the same location .......

There is one more problem that I faced
When I was doing Module debugging and I was calling the kgdb_breakpoints, the machine was hanging and I was not able to connect to Dev machine.... The logical reason was, it was hanging due to breakpoint but then it was not able to basically communicate.... and the reason was KGDB I/O driver was not configured.......... In such case you may need to reconfigure the KGDB I/O driver....
You can do the same on Test machine in a following way
[root@test]echo "ttyS0,115200" > /sys/modules/<module name>/parameters


I think after all these i was able to do everything quite perfectly

Hey there, great post, helped alot.

However, I am still having trouble with KGDB...

Particularly with module debugging, "set solib-search-path [path-to-driver]" seems to do nothing.

On my test machine I have my module in /work/ciofd/src/. On my development machine I have built the module in /usr/src/linux-2.6.27.9-7-kgdb/drivers/scsi/ciofd, which is a symlink to /work/ciofd/src.

I "set solib-search-path /work/ciofd/src/:/lib/modules/kernel/2.6.27.9-7-kgdb/drivers/scsi/ciofd" on the Dev machine and "c" to continue. I go over to the target machine and modprobe the driver successfully. I then trip pass control to the dev machine by "echo g > /proc/sysrq-trigger".

It is at this point that I should see that the shared libraries have been loaded, right? Like, "info sharedlibraries" should give me something, right? It says that no shared libraries are loaded at this time.

What am I doing wrong?

Thanks in advance!

Thanks for the awesome post.

Can I ask a real noob question about transferring the vmlinux to the test machine?

Should the test machine already have a linux installation on it or should it be an empty machine?

If it should have linux already on it, what version and distro should it be?


Thanks for helping.

I have few doubts regarding debugging kernel module.

1.
I want to debug a kernel module that is loaded dynamically (using insmod) then still we need to specify kernel boot parameter (kgdbwait kgdboc=ttyS0,115200) while booting the kernel (on test machine)? Can't we boot the test machine normally and later connect gdb to the test machine using target remote /dev/ttyS0 command for debugging the module loaded later.

2.
> (gdb)add-symbol-file <.ko file> <.TEXT section address>
> You load module on Test machine and then do [root@test]cat /sys/modules/<module name>/sections/.text
> This will give you the second argument that you need to give to add-symbol-file commands.....

a. Suppose the module get crashed while loading itself then how we can load/provide the symbol information to gdb. As per my understanding, the .text section would only be accessible after the module loaded successfully.

b. <.ko file> need to be specified with add-symbol-file, should be available on development machine and the same <.ko file> needs to be copied to the test machine for loading. Am I correct?

3.
Assuming my understanding regarding 2(b) is correct, I built my module on development machine (Say test.ko) and copied this to test machine.
Now, i performed the below step to load the symbol information

<Test Machine>
root@test:-$ cat /sys/module/test/sections/.text
say output of this command is 0xFFFFFFFF

<Development Machine>
(gdb)add-symbol-file <path to test.ko file in development system> 0xFFFFFFFF
This command loaded symbol information successfully.

(gdb) br test_function
This command set the break point to the test_function.

(gdb)target remote /dev/ttyS0

It waits for some activity on test machine

<Test machine>
Performed some action that uses my loaded driver and according to me the above test_function() must hit due to that action. But it is not hitting the break point as expected.

Please guide me where I went wrong.

Thanks in advance

Meena