Hi All,
I know the pain to set up the kgdb and debug the driver module.
Therefore, after I successfully set up the environmnet and succeed to debug my driver these days, I would like post the procedures of setting to help the people who is tortured by it. I think it is the spirit of open source.
If you have other successful and better experiment, please share with us and help me make it better. I will appreciate it.
BTW, I apply it on 32-bit FC 11 and I built kgdb-light from sourceforge.
Thanks for the help from Caz Yokoyama, Nikhil Nygaard and Paul Fulghum.

KGDB Setting & Debugging Driver Module:
1, on the development machine
1.1 in the root home directory, edit a .gdbinit file with the content as below:
set prompt (kgdb)
set solib-search-path /root/arcsas:/root/arcmsr /*set solib-search-path <.ko file's path1>:<.ko file's path2>:<.ko file's path3>*/
set remotebaud 115200
target remote /dev/ttyS0
/* Note: There is no need to add two entries, set debug remote 1 and set debugkernel 1*/
/* Note: The order among the entries is important.*/
1.2 Build a kernel with CONFIG_KGDB=y and CONFIG_KGDB_SERIAL_CONSOLE=y on the development machine, and copy the vmlinuz-XXX and initrd-XXX and system.map-XXX to test machine.
Make a hyperlink for vmlinuz-XXX, initrd-XXX and system.map-XXX.

2, on the test machine
2.1 edit the /etc/rc.d/rc.local file as below,
stty -F /dev/ttyS0 115200 /*This is very important especially on FC 11. Because after booting up, the speed of ttyS* will be reset.
Once gdb can not connect the test machine while Alt+Sys_Rq+g is launched on the test machine.
This entry should be there for this purpose.*/
echo 0 > /proc/sysrq-trigger /*Sets the console log level, controlling which kernel messages will be printed to your console.
'0' would make it so that only emergency messages like PANICs or OOPSes would make it to your console.*/
echo 1 > /proc/sys/kernel/sysrq /*The detailed description can be found in Documentation/sysrq.txt. '1' means enable all functions of sysrq*/
2.2 edit /etc/grub.conf as below,
kgdb=ttyS0,115200 kgdboc=ttyS0,115200 kgdbwait /*This is the only feasible kernel parameter so far*/

3, After the message, "kgdb: Waiting for connection from remote gdb...", showned on the test machine,
run with "gdb vmlinux", which vmlinux is not the bzImage, and it should be the kernel built in the /usr/src/linux/
after the "make && make modules_install && make install" procedure.

4, Press "c" or "continue" after the gdb prompt on the development machine.

5, Insert the mgsl_get_text_ptr() & BREAKPOINT() in the driver's initial function(the first function of driver), where mgsl_get_text_ptr() & BREAKPOINT() are duplicated from drivers/char/synclink.c.

6, Insert driver module on the test machine, then test machine will be stopped automatically.
There is a Hex number appeared in the gdb. Keep it in mind. Then make test machine continue to run.

7, On the development machine, run "more /sys/module/driver_name/sections/.text", and collect the .text section address.

8, Manually calculate the difference of the values, which respectively are collect from 6, and 7,. Keep it.
Reboot the test machine and repeat the Step 3,4,6.

9, Subtrat the value of Step 8 from the hex number you got in the gdb on the development machine. You can use it as the .text section address
while run "add-symbol-file <driver .ko file> <.text section address>".

Thanks for the great post, Nick. Following your procedure, I am now very close to be able to step through my custom module using kgdb.

I can break on my test routine by name, but am unable to step through the code because for some reason it cannot find the line numbers. After I load the module, I can cat the .text/.data/.bss files under /sys/module/<module name>/sections to get the offsets, and then I use the add-symbol-file command to send this information to gdb. That seems to be enough for gdb to know where to break by routine name, but after I break it can't find the line numbers in the source file for some reason.

I noticed in your post you said something about adding a BREAK() command at the first routine in my module and then hitting the break to find out the first text address where my module was loaded. Then you
say something about subtracting this value from the cat .text/.data/.bss values optained from prior step. Why do we have to
do this? Aren't the cat values all we need for the add-symbol-file command or am I missing some important detail? Thanks for explaining if it's not too much trouble.

Best Regards,

Ray Copeland

Hi Raycope,
Actually, my intent is to debug the initialization porcess of my driver, because my driver would die in the middle of the process.
Ideally while you run "set solib-search-path <.ko path>", the driver symbols will be loaded automatically. It is not necessary to calculate the .text address or run "add-symbol-file <.ko file> <.text section address>" by the Caz's comments.
It makes sense because if someone like me got a driver which would hangs the system during the initialization process, he would never got the .text address on the test machine to debug the driver further.
Therefore, I have to make some calculation.
As to your issue, I just can make a wild guess.
If I were you, I would make the driver on the target machine and then copy the .ko file to the development machine.
Do you have "set solib-search-path <.ko path>" in the .gdinit?
I keep this folder path having the only .ko file you need.
I hope it works for you.

Cheers,

Thanks, Nick. I opened Linux Questions first thing this morning and it was nice to see your reply.

It is good to know why you needed to break at your module start routine. If your module fails before it can load successfully, that makes sense. In my case, the module runs perfectly fine, I just want to break on a write routine when I do a proc I/O (cat) command.

I did not mention in my prior post, but I do have a set solib-search-path cmd in my .gdbinit. I build my module on my home Linux machine and then copy over to the kgdb host. I then start my session and load the module on the target. When I load the module, I get the text, data, & bss offsets and do the add-symbol-file ***using the .ko I copied to my host machine***.

I will try building on the target machine, and then doing the solib and
add-symbol-file commands on the target side insteasd of host side to see if it makes any diff.

Thanks again for your help. I will reply to this post when I arrive at the solution so everyone can see. Any more ideas/suggestions welcome.

Gracie, Ray C

Hello again, Nick and other LinuxQuestioners...

I spent some time on my process just to make sure I have everything correct to the best of my ability but here is what I have come down
to (close but no cigar yet):

1. Establish kgdb session over serial link and successfully break
and single step on EMBEDDED routine (e.g., ./kernel/module.c) OK.

2. Start up my target kernel under kgdb host control and able to
insmod target module and addded symbols using add-symbol-file
with target module object (.o) file OK. (Note I got .text/.data/
.bss offsets from /sys/module/<target module>/sections as per
standard setup.)

3. Used objdump -S -x <target module>.o command to verify target module
was compiled with debug symbols. Seems OK, meaning I see routine
names and source code information.

4. Able to break on target module routine name OK. However, when I try
the gdb 'n' or 'list' commands, it says it has **no line number
information**. 'n' acts like 'cont' meanings it just runs through
the whole routine without stopping. 'stepi' works fine in
dissasembly mode, so for temp workaround I am using stepi combined
with objdump information to debug but NEED FULLY SYMBOLIC SOURCE
CODE DEBUG facility to be really effective.

Any ideas, suggestions what might be wrong? Note I have spent a lot of
time on this and have tried all the basics including running from my native build machine as the kgdb host vs. tarring/un-tarring source code on another machine. Re:linux config, I have the following settings:

CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_PROC_KCORE=y
CONFIG_DEBUG_INFO=y

Any other config settings I might be missing to debug KLM's (downloadable modules)? Module was gcc compiled with -g and -O0 switch settings.

Thanks in advance for all replies.

Hi Gracie,
Have you ever tried to execute "dos2unix driver_source"?
I have no idea if it helps but it has no harm to try.

I am copying sources from Linux-to-Linux so CR/LF translation is not the issue for sure. Thanks anyway for your suggestion.

Hi Gracie,
I recommend you to read this book, Embedded Linux Primer: A Practical, Real-World Approach.
It has a great introduction for the kgdb.

Hi Gracie,

I face exactly the same problem. I was wondering if you were able to solve it. If so, could you, please,
share your experience.

Thank you,
Best,
Grigory

Just in case,

The solution is to use gdb for kgdb-light - http://sourceforge.net/projects/kgdb-light/

Thanks, Grigory, I will give kgdb-light a try ASAP. What makes it so different from regular gdb to add symbolic debug info? Thanks for any
details you can provide.