Kernel panic - not syncing: Attempted to kill init! I am without any clue...

Hi!

I am upgrading a system based on a AT91RM9200 running Linux 2.4.x to Kernel 2.6.27.10 using buildroot enviroment ( Kernel, uClibc, busybox).

While I had the hardware and board specific things implemented into the kernel quite fast, the system boot up now costed me about 4 weeks...

The system boots from NFS or NOR Flash without problems until here:

at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:11 UTC (883612811)
eth0: Link now 100-FullDuplex
IP-Config: Complete:
     device=eth0, addr=192.168.10.22, mask=255.255.255.0, gw=192.168.10.1,
     host=CMC-TC-PU2, domain=, nis-domain=(none),
     bootserver=192.168.10.116, rootserver=192.168.10.116, rootpath=
Looking up port of RPC 100003/2 on 192.168.10.116
Looking up port of RPC 100005/1 on 192.168.10.116
VFS: Mounted root (nfs filesystem).
Freeing init memory: 128K
Kernel panic - not syncing: Attempted to kill init!

I followed a lot of links and did a lot of requests in other at91 specific boars, but there seems to be kind of vacation times...

I compared lot of things with ELDK as a concurrent build enviroment and found out, that if I mount the example romfs-image from ELDK via NFS it works down to busybox and shell. But it doesn't with buildroot.

I also wrote a statically linked hello_world program and called it via kernel command line init=/bin/hello and it worked. I did the compile run with the buildroot tools, so these seem to work. As they also where used to build my kernel, I didn't expect anything else.

So, now I am pretty stuck, as the kernel, busybox or whoever is the reason for the dying init, doesn't tell me anything why it's doing that. Can someone please tell me, how to enable debugging, or any other sort of things to find out the reason, why init dies?

If there's someone who knows a bit more, what Du you need from my configs, logs or sources, I'll post it right away.

But I am really stuck now, so any help is really appreciated, really

Best regards and thanks in advance,

Ulrich

Originally Posted by Astralix So, now I am pretty stuck, as the kernel, busybox or whoever is the reason for the dying init, doesn't tell me anything why it's doing that. Can someone please tell me, how to enable debugging, or any other sort of things to find out the reason, why init dies?

You can add a bunch of bare printks to init. A bare printk is a printk without the "KERN_XXX" prefix before the format string.

Here is a link to a thread where the OP solved a similar problem by using an earlier kernel.

Ok, yes, I saw this thread an read it twice. But the thing is, that I really like to get one of the latest kernels going with my system. You know, I am pretty new to kernel programming. I did programming with a lot of systems from 8 Bit to DVD Cores, but none of them was with a real runtime system. So I learn. While it was quite easy to write a board.c for my system and get it running with the kernel, I don't much about Linux programming, I think. I am more on the hardware side.
But neverthless I would like to learn. So may be you can take my by the hand and guide me a bit through it?
I am shure it will help a lot of readers as I didn't find anything like that in all those discussion I am crawling through the last weeks.

If you like, lets start here:
I've edited init.c of buildroot and inserted some printk("INIT, I'm here...\n");
Unfortunately init.c doesn't know much of printk.
If I'd like to include a header file for it I searched 'toolchain_build_arm/linux/include' where I guessed the header files of the kernel should be. But grepping for printk gives thousands of hits in hundrets of .h files.
looking in kernel.h (buildroot/build_arm/staging_dir/usr/include/linux/kernel.h) shows only a small file not containing any printk definition.
No printk... And I set buildroot compilation to abort on any error...

Regards, Ulrich

Originally Posted by Astralix I've edited init.c of buildroot and inserted some printk("INIT, I'm here...\n");

This is a link to the code for the ARM version of init.c on the Linux cross-reference site.

 397 static inline void free_area(unsigned long addr, unsigned long end, char *s)
 398 {
 399         unsigned int size = (end - addr) >> 10;
 400
 401         for (; addr < end; addr += PAGE_SIZE) {
 402                 struct page *page = virt_to_page(addr);
 403                 ClearPageReserved(page);
 404                 init_page_count(page);
 405                 free_page(addr);
 406                 totalram_pages++;
 407         }
 408
 409         if (size && s)
 410                 printk(KERN_INFO "Freeing %s memory: %dK\n", s, size);
 411 }

As you can see, it calls printk to display the last message you are seeing before your kernel panic. So you must be putting the printk in the wrong place or you have some other build problem.

Ah, ok, I added the printk to the init.c of busybox, not the kernel.
I expected the busybox init function to die, not that one of the kernel...

But checking for kernel.h in the buildroot system shows up, that the function is defined in
'project_build_arm/cmc_pu2/linux-2.6.27.10/include/linux/kernel.h'
But that is not the directory where the files for busybox are referencing to. These are somwhere else inside the buildroot subtrees. So I messed something up there and busybox is build agains other headers? Or is buildroot doing something else?

I am puzzled... But thanks for helping me.

Regards, Ulrich

Originally Posted by Astralix But that is not the directory where the files for busybox are referencing to.

busybox is a user mode application that provides all of your shell commands in one executable. It will include user mode header files (e.g. /usr/include), not kernel header files (e.g. not /usr/src/linux/include). These example paths are for a desktop system, so the actual paths used by your embedded system will be much different.

OK, something learned

But how to debug the reason of the crash then?
I mean, I can put a lot of funny comments in the kernel's init.c but what could help me out of my problem. I thought the init of the busybox died, so the kernel should work. I thought you'd like me to put some debug in that busybox init and see, if it comes up and where it goes down.

As the system runs into busybox and saying hello to me with a working shell if I use exactly the same kernel, but I mount the default rootfs from the LEDK system instead of the one generated by busybox, I expect the kernel to be fine so far, but either busybox or uclibc from buildroot is crappy. So how to debug that?
Is there additional library debug possible, or memory debug, or can I put any other xprintx into the busybox to track down, the problem. Or should I throw buildroot into the yard and try ELDK?

What is your idea to that, or with what toolchain did you have success on your systems?

Regards, Ulrich

Originally Posted by Astralix But how to debug the reason of the crash then? I mean, I can put a lot of funny comments in the kernel's init.c but what could help me solve my problem?

The last line emitted before the kernel panic was

Freeing init memory: 128K

The kernel panic occurs after that. That message is emitted by free_initmem. If you look in the code for init_post, you will see where "free_initmem" is called. You can start adding debug trace after that call to see which function fails. Then you add debug trace to the function that fails, to see where it fails. Repeat the process until you have found the cause of the problem.

Originally Posted by Astralix I thought the init of the busybox died, so the kernel should work. I thought you'd like me to put some debug in that busybox init and see, if it comes up and where it goes down.

A kernel panic only occurs in kernel code. busybox is user mode code. You can put debug trace in busybox, but you would be starting with the assumption that busybox is the problem. With my method, you are guaranteed to find the source of the problem.

If kgdb is supported for your target, you can enable it and use GDB over serial or Ethernet to step through your kernel code. It is difficult to get kgdb setup the first time you use it, but it saves a lot of time tracking down these kind of problems.

A kernel panic only occurs in kernel code. busybox is user mode code. With my method, you are guaranteed to find the source of the problem.

Yep, you're right, it's normally my way too... I was to eager to get it running. The stormy run through isn't the best if you'd like to learn.

If kgdb is supported for your target, you can enable it and use GDB over serial or Ethernet to step through your kernel code. It is difficult to get kgdb setup the first time you use it, but it saves a lot of time tracking down these kind of problems.

Serial console has only 9600, DBGU has no V24 Driver... But ethernet is working fine and up an running. I am thinnking of going the harder way and setup KGDB as I need it anyways. On my board are some more Chips that are not supported from linux so far and I will have to implement them.


Best regards and a happy new year!
Ulrich

Originally Posted by Astralix I am thinnking of going the harder way and setup KGDB as I need it anyways.

If you get a chance, post the steps you use to get KGDB working on your target. You may save someone else from going bald by tearing their hair out...

maybe
please select your root filesystem while building your kernel.

and compress your image with right format..