LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware > Linux - Embedded & Single-board computer
User Name
Password
Linux - Embedded & Single-board computer This forum is for the discussion of Linux on both embedded devices and single-board computers (such as the Raspberry Pi, BeagleBoard and PandaBoard). Discussions involving Arduino, plug computers and other micro-controller like devices are also welcome.

Notices

Reply
 
Search this Thread
Old 12-30-2008, 11:30 AM   #1
Astralix
LQ Newbie
 
Registered: Dec 2008
Location: Germany
Distribution: Fedora, Ubuntu
Posts: 5

Rep: Reputation: 0
Unhappy Kernel panic - not syncing: Attempted to kill init! I am without any clue...


Hi!

I am upgrading a system based on a AT91RM9200 running Linux 2.4.x to Kernel 2.6.27.10 using buildroot enviroment ( Kernel, uClibc, busybox).

While I had the hardware and board specific things implemented into the kernel quite fast, the system boot up now costed me about 4 weeks...

The system boots from NFS or NOR Flash without problems until here:
Code:
at91_rtc at91_rtc: setting system clock to 1998-01-01 00:00:11 UTC (883612811)
eth0: Link now 100-FullDuplex
IP-Config: Complete:
     device=eth0, addr=192.168.10.22, mask=255.255.255.0, gw=192.168.10.1,
     host=CMC-TC-PU2, domain=, nis-domain=(none),
     bootserver=192.168.10.116, rootserver=192.168.10.116, rootpath=
Looking up port of RPC 100003/2 on 192.168.10.116
Looking up port of RPC 100005/1 on 192.168.10.116
VFS: Mounted root (nfs filesystem).
Freeing init memory: 128K
Kernel panic - not syncing: Attempted to kill init!
I followed a lot of links and did a lot of requests in other at91 specific boars, but there seems to be kind of vacation times...

I compared lot of things with ELDK as a concurrent build enviroment and found out, that if I mount the example romfs-image from ELDK via NFS it works down to busybox and shell. But it doesn't with buildroot.

I also wrote a statically linked hello_world program and called it via kernel command line init=/bin/hello and it worked. I did the compile run with the buildroot tools, so these seem to work. As they also where used to build my kernel, I didn't expect anything else.

So, now I am pretty stuck, as the kernel, busybox or whoever is the reason for the dying init, doesn't tell me anything why it's doing that. Can someone please tell me, how to enable debugging, or any other sort of things to find out the reason, why init dies?

If there's someone who knows a bit more, what Du you need from my configs, logs or sources, I'll post it right away.

But I am really stuck now, so any help is really appreciated, really

Best regards and thanks in advance,

Ulrich
 
Old 12-30-2008, 01:04 PM   #2
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Astralix View Post
So, now I am pretty stuck, as the kernel, busybox or whoever is the reason for the dying init, doesn't tell me anything why it's doing that. Can someone please tell me, how to enable debugging, or any other sort of things to find out the reason, why init dies?
You can add a bunch of bare printks to init. A bare printk is a printk without the "KERN_XXX" prefix before the format string.

Here is a link to a thread where the OP solved a similar problem by using an earlier kernel.
 
Old 12-30-2008, 03:00 PM   #3
Astralix
LQ Newbie
 
Registered: Dec 2008
Location: Germany
Distribution: Fedora, Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
Ok, yes, I saw this thread an read it twice. But the thing is, that I really like to get one of the latest kernels going with my system. You know, I am pretty new to kernel programming. I did programming with a lot of systems from 8 Bit to DVD Cores, but none of them was with a real runtime system. So I learn. While it was quite easy to write a board.c for my system and get it running with the kernel, I don't much about Linux programming, I think. I am more on the hardware side.
But neverthless I would like to learn. So may be you can take my by the hand and guide me a bit through it?
I am shure it will help a lot of readers as I didn't find anything like that in all those discussion I am crawling through the last weeks.

If you like, lets start here:
I've edited init.c of buildroot and inserted some printk("INIT, I'm here...\n");
Unfortunately init.c doesn't know much of printk.
If I'd like to include a header file for it I searched 'toolchain_build_arm/linux/include' where I guessed the header files of the kernel should be. But grepping for printk gives thousands of hits in hundrets of .h files.
looking in kernel.h (buildroot/build_arm/staging_dir/usr/include/linux/kernel.h) shows only a small file not containing any printk definition.
No printk... And I set buildroot compilation to abort on any error...

Regards, Ulrich
 
Old 12-30-2008, 03:37 PM   #4
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Astralix View Post
I've edited init.c of buildroot and inserted some printk("INIT, I'm here...\n");
This is a link to the code for the ARM version of init.c on the Linux cross-reference site.
Code:
 397 static inline void free_area(unsigned long addr, unsigned long end, char *s)
 398 {
 399         unsigned int size = (end - addr) >> 10;
 400
 401         for (; addr < end; addr += PAGE_SIZE) {
 402                 struct page *page = virt_to_page(addr);
 403                 ClearPageReserved(page);
 404                 init_page_count(page);
 405                 free_page(addr);
 406                 totalram_pages++;
 407         }
 408
 409         if (size && s)
 410                 printk(KERN_INFO "Freeing %s memory: %dK\n", s, size);
 411 }
As you can see, it calls printk to display the last message you are seeing before your kernel panic. So you must be putting the printk in the wrong place or you have some other build problem.
 
Old 12-30-2008, 03:45 PM   #5
Astralix
LQ Newbie
 
Registered: Dec 2008
Location: Germany
Distribution: Fedora, Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
Ah, ok, I added the printk to the init.c of busybox, not the kernel.
I expected the busybox init function to die, not that one of the kernel...

But checking for kernel.h in the buildroot system shows up, that the function is defined in
'project_build_arm/cmc_pu2/linux-2.6.27.10/include/linux/kernel.h'
But that is not the directory where the files for busybox are referencing to. These are somwhere else inside the buildroot subtrees. So I messed something up there and busybox is build agains other headers? Or is buildroot doing something else?

I am puzzled... But thanks for helping me.

Regards, Ulrich

Last edited by Astralix; 12-30-2008 at 04:00 PM.
 
Old 12-30-2008, 04:26 PM   #6
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Astralix View Post
But that is not the directory where the files for busybox are referencing to.
busybox is a user mode application that provides all of your shell commands in one executable. It will include user mode header files (e.g. /usr/include), not kernel header files (e.g. not /usr/src/linux/include). These example paths are for a desktop system, so the actual paths used by your embedded system will be much different.
 
Old 12-30-2008, 04:51 PM   #7
Astralix
LQ Newbie
 
Registered: Dec 2008
Location: Germany
Distribution: Fedora, Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
OK, something learned

But how to debug the reason of the crash then?
I mean, I can put a lot of funny comments in the kernel's init.c but what could help me out of my problem. I thought the init of the busybox died, so the kernel should work. I thought you'd like me to put some debug in that busybox init and see, if it comes up and where it goes down.

As the system runs into busybox and saying hello to me with a working shell if I use exactly the same kernel, but I mount the default rootfs from the LEDK system instead of the one generated by busybox, I expect the kernel to be fine so far, but either busybox or uclibc from buildroot is crappy. So how to debug that?
Is there additional library debug possible, or memory debug, or can I put any other xprintx into the busybox to track down, the problem. Or should I throw buildroot into the yard and try ELDK?

What is your idea to that, or with what toolchain did you have success on your systems?

Regards, Ulrich
 
Old 12-31-2008, 08:30 AM   #8
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Astralix View Post
But how to debug the reason of the crash then?
I mean, I can put a lot of funny comments in the kernel's init.c but what could help me solve my problem?
The last line emitted before the kernel panic was
Code:
Freeing init memory: 128K
The kernel panic occurs after that. That message is emitted by free_initmem. If you look in the code for init_post, you will see where "free_initmem" is called. You can start adding debug trace after that call to see which function fails. Then you add debug trace to the function that fails, to see where it fails. Repeat the process until you have found the cause of the problem.

Quote:
Originally Posted by Astralix View Post
I thought the init of the busybox died, so the kernel should work. I thought you'd like me to put some debug in that busybox init and see, if it comes up and where it goes down.
A kernel panic only occurs in kernel code. busybox is user mode code. You can put debug trace in busybox, but you would be starting with the assumption that busybox is the problem. With my method, you are guaranteed to find the source of the problem.

If kgdb is supported for your target, you can enable it and use GDB over serial or Ethernet to step through your kernel code. It is difficult to get kgdb setup the first time you use it, but it saves a lot of time tracking down these kind of problems.
 
Old 01-01-2009, 06:32 AM   #9
Astralix
LQ Newbie
 
Registered: Dec 2008
Location: Germany
Distribution: Fedora, Ubuntu
Posts: 5

Original Poster
Rep: Reputation: 0
Quote:
A kernel panic only occurs in kernel code. busybox is user mode code. With my method, you are guaranteed to find the source of the problem.
Yep, you're right, it's normally my way too... I was to eager to get it running. The stormy run through isn't the best if you'd like to learn.

Quote:
If kgdb is supported for your target, you can enable it and use GDB over serial or Ethernet to step through your kernel code. It is difficult to get kgdb setup the first time you use it, but it saves a lot of time tracking down these kind of problems.
Serial console has only 9600, DBGU has no V24 Driver... But ethernet is working fine and up an running. I am thinnking of going the harder way and setup KGDB as I need it anyways. On my board are some more Chips that are not supported from linux so far and I will have to implement them.


Best regards and a happy new year!
Ulrich
 
Old 01-01-2009, 10:18 AM   #10
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Astralix View Post
I am thinnking of going the harder way and setup KGDB as I need it anyways.
If you get a chance, post the steps you use to get KGDB working on your target. You may save someone else from going bald by tearing their hair out...
 
Old 01-05-2009, 01:11 AM   #11
ikennith
LQ Newbie
 
Registered: Jan 2009
Location: China
Distribution: fedora,Suse
Posts: 3

Rep: Reputation: 0
maybe
please select your root filesystem while building your kernel.

and compress your image with right format..
 
  


Reply

Tags
busybox, uclibc


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Kernel panic - not syncing: Attempted to kill init! dwhitney67 Linux - Kernel 13 09-26-2009 08:27 AM
Kernel Panic - not syncing: Attempted to kill init! Adimelo Fedora - Installation 7 07-06-2007 07:25 PM
Kernel panic - not syncing: Attempted to kill init! Kotresh Linux - Software 3 04-16-2007 08:48 AM
kernel panic - not syncing: Attempted to kill init! sharad Linux - Kernel 7 02-19-2007 01:44 AM
kernel panic - no syncing: Attempted to kill init majara Linux - General 3 02-06-2006 01:01 AM


All times are GMT -5. The time now is 07:40 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration