Kernel panic trying to use NFS as root filesystem

qwerty4061 · 01-09-2012, 04:00 AM

Hi,
I have an MIPS board to which I am trying to port linux (2.6.29). I am having 2 vmlinux images. Both of them are using "initramfs" as their sole filesystem. They were both generated using the same ".config" file. The only difference between them is that one uses busybox and the other uses bash shell.

I am able to boot the kernel using busybox but the one using bash gives kernel panic saying that it cannot access NFS. The kernel command line of both of them is same "Kernel command line: ip=192.168.16.225 nfsroot=192.168.2.23:/root/timesys/rfs console=ttyS0,38400n8r".

I am not able to understand why one is failing and the other one is able to boot successfully. Seeing the kernel commandline I feel that both should fail trying to access NFS as I have not configured NFS on my host machine

panic dmesg:

Quote:

LINUX started...

memsize not set in boot prom, set to default (480Mb)
Config serial console: console=ttyS0,38400n8r
console [early0] enabled
CPU revision is: 0a019554 (MIPS 34Kc)
Determined physical RAM map:
memory: 059ff000 @ 00000000 (reserved)
memory: 00c82000 @ 05a00000 (reserved)
memory: 1597d000 @ 06682000 (usable)
memory: 02000000 @ 1bfff000 (reserved)
Wasting 839744 bytes for tracking 26242 unused pages
Initrd not found or empty - disabling initrd
Zone PFN ranges:
Normal 0x00000000 -> 0x0001bfff
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000000 -> 0x000059ff
0: 0x00005a00 -> 0x0001bfff
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 113790
Kernel command line: ip=192.168.16.225 nfsroot=192.168.2.23:/root/timesys/rfs console=ttyS0,38400n8r
Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
MIPS secondary cache 512kB, 8-way, linesize 32 bytes.
Writing ErrCtl register=80000000
Readback ErrCtl register=80000000
Cache parity protection enabled
PID hash table entries: 2048 (order: 11, 8192 bytes)
CPU frequency 600.00 MHz
Console: colour dummy device 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 349444k/353780k available (2248k kernel code, 3992k reserved, 490k data, 9844k init, 0k highmem)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 399.36 BogoMIPS (lpj=798720)
Mount-cache hash table entries: 512
net_namespace: 520 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
fuse init (API version 7.11)
msgmni has been set to 683
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
console handover: boot [early0] -> real [ttyS0] 16550A
brd: module loaded
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
WpENET (): not using net_device_ops yet
WpENET: device found at 0xb000, using IRQ 49.
physmap platform flash device: 00800000 at 1f800000
physmap-flash.0: Found 1 x16 devices at 0x0 in 16-bit bank
NOR chip too large to fit in mapping. Attempting to cope...
Amd/Fujitsu Extended Query Table at 0x0040
physmap-flash.0: CFI does not contain boot bank location. Assuming top.
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
Reducing visibility of 16384KiB chip to 8192KiB
cmdlinepart partition parsing not available
RedBoot partition parsing not available
mtd: Giving out device 0 to physmap-flash.0
Searching for NAND flash...
NAND device: Manufacturer ID: 0x20, Chip ID: 0xf1 (ST Micro NAND 128MiB 3,3V 8-bit)
cmdlinepart partition parsing not available
Using static partition definition
Creating 3 MTD partitions on "wds-nand":
0x000000000000-0x000000200000 : "boot"
mtd: Giving out device 1 to boot
0x000000200000-0x000001600000 : "kernel"
mtd: Giving out device 2 to kernel
0x000001600000-0x000008000000 : "osfs"
mtd: Giving out device 3 to osfs
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
WpENET: found, using IRQ 49.
IP-Config: Guessing netmask 255.255.255.0
IP-Config: Complete:
device=WpENET, addr=192.168.16.225, mask=255.255.255.0, gw=255.255.255.255,
host=192.168.16.225, domain=, nis-domain=(none),
bootserver=255.255.255.255, rootserver=192.168.2.23, rootpath=
Looking up port of RPC 100003/2 on 192.168.2.23
rpcbind: server 192.168.2.23 not responding, timed out
Root-NFS: Unable to get nfsd port number from server, using default
Looking up port of RPC 100005/1 on 192.168.2.23
rpcbind: server 192.168.2.23 not responding, timed out
Root-NFS: Unable to get mountd port number from server, using default
Root-NFS: Server returned error -5 while mounting /root/timesys/rfs
VFS: Unable to mount root fs via NFS, trying floppy.
VFS: Cannot open root device "<NULL>" or unknown-block(2,0)
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

dmesg of working kernel:

Quote:

LINUX started...

memsize not set in boot prom, set to default (480Mb)
Config serial console: console=ttyS0,38400n8r
console [early0] enabled
CPU revision is: 0a019554 (MIPS 34Kc)
Determined physical RAM map:
memory: 059ff000 @ 00000000 (reserved)
memory: 008f6000 @ 05a00000 (reserved)
memory: 15d09000 @ 062f6000 (usable)
memory: 02000000 @ 1bfff000 (reserved)
Wasting 810688 bytes for tracking 25334 unused pages
Initrd not found or empty - disabling initrd
Zone PFN ranges:
Normal 0x00000000 -> 0x0001bfff
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000000 -> 0x000059ff
0: 0x00005a00 -> 0x0001bfff
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 113790
Kernel command line: ip=192.168.16.225 nfsroot=192.168.2.23:/root/timesys/rfs console=ttyS0,38400n8r
Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
MIPS secondary cache 512kB, 8-way, linesize 32 bytes.
Writing ErrCtl register=80000000
Readback ErrCtl register=80000000
Cache parity protection enabled
PID hash table entries: 2048 (order: 11, 8192 bytes)
CPU frequency 600.00 MHz
Console: colour dummy device 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 353156k/357412k available (2243k kernel code, 3992k reserved, 496k data, 6212k init, 0k highmem)
SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 399.36 BogoMIPS (lpj=798720)
Mount-cache hash table entries: 512
net_namespace: 520 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
fuse init (API version 7.11)
msgmni has been set to 690
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
console handover: boot [early0] -> real [ttyS0] 16550A
brd: module loaded
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
WpENET (): not using net_device_ops yet
WpENET: device found at 0xb000, using IRQ 49.
physmap platform flash device: 00800000 at 1f800000
physmap-flash.0: Found 1 x16 devices at 0x0 in 16-bit bank
NOR chip too large to fit in mapping. Attempting to cope...
Amd/Fujitsu Extended Query Table at 0x0040
physmap-flash.0: CFI does not contain boot bank location. Assuming top.
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
Reducing visibility of 16384KiB chip to 8192KiB
cmdlinepart partition parsing not available
RedBoot partition parsing not available
mtd: Giving out device 0 to physmap-flash.0
Searching for NAND flash...
NAND device: Manufacturer ID: 0x20, Chip ID: 0xf1 (ST Micro NAND 128MiB 3,3V 8-bit)
cmdlinepart partition parsing not available
Using static partition definition
Creating 3 MTD partitions on "wds-nand":
0x000000000000-0x000000200000 : "boot"
mtd: Giving out device 1 to boot
0x000000200000-0x000001600000 : "kernel"
mtd: Giving out device 2 to kernel
0x000001600000-0x000008000000 : "osfs"
mtd: Giving out device 3 to osfs
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
WpENET: found, using IRQ 49.
IP-Config: Guessing netmask 255.255.255.0
IP-Config: Complete:
device=WpENET, addr=192.168.16.225, mask=255.255.255.0, gw=255.255.255.255,
host=192.168.16.225, domain=, nis-domain=(none),
bootserver=255.255.255.255, rootserver=192.168.2.23, rootpath=
Freeing unused kernel memory: 6212k freed
init started: BusyBox v1.19.3 (2011-12-05 05:59:30 EST)

:
:
This successfuly completes

mulyadi.santosa · 01-10-2012, 12:33 AM

Hi qwerty

I could only guess several causes below (let's focus on "time out..." line...I believe you saw that too):

1. There is a firewall blocking the traffic from your machine when boots the bash based initramfs. It could be on the client side or the NFS server side, or boot.

2. You forgot to install the NFS/RPC userland daemon/utilities in the bash based initramfs.

No other possibilities that I can think of right now. Feel free to follow up my above suspicion....

qwerty4061 · 01-10-2012, 11:29 AM

Thanks Mulayadi for your answer, but I guess I wasn't very clear in my question. What I was interested was to know what is prompting the kernel to look for the root filesystem on the network. I don't want to use NFS but I am not able to understand why the busybox based one isn't having any problem whereas the bash based one is looking for NFS.

mulyadi.santosa · 01-10-2012, 11:44 AM

Hi qwerty

Sorry if there was misunderstanding.

Well, looking at the kernel command line, it seems that indeed both bash and busybox based kernel are instructed to do nfs root booting. But, perhaps the bash based one failed to failover to normal disk based booting, whereas busybox one is fine.

If the above logic is correct, then likely there are additional kernel modules that needs to add, either as built in part of kernel or inserted into initrd/initramfs. The candidate could be filesystem modules or SCSI/SATA/EIDE or any related disk controller driver. perhaps you are better than me to recognize this issue.

Hope it helps

qwerty4061 · 01-10-2012, 01:33 PM

Hi,
I don't think it is a case of missing kernel modules, because both of them had the same .config file

theNbomr · 01-12-2012, 09:47 AM

Are you sure that all libraries required by bash are on the initramfs? Use ldd to check.
Your kernel commandline includes 'nfsroot=192.168.2.23:/root/timesys/rfs' in both cases, so that would account for the attempt to mount an NFS share. Without the additional 'root=/dev/nfs', it doesn't try to use an NFS share as root filesystem. Where did the kernel commandline come from; bootloader or built-in to kernel?

--- rod.

qwerty4061 · 01-13-2012, 10:31 AM

Quote:

Are you sure that all libraries required by bash are on the initramfs? Use ldd to check.
Your kernel commandline includes 'nfsroot=192.168.2.23:/root/timesys/rfs' in both cases, so that would account for the attempt to mount an NFS share. Without the additional 'root=/dev/nfs', it doesn't try to use an NFS share as root filesystem.

I guess the command line passed doesn't make much sense. I didn't know that until you answered my other question http://www.linuxquestions.org/questi...96#post4572896. I will modify it and update the thread.

Quote:

Where did the kernel commandline come from; bootloader or built-in to kernel?

It came from the bootloader

theNbomr · 01-13-2012, 11:11 AM

Quote:

Originally Posted by qwerty4061

It came from the bootloader

So, it is simple enough to get rid of, although that doesn't really seem to be germane to the problem.
--- rod.