[SOLVED] Unable to handle kernel NULL pointer - our hardware platform

hi
after a year, our modified version of mars-board with an arm (A10 -Allwinner) processor was implemented. we use merri core-board (C11 - with 512MB RAM, 4G FLASH) which control other -i/o- parts.

the only difference between our final extended-board vs mars is ram. (our board=512B vs mars board=1024MB)

after soldering PHY, rs232/rs485 drivers, ESD, logocl and analog ICs, we successfully compiled our customized linux kernel which the sources are downloaded from
http://linux-sunxi.org/.

it should be noted that we use the recommended compiler
arm-linux-gnueabihf-***

moreover we change the fex file and make kernel with appropriate export (ARC=arm, CROSS...) and also make our u-boot.

fortunately our embedded linux successfully boot with the default rootfs BUT
our kernel unexpectedly crash on :
:a big write on USB
:apt-get update/install
...
with the message: Unable to handle kernel NULL pointer dereference at virtual address xxxxxxxx
but exactly the same sd-card work fine in mars-board.

we checked our fex [dram-para] and our kernel compilation many times and can't reach any useful hint.

due to the virtual memory errors I write a C program below in order to check our kernel integrity:
#include <stdlib.h>
#include <stdio.h>

void main ()
{
int count=10;
int i;
unsigned long int j, len=25000000;
char *p[count];

for (i=0; i<count; i++)
{
p[i] = malloc(len);

if (p[i]==0)
{
i--;
continue;
}

printf("\nAddress %d: [%p]", i+1, p[i]);
}

for (i=0;i<count;i++)
{
for (j=0;j<len/2;j++)
{
p[i][j]=0;
p[i][len-j-1]=0;
}

printf("\nAddress %d OK.", i+1);
free(p[i]);
}
}

after cross-compiling and running the binary in sh-shell we receive some kernel oops as below:

32.121363] [000007e0] *pgd=5faeb831, *pte=00000000, *ppte=00000000
[ 32.127660] Internal error: Oops: 817 [#1] PREEMPT ARM
[ 32.132791] Modules linked in:
[ 32.135856] CPU: 0 Not tainted (3.4.103-g9a1cd03 #6)
[ 32.141172] PC is at free_pcppages_bulk+0x1ac/0x39c
[ 32.146045] LR is at 0x7e0
[ 32.148750] pc : [<c0099a9c>] lr : [<000007e0>] psr: 800f0093
[ 32.148755] sp : dfaf7d78 ip : d1360294 fp : ffffffff
[ 32.160224] r10: 00200200 r9 : 00100100 r8 : 00000002
[ 32.165443] r7 : 00000000 r6 : 00000001 r5 : ffffff80 r4 : d134fc00
[ 32.171960] r3 : d134fc80 r2 : 00000003 r1 : 000007e0 r0 : c0683b44
[ 32.178482] Flags: Nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 32.185695] Control: 10c5387d Table: 5fb18019 DAC: 00000015
[ 32.191432]
[ 32.191435] PC: 0xc0099a1c:
[ 32.195708] 9a1c e1a03623 e08312c1 e7ea1051 e021c216 e5948000 e061300c e00ce001 e061700e
[ 32.203969] 9a3c e7945283 e0843283 e1a05f25 e1550f28 1a00003f e593500c e3750080 1a00003c
[ 32.212229] 9a5c e593801c e1520008 1a000039 e593c000 e1a0100e e593e018 e0844287 e31c0902
[ 32.220489] 9a7c e593c014 e2800034 11a07002 01a07003 e2822001 e5977010 e352000a e58ce004
[ 32.228748] 9a9c e58ec000 e5839014 e583a018 e590c02c e24cc001 e580c02c e3a0c000 e583c01c
[ 32.237008] 9abc e593c00c e583b00c 1affffd7 e59d8024 e594300c e584201c e584500c e3a03034
[ 32.245268] 9adc e59dc014 e0030293 e2841014 e083318c e2833030 e088c003 e7980003 e5801004
[ 32.253528] 9afc e5840014 e584c018 e7881003 e3a03034 e0228293 e5923060 e2833001 e5823060
[ 32.261789]
[ 32.261792] SP: 0xdfaf7cf8:
[ 32.266063] 7cf8 00000017 00000001 00000004 c07094c8 c07094c0 dfaf6000 c06552b0 00000101
[ 32.274323] 7d18 c0099a9c 800f0093 ffffffff dfaf7d64 00000002 c000d918 c0683b44 000007e0
[ 32.282583] 7d38 00000003 d134fc80 d134fc00 ffffff80 00000001 00000000 00000002 00100100
[ 32.290843] 7d58 00200200 ffffffff d1360294 dfaf7d78 000007e0 c0099a9c 800f0093 ffffffff
[ 32.299102] 7d78 00000000 00000006 d14034c8 00000003 00000006 00000002 c071bc64 c064e900
[ 32.307362] 7d98 d14034ac c0683aa8 0000000f dfaf6038 c0683410 d136fc20 00000001 d14034ac
[ 32.315621] 7db8 800f0013 600f0013 dfaf6000 00100100 00000000 c009ab68 d136fc34 d136fc60
[ 32.323882] 7dd8 dfaf7e10 00000000 0000000e c009b2cc d136f9e0 0000000e d801b7dc c009ebb4
[ 32.332142]
[ 32.332144] IP: 0xd1360214:
[ 32.336416] 0214 d13601f4 d1360234 00000000 40080068 dfabc721 000b3b66 00000000 00000001
[ 32.344676] 0234 d1360214 d13613f4 00000000 40000000 00000000 000b3953 ffffff80 00000000
[ 32.352936] 0254 d13e52d4 d1360414 00000001 40000000 00000000 000b2e3c ffffffff 00000000
[ 32.361195] 0274 00100100 00200200 00000000 40000000 00000000 000b3952 ffffff80 00000000
[ 32.369456] 0294 d13f9094 000007e0 00000002 40000000 00000000 000b2e3d ffffffff 00000000
[ 32.377716] 02b4 00100100 00200200 00000000 40000000 00000000 000b3951 ffffffff 00000000
[ 32.385976] 02d4 00100100 00200200 00000002 40000000 00000000 000b2e3e ffffffff 00000000
[ 32.394235] 02f4 00100100 00200200 00000000 40000000 00000000 000b3950 ffffff80 00000000
[ 32.402496]
[ 32.402498] R0: 0xc0683ac4:
[ 32.406770] 3ac4 00000000 0000021c d14034ac 00000000 00000000 d13003b4 d13003b4 d13e8334
[ 32.415030] 3ae4 d13e8334 d1346314 d1346254 c0683af0 c0683af0 c0683af8 c0683af8 c0683b00
[ 32.423290] 3b04 c0683b00 000008c1 d13003d4 d13003d4 d13e8454 d13e8454 d134f8d4 d131a894
[ 32.431550] 3b24 c0683b24 c0683b24 c0683b2c c0683b2c c0683b34 c0683b34 00000081 c0683b40
[ 32.439810] 3b44 c0683b40 d13e8494 d13e8494 d134fc94 d133df14 c0683b58 c0683b58 c0683b60
[ 32.448070] 3b64 c0683b60 c0683b68 c0683b68 0000000d c0683b74 c0683b74 d13e8514 d13e8514
[ 32.456330] 3b84 d134f914 d12ede14 c0683b8c c0683b8c c0683b94 c0683b94 c0683b9c c0683b9c
[ 32.464590] 3ba4 00000011 c0683ba8 c0683ba8 d13e8614 d13e8614 d134fa14 d12a8a14 c0683bc0
[ 32.472850]
[ 32.472853] R3: 0xd134fc00:
[ 32.477125] fc00 40000000 00000000 000b3972 ffffffff 00000000 00100100 00200200 00000002
[ 32.485385] fc20 40000000 00000000 000b2e1d ffffffff 00000000 00100100 00200200 00000000
[ 32.493644] fc40 40000000 00000000 000b3971 ffffffff 00000000 00100100 00200200 00000000
[ 32.501904] fc60 40000000 00000000 000b2e1e ffffffff 00000000 00100100 00200200 00000000
[ 32.510164] fc80 40000000 00000000 000b3970 ffffff80 00000000 d1360294 c0683b50 00000002
[ 32.518423] fca0 40000000 00000000 000b2e1f ffffffff 00000000 00100100 00200200 00000000
[ 32.526683] fcc0 40000000 00000000 000b396f ffffffff 00000000 00100100 00200200 00000002
[ 32.534942] fce0 40000000 00000000 000b2e20 ffffffff 00000000 00100100 00200200 00000000
[ 32.543202]
[ 32.543205] R4: 0xd134fb80:
[ 32.547477] fb80 40000000 00000000 000b3974 ffffffff 00000000 00100100 00200200 00000000
[ 32.555736] fba0 40000000 00000000 000b2e1b ffffffff 00000000 00100100 00200200 00000000
[ 32.563996] fbc0 40000000 00000000 000b3973 ffffffff 00000000 00100100 00200200 00000002
[ 32.572256] fbe0 40000000 00000000 000b2e1c ffffffff 00000000 00100100 00200200 00000000
[ 32.580515] fc00 40000000 00000000 000b3972 ffffffff 00000000 00100100 00200200 00000002
[ 32.588775] fc20 40000000 00000000 000b2e1d ffffffff 00000000 00100100 00200200 00000000
[ 32.597034] fc40 40000000 00000000 000b3971 ffffffff 00000000 00100100 00200200 00000000
[ 32.605294] fc60 40000000 00000000 000b2e1e ffffffff 00000000 00100100 00200200 00000000
[ 32.613554]
[ 32.613557] R5: 0xffffff00:
[ 32.617828] ff00 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.626091] ff20 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.634352] ff40 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.642613] ff60 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.650873] ff80 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.659134] ffa0 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.667395] ffc0 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.675657] ffe0 ******** ******** ******** ******** ******** ******** ******** ********
[ 32.683920] Process mem-test (pid: 55, stack limit = 0xdfaf62e8)
[ 32.689919] Stack: (0xdfaf7d78 to 0xdfaf8000)
[ 32.694272] 7d60: 00000000 00000006
[ 32.702442] 7d80: d14034c8 00000003 00000006 00000002 c071bc64 c064e900 d14034ac c0683aa8
[ 32.710612] 7da0: 0000000f dfaf6038 c0683410 d136fc20 00000001 d14034ac 800f0013 600f0013
[ 32.718781] 7dc0: dfaf6000 00100100 00000000 c009ab68 d136fc34 d136fc60 dfaf7e10 00000000
[ 32.726951] 7de0: 0000000e c009b2cc d136f9e0 0000000e d801b7dc c009ebb4 c009e328 c068375c
[ 32.735120] 7e00: c0718f8c 00200200 dfaf6038 00000000 d136f9f4 d136fd34 d801b000 0000000e
[ 32.743290] 7e20: d136f9e0 d801b7dc 0000000e 00000216 d801b7e0 dfaf7f00 b3bdb000 c00bf604
[ 32.751459] 7e40: bf000000 00000000 dfbb676c d1386260 b3c00000 dfbaf630 00000000 c00af904
[ 32.759629] 7e60: c0684490 00000000 5c31334f dfaf6020 dfb1acec dfabf300 dfaf6000 c071bc64
[ 32.767798] 7e80: c064e900 b3fb4000 dfb1ace8 b3fb3fff 00000001 00000000 fffffe24 00000000
[ 32.775968] 7ea0: dfaf6010 dfbaf630 b3fb4000 dfaf7f00 b27dc000 dfaf7efc 00000000 b27dc000
[ 32.784138] 7ec0: dfbaf630 c00b0d00 dfaf7efc 00000000 dfae3d68 dfae39a0 dfaf7f20 dfbaf630
[ 32.792307] 7ee0: dfae3d68 dfabf300 b3fb4000 c00b3ec4 dfaf7efc 00000000 00000000 000017d8
[ 32.800477] 7f00: dfabf300 00000000 dfbaf630 bf000000 00000000 00000400 00000400 d801b000
[ 32.808646] 7f20: b27dc000 00000000 00000000 dfabf300 00000000 c00b4b14 dfbaf630 dfaf6018
[ 32.816816] 7f40: dfabf300 dfbaf630 b3fb4000 b27dc000 dfae39a0 dfabf304 dfae39ac c00b50dc
[ 32.824986] 7f60: b3fb4000 00000000 00000000 dfabf338 dfabf300 017d8000 b27dc000 c000e068
[ 32.833156] 7f80: dfaf6000 00000000 00000000 c00b5200 0006c4c4 0006c93c 0a6e8000 0bec0000
[ 32.841325] 7fa0: 0000005b c000dec0 0006c93c 0a6e8000 b27dc000 017d8000 0bec0000 0006c4c4
[ 32.849495] 7fc0: 0006c93c 0a6e8000 0bec0000 0000005b be990d38 00000000 00009031 00000000
[ 32.857665] 7fe0: b578c00d be990ce4 00012103 00019a76 600f0030 b27dc000 5ff9d821 5ff9dc21
[ 32.865850] [<c0099a9c>] (free_pcppages_bulk+0x1ac/0x39c) from [<c009ab68>] (free_hot_cold_page+0x10c/0x19c)
[ 32.875671] [<c009ab68>] (free_hot_cold_page+0x10c/0x19c) from [<c009b2cc>] (free_hot_cold_page_list+0x2c/0x48)
[ 32.885754] [<c009b2cc>] (free_hot_cold_page_list+0x2c/0x48) from [<c009ebb4>] (release_pages+0x230/0x27c)
[ 32.895405] [<c009ebb4>] (release_pages+0x230/0x27c) from [<c00bf604>] (free_pages_and_swap_cache+0xb0/0xbc)
[ 32.905234] [<c00bf604>] (free_pages_and_swap_cache+0xb0/0xbc) from [<c00af904>] (unmap_single_vma+0x3c8/0x628)
[ 32.915315] [<c00af904>] (unmap_single_vma+0x3c8/0x628) from [<c00b0d00>] (unmap_vmas+0x5c/0x70)
[ 32.924098] [<c00b0d00>] (unmap_vmas+0x5c/0x70) from [<c00b3ec4>] (unmap_region+0xa8/0x1ec)
[ 32.932445] [<c00b3ec4>] (unmap_region+0xa8/0x1ec) from [<c00b50dc>] (do_munmap+0x1e4/0x2cc)
[ 32.940877] [<c00b50dc>] (do_munmap+0x1e4/0x2cc) from [<c00b5200>] (vm_munmap+0x3c/0x50)
[ 32.948969] [<c00b5200>] (vm_munmap+0x3c/0x50) from [<c000dec0>] (ret_fast_syscall+0x0/0x30)
[ 32.957400] Code: e2822001 e5977010 e352000a e58ce004 (e58ec000)
[ 32.973527] ---[ end trace 807de48ca43a71c0 ]---
[ 32.978146] note: mem-test[55] exited with preempt_count 1
Address 10: [0xa811c008]
Address 2 OK.
Address 1 OK.
Address 10: [0xa8135008]
Address 2 OK.
Address 10: [0xa813b008]
Address 1 OK.
Address 2 OK.
Address 1 OK.
Address 1 OK.
Address 3 OK.
Address 2 OK.
Address 1 OK.
Address 3 OK.
Address 1 OK.
Address 2 OK.
Address 3 OK.
Address 2 OK.
Address 2 OK.
Address 4 OK.
Address 3 OK.
Address 2 OK.
Address 4 OK.
Address 2 OK.
Address 3 OK.
Address 3 OK.
Address 4 OK.
Address 3 OK.
Address 5 OK.
Address 4 OK.
Address 3 OK.
Address 5 OK.
Address 3 OK.
Address 4 OK.
Address 4 OK.
Address 5 OK.
Address 4 OK.
Address 5 OK.
Address 6 OK.
Address 4 OK.
Address 6 OK.
Address 4 OK.
Address 5 OK.
Address 5 OK.
Address 6 OK.
Address 5 OK.
Address 6 OK.
Address 7 OK.
Address 5 OK.
Address 7 OK.
Address 5 OK.
Address 6 OK.
Address 6 OK.
Address 7 OK.
Address 6 OK.
Address 7 OK.
Address 8 OK.
Address 6 OK.
Address 8 OK.
Address 6 OK.
Address 7 OK.
Address 7 OK.
Address 8 OK.
Address 7 OK.
Address 8 OK.
Address 7 OK.
Address 9 OK.
Address 10 OK.Address 9 OK.
Address 10 OK.Address 7 OK.
Address 8 OK.
Address 8 OK.
Address 9 OK.
Address 10 OK.[ 64.294653] BUG: Bad rss-counter state mm:dfabfc00 idx:1 val:13
Address 8 OK.
Address 9 OK.
Address 8 OK.
Address 10 OK.Address 8 OK.
Address 9 OK.
Address 10 OK.Address 9 OK.
Address 10 OK.Address 9 OK.
Address 10 OK.Address 9 OK.
Address 10 OK.Address 9 OK.

surprisingly, our program is running after oops, but our kernel is actually crashed and do no reply.
it also should be noted that this binary run properly in marsboard.

please help me. it is a hardware error, or it is origin from mismatched fex [dram-para section]

i also now that in all caces:
PC is at free_pcppages_bulk+0x1ac/0x39c
and lr register is incorrect

Hi
I want to use c11 core board but I cant find any data about linux and fex file for this board.can you help me?I already work with cubieboard.

Originally Posted by jalila211 Hi I want to use c11 core board but I cant find any data about linux and fex file for this board.can you help me?I already work with cubieboard.

although all the resources are accessible in this product (i.e. 8 UART 4 sdcard controller , ...), i strongly recommend that DONOT USE THIS CORE BOARD due to the nonstandard connector pitch (i.e. 1.1mm). moreover JTAG IS NOT AVAILABLE and you can only debug the final product via terminal/kernel-hacking features. however, it is good for evaluating/debugging sample boards.

and finally, about fex file:
this file has not a fix features and MUST BE MATCHED WITH YOUR FINAL DESIGN. it is strongly recommended that use a sample and customize it in according to your design. bellow, good information can be found (from kernel supplier of ALLWINNER co. i.e. sunxi)
http://linux-sunxi.org/Fex_Guide

it is my kernel message during boot-process:
Starting kernel ...

Booting Linux on physical CPU 0
Initializing cgroup subsys cpuset
Linux version 3.4.103-g9a1cd03 (root@sap-Inspiron-N5010) (gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4) ) #13 PREEMPT Fri Dec 19 17:13:34 IRS4
CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: sun4i
Memory Reserved:
SYS : 0x43000000 - 0x4300ffff ( 64 kB)
G2D : 0x50000000 - 0x50ffffff ( 16 MB)
cma: CMA: reserved 192 MiB at 43c00000
Memory policy: ECC disabled, Data cache writeback
sunxi: Allwinner A10 revision C (AW1623/sun4i) detected.
On node 0 totalpages: 131072
free_area_init_node: node 0, pgdat c06b32f4, node_mem_map d1000000
DMA zone: 512 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 65024 pages, LIFO batch:15
Normal zone: 512 pages used for memmap
Normal zone: 65024 pages, LIFO batch:15
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: console=ttyS0,115200 root=/dev/mmcblk0p2 rootwait loglevel=8 panic=10
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 512MB = 512MB total
Memory: 298784k/298784k available, 225504k reserved, 0K highmem
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc064b000 (6412 kB)
.init : 0xc064b000 - 0xc0674000 ( 164 kB)
.data : 0xc0674000 - 0xc06b41c0 ( 257 kB)
.bss : 0xc06b41e4 - 0xc07577c0 ( 654 kB)

as seen, i have only 512 MB of ram but adding vmalloc+lomwm is approximately 1GB. I suspect that our kernel configuration (kernel split 1G/3G kernel/user space). How can I change vmalloc/lowmem size?

any other information is needed?
Please help

thanks for your answer but i bought the card and now the problem is i cant find any pin number on pcb. there isnt any data about pin number on datasheet.

It seems to me that memory is allocated, but that malloc() is not checked for returning a null pointer. And successively this null pointer is being used.

GDB would show the call stack trace, but since this is happening in kernel/driver space, I am not sure you can use those debugging tools.

jlinkels

Originally Posted by jlinkels It seems to me that memory is allocated, but that malloc() is not checked for returning a null pointer. And successively this null pointer is being used. GDB would show the call stack trace, but since this is happening in kernel/driver space, I am not sure you can use those debugging tools. jlinkels

hi jlinkels


thank for your wise comment.
actually after testing that code, I find this point and rewrite the code as you mentioned . but the error still exists.
amazingly, Mars-board work OK but my board crashes.
again thank you so mush for your time

Originally Posted by jalila211 thanks for your answer but i bought the card and now the problem is i cant find any pin number on pcb. there isnt any data about pin number on datasheet.

Dear jalila211
I had hoped that you contribute in order to solve my problem . you can get such information from hardware supplier. I try to attache a PCB-doc to this reply but that file type is not supported. you can email me sap1359@gmail.com .

PROBLEM IS SOLVED.


Problem was neither KERNEL COMPILTION nor CORE BOARD ISSUES. It was approximately 0.1 volt . I used LINEAR REGULATOR to produce 1.2v for internal/RTC logic. however other Embedded boards (such as MARS CUBIE , ...) use non-linear regulator withapproximately 1.3V. so core board could not work properly.