program got random crash on LINUX, big memory issue? LINUX bug?

IndianScorpion · 12-01-2014, 05:25 PM

A program got random crash on LINUX, the system was already set to ulimit unlimited memory (ulimit).

Same program works perfect on AIX.

Under LINUX, is there a compile flag similar to AIX bmaxdata?
Is there any known bug for this?

platform:
CentOS 4.8 Final
gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)

John VV · 12-01-2014, 06:28 PM

well without knowing any of the needed information
like
WHAT is this unknown / unnamed program
what the UNKNOWN ERRORS are

there is no real way we can help

the only thing I can advise is for you to use a SUPPORTED operating system

you are running on a DEAD and VERY unsupported operating system

CentOS 4.8 went END OF LIFE back in march of 2011
even the LAST upgrade CentOS 4.9 is dead and UNSUPPORTED
4.9 went END of LIFE on 29 February 2012

install one of the currently supported versions of cent
the OLDER legacy hardware/ software Operating system

CentOS 5.11

the current stable
CentOS 6.6

and the just released NEW version
CentOS 7.0

ONLY!!!!
those 3 versions are supported
5.10 IS NOT
4.9 IS NOT
4.8 IS NOT

IndianScorpion · 12-01-2014, 07:30 PM

The core dumps were in socket layer, should be relating to resource.
under LINUX, is there a compile flag similar to AIX bmaxdata?

John VV · 12-01-2014, 08:04 PM

Quote:

is there a compile flag similar to AIX bmaxdata

for the completely UNSUPPORTED cent 4.8

a operating system that is missing 3 YEARS of security updates

running the 32 bit 4.8 on new 64 bit hardware will well , not work well

the gcc flag would be ( m32 or m64 )
but autotools should auto detect that and auto set it

you would need a 64 bit version of cent for using 64 bit memory registers
and 4.8 and 4.9 are both DEAD
this is a quote for the "place holder" on the cent os mirrors

Quote:

This directory (and version of CentOS) is depreciated. For normal users,
you should use /4/ and not /4.8/ in your path. Please see this FAQ
concerning the CentOS release scheme:

http://www.centos.org/modules/smartfaq/faq.php?faqid=34

If you know what you are doing, and absolutely want to remain at the 4.8
level, go to http://vault.centos.org/ for packages.

install the currently supported 64 BIT CentOS 6.6 or 7.0

IndianScorpion · 12-05-2014, 07:43 PM

platform info: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
(gcc version 4.1.2 20080704 (Red Hat 4.1.2-50))

the strace file looks like below:

09:01:20 write(3, "09:01:20 xxx..."..., 99) = 99
09:01:20 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
09:01:20 gettimeofday({1417766480, 285873}, NULL) = 0
09:01:20 rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
09:01:20 uname({sys="Linux", node="xxx01", ...}) = 0
09:01:20 rt_sigprocmask(SIG_SETMASK, ~[BUS SEGV RTMIN RT_1], [], 8) = 0
09:01:20 rt_sigaction(SIGBUS, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGBUS, {0x56c47434, [ALRM], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigaction(SIGSEGV, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGSEGV, {0x56c47434, [ALRM], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigprocmask(SIG_BLOCK, NULL, ~[BUS KILL SEGV STOP RTMIN RT_1], 8) = 0
09:01:20 rt_sigaction(SIGBUS, {0x56c536e8, [], SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigaction(SIGSEGV, {0x56c536e8, [], SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
...
09:01:20 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
09:01:20 open("/u01/app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1", O_RDONLY|O_LARGEFILE) = 30
09:01:20 mmap2(NULL, 4096, PROT_READ, MAP_SHARED, 30, 0x26fb) = 0x5755a000
...
09:01:20 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
09:01:20 munmap(0x5b152000, 2691072) = 0
09:01:20 close(30) = 0
09:01:20 munmap(0x580fd000, 294912) = 0
09:01:20 close(29) = 0
09:01:20 gettimeofday({1417766480, 648120}, NULL) = 0
09:01:20 mkdir("/u01/app/oracle/oradiag_xxx", 0775) = -1 EACCES (Permission denied)
09:01:20 lseek(5, 4096, SEEK_SET) = 4096
09:01:20 read(5, "\n\0\5\274\0\0D\0\6\274\0\0u\0\7\274\0\0\254\0\10\274\0\0\311\0\t\274\0\0\346\0"..., 512) = 512
09:01:20 lseek(5, 7168, SEEK_SET) = 7168
09:01:20 read(5, "\v\0=\274\0\0J\0>\274\0\0o\0?\274\0\0\214\0@\274\0\0\303\0A\274\0\0\343\0"..., 512) = 512
09:01:20 rt_sigprocmask(SIG_SETMASK, ~[BUS SEGV RTMIN RT_1], [], 8) = 0
...
09:01:20 rt_sigaction(SIGXCPU, NULL, NULL, 8) = 0
09:01:20 rt_sigaction(SIGXCPU, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGXCPU, {SIG_DFL, [], 0}, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGXFSZ, NULL, NULL, 8) = 0
09:01:20 rt_sigaction(SIGXFSZ, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGXFSZ, {SIG_DFL, [], 0}, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 tgkill(25756, 25756, SIGSEGV) = 0
09:01:20 --- SIGSEGV (Segmentation fault) @ 0 (0) ---

Oracle permission denied? does it matter?

IndianScorpion · 02-11-2015, 05:06 PM

the latest crash strace file:
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 fcntl64(10, F_SETLK, {type=F_WRLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 read(10, "6\1\0\0\370\4\0\0\1\0\0\0t\361\2\0\0\4*\0\1\0\0\0'\1\0\0\332\4\0\0"..., 1272) = 1272
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 fcntl64(11, F_SETLK, {type=F_WRLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 read(11, "6\1\0\0\2601\0\0\10\0\0\0\316%\10\0\0\0*\0\n\0\0\0%\0\0\0\254\0\0\0"..., 12720) = 12720
09:04:54 lseek(11, 20538, SEEK_SET) = 20538
09:04:54 read(11, "\22\2346\0\0\202~\16\0\0030003000004871341\\Z\5\0\351\5"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 read(11, "\25\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 20538, SEEK_SET) = 20538
09:04:54 read(11, "\22\2346\0\0\202~\16\0\0030003000004871341\\Z\5\0\351\5"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 read(11, "\25\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 write(11, "\24\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 20538, SEEK_SET) = 20538
09:04:54 read(11, "\22\2346\0\0\202~\16\0\0030003000004871341\\Z\5\0\351\5"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 read(11, "\24\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 write(11, "\25\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 21846, SEEK_SET) = 21846
09:04:54 read(11, "\22Z9\0\0\213~\16\0\3INTERNAL\0\0\0\0\2661\3\0j\3\0\0\237~"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 read(11, "\23\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 21846, SEEK_SET) = 21846
09:04:54 read(11, "\22Z9\0\0\213~\16\0\3INTERNAL\0\0\0\0\2661\3\0j\3\0\0\237~"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 read(11, "\23\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 write(11, "\22\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 21846, SEEK_SET) = 21846
09:04:54 read(11, "\22Z9\0\0\213~\16\0\3INTERNAL\0\0\0\0\2661\3\0j\3\0\0\237~"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 read(11, "\22\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 write(11, "\23\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(10, 75704, SEEK_SET) = 75704
09:04:54 write(10, "\277~\16\0\10\0\0\0000003000004871402\0\0\0\0\0\0\0\0"..., 124) = 124
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 write(11, "6\1\0\0\2601\0\0\10\0\0\0\316%\10\0\0\0*\0\n\0\0\0%\0\0\0\254\0\0\0"..., 12720) = 12720
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 fcntl64(11, F_SETLK, {type=F_UNLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 write(10, "6\1\0\0\370\4\0\0\1\0\0\0t\361\2\0\0\4*\0\1\0\0\0'\1\0\0\332\4\0\0"..., 1272) = 1272
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 fcntl64(10, F_SETLK, {type=F_UNLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
09:04:54 gettimeofday({1423209894, 437334}, NULL) = 0
09:04:54 rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
09:04:54 uname({sys="Linux", node="xxx", ...}) = 0
09:04:54 rt_sigprocmask(SIG_SETMASK, ~[BUS SEGV RTMIN RT_1], [], 8) = 0
09:04:54 rt_sigaction(SIGBUS, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0

IndianScorpion · 02-11-2015, 08:23 PM

Is Linux 2.6.18-308.el5 a stable version?

rtmistler · 02-12-2015, 09:59 AM

Quote:

Originally Posted by IndianScorpion

Is Linux 2.6.18-308.el5 a stable version?

It was 2 years ago.

RHEL 5.8 is 2 years old and version 5 is not receiving any substantial updates, it's in the phase where they offer discretionary releases; meaning they're supporting it per their discretion but not really working extensively at it and if they find something difficult to integrate into that release, they do not include it.

RHEL main releases are up to 7 as of a year ago.

You went from a very old CentOS to an old RHEL.

That's not helping really.

If you actually own RHEL 5.8, ask RedHat for support on this matter.

IndianScorpion · 02-12-2015, 12:15 PM

Thanks @rtmistler, I will try.
the binary was built on 32bit
Linux xxx 2.6.16.60-0.21-default #1 Tue May 6 12:41:02 UTC 2008 i686 i686 i386 GNU/Linux
and running on 64 bit
Linux 2.6.18-308.el5

rtmistler · 02-12-2015, 12:21 PM

Quote:

Originally Posted by IndianScorpion

Thanks @rtmistler, I will try.
the binary was built on 32bit
Linux xxx 2.6.16.60-0.21-default #1 Tue May 6 12:41:02 UTC 2008 i686 i686 i386 GNU/Linux
and running on 64 bit
Linux 2.6.18-308.el5

So ... you're compiling this program? Or have compiled it?

If you have the source, build it on the actual target system you intend to run it on. That likely will solve almost all of these problems.

Use ulimit -c unlimited to cause core file dumps

When you compile, use -ggdb to generate a GDB debug capable image

If/When it next crashes given those prior recommendations of ulimit and GDB compile flag, you can do two or more things:

Analyze the core file using GDB
Debug the program by running it out of the debugger
Editing the program to add lots of debug showing progress as it loads up and begins, or operates

IndianScorpion · 02-12-2015, 12:29 PM

rtmistler, yes, I compiled it. and I have done all these steps you mentioned, ulimited, gdb bt, strace.
The core dump and strace file was showing that it crashed on kernel socket layer to receive a packet.

rtmistler · 02-12-2015, 12:38 PM

Quote:

Originally Posted by IndianScorpion

rtmistler, yes, I compiled it. and I have done all these steps you mentioned, ulimited, gdb bt, strace.
The core dump and strace file was showing that it crashed on kernel socket layer to receive a packet.

I see two instances of segmentation violations, but no backtrace. You can run this program in GDB and wait for it to SEGV and then do backtrace to see where the program backtrace. Seeing only the list of system calls via strace doesn't get you much here. For instance if you have a NULL pointer, then of course it's going to fail in a system call, but that doesn't mean it had anything to do with that particular system call.

IndianScorpion · 02-12-2015, 02:13 PM

another running machine is
Linux xxx 2.6.9-89.ELlargesmp #1 SMP Mon Jun 22 12:46:58 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

IndianScorpion · 02-12-2015, 02:16 PM

rtmistler, the backtrace is as below:
Program terminated with signal 11, Segmentation fault.
#0 0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x003c5df0 in ?? ()
#2 0x57440300 in ?? ()
#3 0x0000000b in ?? ()
#4 0xffe1003c in ?? ()
#5 0x56c53503 in ?? ()
#6 0x0000000b in ?? ()
#7 0xffe0ff60 in ?? ()
#8 0x56ebf98a in ?? ()
#9 0x00000000 in ?? ()

rtmistler · 02-12-2015, 02:27 PM

Nothing more to that backtrace? Does it go below #9? Or has this not been properly compiled using -ggdb flag?

For instance here's an example of code containing an obvious segment violation and the result of a debug session:

Code:

#include <stdio.h>
#include <string.h>

int main(int argc, char **argv)
{
    int *bad_addr = NULL;

    *bad_addr = 12345;

    return 0;
}

Code:

testcode$ gcc -ggdb -o segv segv.c ## Compilation example using -ggdb flag, if you don't do this then you're not going to see symbols and thus backtrace isn't going to get you any useful information
testcode$ gdb segv
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from ~testcode/segv...done. ## Note here that when loading GDB reads the symbols from the binary image, this needs to happen otherwise you see nothing of merit
(gdb) r
Starting program: ~testcode/segv 

Program received signal SIGSEGV, Segmentation fault.
0x080483c4 in main (argc=1, argv=0xbffff3a4) at segv.c:8
8	    *bad_addr = 12345;
(gdb) bt ## Backtrace here is very small, but shows that in segv.c at line 8 was the last program execution point
#0  0x080483c4 in main (argc=1, argv=0xbffff3a4) at segv.c:8
(gdb) p bad_addr
$1 = (int *) 0x0 ## And here's the reason why the segment violation occurred
(gdb)

In short, I feel that your backtrace should show symbols. Are you not seeing the symbols when you start GDB?