LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-01-2014, 05:25 PM   #1
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Rep: Reputation: Disabled
program got random crash on LINUX, big memory issue? LINUX bug?


A program got random crash on LINUX, the system was already set to ulimit unlimited memory (ulimit).

Same program works perfect on AIX.

Under LINUX, is there a compile flag similar to AIX bmaxdata?
Is there any known bug for this?

platform:
CentOS 4.8 Final
gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)
 
Old 12-01-2014, 06:28 PM   #2
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,627

Rep: Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651
well without knowing any of the needed information
like
WHAT is this unknown / unnamed program
what the UNKNOWN ERRORS are

there is no real way we can help

the only thing I can advise is for you to use a SUPPORTED operating system


you are running on a DEAD and VERY unsupported operating system

CentOS 4.8 went END OF LIFE back in march of 2011
even the LAST upgrade CentOS 4.9 is dead and UNSUPPORTED
4.9 went END of LIFE on 29 February 2012


install one of the currently supported versions of cent
the OLDER legacy hardware/ software Operating system

CentOS 5.11

the current stable
CentOS 6.6

and the just released NEW version
CentOS 7.0

ONLY!!!!
those 3 versions are supported
5.10 IS NOT
4.9 IS NOT
4.8 IS NOT
 
Old 12-01-2014, 07:30 PM   #3
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
The core dumps were in socket layer, should be relating to resource.
under LINUX, is there a compile flag similar to AIX bmaxdata?
 
Old 12-01-2014, 08:04 PM   #4
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,627

Rep: Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651
Quote:
is there a compile flag similar to AIX bmaxdata
for the completely UNSUPPORTED cent 4.8

a operating system that is missing 3 YEARS of security updates

running the 32 bit 4.8 on new 64 bit hardware will well , not work well

the gcc flag would be ( m32 or m64 )
but autotools should auto detect that and auto set it

you would need a 64 bit version of cent for using 64 bit memory registers
and 4.8 and 4.9 are both DEAD
this is a quote for the "place holder" on the cent os mirrors
Quote:
This directory (and version of CentOS) is depreciated. For normal users,
you should use /4/ and not /4.8/ in your path. Please see this FAQ
concerning the CentOS release scheme:

http://www.centos.org/modules/smartfaq/faq.php?faqid=34

If you know what you are doing, and absolutely want to remain at the 4.8
level, go to http://vault.centos.org/ for packages.

install the currently supported 64 BIT CentOS 6.6 or 7.0
 
Old 12-05-2014, 07:43 PM   #5
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
platform info: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
(gcc version 4.1.2 20080704 (Red Hat 4.1.2-50))

the strace file looks like below:

09:01:20 write(3, "09:01:20 xxx..."..., 99) = 99
09:01:20 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
09:01:20 gettimeofday({1417766480, 285873}, NULL) = 0
09:01:20 rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
09:01:20 uname({sys="Linux", node="xxx01", ...}) = 0
09:01:20 rt_sigprocmask(SIG_SETMASK, ~[BUS SEGV RTMIN RT_1], [], 8) = 0
09:01:20 rt_sigaction(SIGBUS, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGBUS, {0x56c47434, [ALRM], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigaction(SIGSEGV, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGSEGV, {0x56c47434, [ALRM], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigprocmask(SIG_BLOCK, NULL, ~[BUS KILL SEGV STOP RTMIN RT_1], 8) = 0
09:01:20 rt_sigaction(SIGBUS, {0x56c536e8, [], SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigaction(SIGSEGV, {0x56c536e8, [], SA_SIGINFO}, NULL, 8) = 0
09:01:20 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
...
09:01:20 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
09:01:20 open("/u01/app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1", O_RDONLY|O_LARGEFILE) = 30
09:01:20 mmap2(NULL, 4096, PROT_READ, MAP_SHARED, 30, 0x26fb) = 0x5755a000
...
09:01:20 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
09:01:20 munmap(0x5b152000, 2691072) = 0
09:01:20 close(30) = 0
09:01:20 munmap(0x580fd000, 294912) = 0
09:01:20 close(29) = 0
09:01:20 gettimeofday({1417766480, 648120}, NULL) = 0
09:01:20 mkdir("/u01/app/oracle/oradiag_xxx", 0775) = -1 EACCES (Permission denied)
09:01:20 lseek(5, 4096, SEEK_SET) = 4096
09:01:20 read(5, "\n\0\5\274\0\0D\0\6\274\0\0u\0\7\274\0\0\254\0\10\274\0\0\311\0\t\274\0\0\346\0"..., 512) = 512
09:01:20 lseek(5, 7168, SEEK_SET) = 7168
09:01:20 read(5, "\v\0=\274\0\0J\0>\274\0\0o\0?\274\0\0\214\0@\274\0\0\303\0A\274\0\0\343\0"..., 512) = 512
09:01:20 rt_sigprocmask(SIG_SETMASK, ~[BUS SEGV RTMIN RT_1], [], 8) = 0
...
09:01:20 rt_sigaction(SIGXCPU, NULL, NULL, 8) = 0
09:01:20 rt_sigaction(SIGXCPU, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGXCPU, {SIG_DFL, [], 0}, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGXFSZ, NULL, NULL, 8) = 0
09:01:20 rt_sigaction(SIGXFSZ, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 rt_sigaction(SIGXFSZ, {SIG_DFL, [], 0}, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
09:01:20 tgkill(25756, 25756, SIGSEGV) = 0
09:01:20 --- SIGSEGV (Segmentation fault) @ 0 (0) ---

Oracle permission denied? does it matter?
 
Old 02-11-2015, 05:06 PM   #6
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
the latest crash strace file:
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 fcntl64(10, F_SETLK, {type=F_WRLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 read(10, "6\1\0\0\370\4\0\0\1\0\0\0t\361\2\0\0\4*\0\1\0\0\0'\1\0\0\332\4\0\0"..., 1272) = 1272
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 fcntl64(11, F_SETLK, {type=F_WRLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 read(11, "6\1\0\0\2601\0\0\10\0\0\0\316%\10\0\0\0*\0\n\0\0\0%\0\0\0\254\0\0\0"..., 12720) = 12720
09:04:54 lseek(11, 20538, SEEK_SET) = 20538
09:04:54 read(11, "\22\2346\0\0\202~\16\0\0030003000004871341\\Z\5\0\351\5"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 read(11, "\25\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 20538, SEEK_SET) = 20538
09:04:54 read(11, "\22\2346\0\0\202~\16\0\0030003000004871341\\Z\5\0\351\5"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 read(11, "\25\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 write(11, "\24\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 20538, SEEK_SET) = 20538
09:04:54 read(11, "\22\2346\0\0\202~\16\0\0030003000004871341\\Z\5\0\351\5"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 read(11, "\24\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 178306, SEEK_SET) = 178306
09:04:54 write(11, "\25\0\0\0\0v~\16\0\0050003000004871329\0\0\0\0&\3"..., 702) = 702
09:04:54 lseek(11, 21846, SEEK_SET) = 21846
09:04:54 read(11, "\22Z9\0\0\213~\16\0\3INTERNAL\0\0\0\0\2661\3\0j\3\0\0\237~"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 read(11, "\23\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 21846, SEEK_SET) = 21846
09:04:54 read(11, "\22Z9\0\0\213~\16\0\3INTERNAL\0\0\0\0\2661\3\0j\3\0\0\237~"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 read(11, "\23\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 write(11, "\22\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 21846, SEEK_SET) = 21846
09:04:54 read(11, "\22Z9\0\0\213~\16\0\3INTERNAL\0\0\0\0\2661\3\0j\3\0\0\237~"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 read(11, "\22\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(11, 89962, SEEK_SET) = 89962
09:04:54 write(11, "\23\0\0\0\0Q\0\0\0\5RSEBTS\0\0\0\0\0\0\0\0\0\0Q\0\0\0p^"..., 606) = 606
09:04:54 lseek(10, 75704, SEEK_SET) = 75704
09:04:54 write(10, "\277~\16\0\10\0\0\0000003000004871402\0\0\0\0\0\0\0\0"..., 124) = 124
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 write(11, "6\1\0\0\2601\0\0\10\0\0\0\316%\10\0\0\0*\0\n\0\0\0%\0\0\0\254\0\0\0"..., 12720) = 12720
09:04:54 lseek(11, 0, SEEK_SET) = 0
09:04:54 fcntl64(11, F_SETLK, {type=F_UNLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 write(10, "6\1\0\0\370\4\0\0\1\0\0\0t\361\2\0\0\4*\0\1\0\0\0'\1\0\0\332\4\0\0"..., 1272) = 1272
09:04:54 lseek(10, 0, SEEK_SET) = 0
09:04:54 fcntl64(10, F_SETLK, {type=F_UNLCK, whence=SEEK_CUR, start=0, len=1}) = 0
09:04:54 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
09:04:54 gettimeofday({1423209894, 437334}, NULL) = 0
09:04:54 rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
09:04:54 uname({sys="Linux", node="xxx", ...}) = 0
09:04:54 rt_sigprocmask(SIG_SETMASK, ~[BUS SEGV RTMIN RT_1], [], 8) = 0
09:04:54 rt_sigaction(SIGBUS, NULL, {0x56c536e8, [], SA_SIGINFO}, 8) = 0
 
Old 02-11-2015, 08:23 PM   #7
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
Is Linux 2.6.18-308.el5 a stable version?
 
Old 02-12-2015, 09:59 AM   #8
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Quote:
Originally Posted by IndianScorpion View Post
Is Linux 2.6.18-308.el5 a stable version?
It was 2 years ago.

RHEL 5.8 is 2 years old and version 5 is not receiving any substantial updates, it's in the phase where they offer discretionary releases; meaning they're supporting it per their discretion but not really working extensively at it and if they find something difficult to integrate into that release, they do not include it.

RHEL main releases are up to 7 as of a year ago.

You went from a very old CentOS to an old RHEL.

That's not helping really.

If you actually own RHEL 5.8, ask RedHat for support on this matter.
 
Old 02-12-2015, 12:15 PM   #9
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
Thanks @rtmistler, I will try.
the binary was built on 32bit
Linux xxx 2.6.16.60-0.21-default #1 Tue May 6 12:41:02 UTC 2008 i686 i686 i386 GNU/Linux
and running on 64 bit
Linux 2.6.18-308.el5
 
Old 02-12-2015, 12:21 PM   #10
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Quote:
Originally Posted by IndianScorpion View Post
Thanks @rtmistler, I will try.
the binary was built on 32bit
Linux xxx 2.6.16.60-0.21-default #1 Tue May 6 12:41:02 UTC 2008 i686 i686 i386 GNU/Linux
and running on 64 bit
Linux 2.6.18-308.el5
So ... you're compiling this program? Or have compiled it?

If you have the source, build it on the actual target system you intend to run it on. That likely will solve almost all of these problems.

Use ulimit -c unlimited to cause core file dumps

When you compile, use -ggdb to generate a GDB debug capable image

If/When it next crashes given those prior recommendations of ulimit and GDB compile flag, you can do two or more things:
  1. Analyze the core file using GDB
  2. Debug the program by running it out of the debugger
  3. Editing the program to add lots of debug showing progress as it loads up and begins, or operates
 
Old 02-12-2015, 12:29 PM   #11
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
rtmistler, yes, I compiled it. and I have done all these steps you mentioned, ulimited, gdb bt, strace.
The core dump and strace file was showing that it crashed on kernel socket layer to receive a packet.
 
Old 02-12-2015, 12:38 PM   #12
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Quote:
Originally Posted by IndianScorpion View Post
rtmistler, yes, I compiled it. and I have done all these steps you mentioned, ulimited, gdb bt, strace.
The core dump and strace file was showing that it crashed on kernel socket layer to receive a packet.
I see two instances of segmentation violations, but no backtrace. You can run this program in GDB and wait for it to SEGV and then do backtrace to see where the program backtrace. Seeing only the list of system calls via strace doesn't get you much here. For instance if you have a NULL pointer, then of course it's going to fail in a system call, but that doesn't mean it had anything to do with that particular system call.
 
Old 02-12-2015, 02:13 PM   #13
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
another running machine is
Linux xxx 2.6.9-89.ELlargesmp #1 SMP Mon Jun 22 12:46:58 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
 
Old 02-12-2015, 02:16 PM   #14
IndianScorpion
LQ Newbie
 
Registered: Dec 2014
Posts: 24

Original Poster
Rep: Reputation: Disabled
rtmistler, the backtrace is as below:
Program terminated with signal 11, Segmentation fault.
#0 0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x003c5df0 in ?? ()
#2 0x57440300 in ?? ()
#3 0x0000000b in ?? ()
#4 0xffe1003c in ?? ()
#5 0x56c53503 in ?? ()
#6 0x0000000b in ?? ()
#7 0xffe0ff60 in ?? ()
#8 0x56ebf98a in ?? ()
#9 0x00000000 in ?? ()
 
Old 02-12-2015, 02:27 PM   #15
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Nothing more to that backtrace? Does it go below #9? Or has this not been properly compiled using -ggdb flag?

For instance here's an example of code containing an obvious segment violation and the result of a debug session:
Code:
#include <stdio.h>
#include <string.h>

int main(int argc, char **argv)
{
    int *bad_addr = NULL;

    *bad_addr = 12345;

    return 0;
}
Code:
testcode$ gcc -ggdb -o segv segv.c ## Compilation example using -ggdb flag, if you don't do this then you're not going to see symbols and thus backtrace isn't going to get you any useful information
testcode$ gdb segv
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from ~testcode/segv...done. ## Note here that when loading GDB reads the symbols from the binary image, this needs to happen otherwise you see nothing of merit
(gdb) r
Starting program: ~testcode/segv 

Program received signal SIGSEGV, Segmentation fault.
0x080483c4 in main (argc=1, argv=0xbffff3a4) at segv.c:8
8	    *bad_addr = 12345;
(gdb) bt ## Backtrace here is very small, but shows that in segv.c at line 8 was the last program execution point
#0  0x080483c4 in main (argc=1, argv=0xbffff3a4) at segv.c:8
(gdb) p bad_addr
$1 = (int *) 0x0 ## And here's the reason why the segment violation occurred
(gdb)
In short, I feel that your backtrace should show symbols. Are you not seeing the symbols when you start GDB?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Random DVD crash issue mint 14 Linux2 Linux Mint 3 04-30-2013 02:38 AM
Why does linux reboot at Random when running my USB communication program Nightbird Linux - Software 3 09-02-2004 07:38 PM
Linux Cluster - Random Node Crash ! insanecrac007 Linux - General 0 08-17-2004 10:30 AM
Random program crashes in Linux, is fedora not stable? xbennyboy Linux - Newbie 13 08-05-2004 11:15 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:28 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration