[SOLVED] How to create core dump when my application crashes?

kingston · 12-29-2010, 03:50 AM

Hi All

I am using RHEL 4.7 (32bit) on HP Proliant 380G6 series server. We are using Electric Cloud Agents on these servers. Nowadays we are facing some memory issues and its creates some kernel panic and then restarts the server. When i reported the issue to my application team, they asked me to come with the core dump. I googled it enough, then i set ulimit value as unlimited. (previously it was 0, then i made a entry in /etc/profile file as follows
ulimit -c unlimited) But still whenever my server restarts due to that kernel panic, it couldnt generate the core dump. My application was installed on /opt

The attached document has the kernel panic logs

ComputerErik · 12-29-2010, 09:26 PM

I don't think a core dump will help if the server is rebooting and kernel panic happening. As far as I have seen a core dump is just created when some application crashes, but the system is still stable and running. What you might want to look at doing is setting up for kernel dumps to get details on what is causing the kernel panic.

Also is there any reason you are running such an old version on new hardware? Have you checked and made sure that all of the drivers being used are certified on that hardware/OS, and all of your firmware is up to date? If these problems are only occurring on the DL380 G6 and you have other models in use it might be a hardware (or driver/firmware/etc) issue and not the application.

kingston · 12-29-2010, 11:54 PM

hi

Thanks for your information. Unfortunately i dont have any other models with working condition. But we have a newer version of servers (380G7 series). We will start using this within a week, after that i can observe whether the new servers also facing the same issues or not.
Can you share some documents for kernel dump configuration?
Thanks again for your valuable time and information

kingston · 04-29-2011, 12:04 AM

Hi All,

I configured netdump services to capture the core dump when my application crashes itself and reboots the server. The following is the procedure to configure the netdump services

Quote:

Netdump configuration requires two computers:
- One acts as the netdump server - the box where all crash logs will be stored;
- The other one acts as the netdump client - the VPS host machine.

I. Prepare boxes

1). Install the "netdump-server" package on the server and the "netdump" package on the client (VPS host).
You can use any update agent (up2date, yum, swup) or download and install packages manually via RPM.
For example:
# rpm -ivh netdump-0.7.14-4.i386.rpm
# rpm -ivh netdump-server-0.7.14-4.i386.rpm
2). Make sure that netdump servers partition where /var/crash directory located (by default the directory where all crashes will be stored) have enough free disk space, at least as your netdump client (VPS host) box RAM size.

II. Configuration

1). On the netdump server, as root, type:
# passwd netdump
and supply a password for netdump user just like what you do to an ordinary user. Then do the following:
# chkconfig netdump-server on
# service netdump-server start
2). On the netdump client (VPS host), edit /etc/sysconfig/netdump then uncomment and set the NETDUMPADDR variable to the IP address of the netdump server.
For example:
NETDUMPADDR=10.199.243.39
Then execute:
# service netdump propagate

and supply the netdump password that was configured on the netdump server. Finally, execute:
# chkconfig netdump on
# service netdump start

III. Testing (optional)

To test if the netdump configuration is correct, perform the following
on the netdump client:
# cp /usr/share/doc/netdump-xxxxxx/crash.c .
# gcc -DKERNEL -DMODULE -I/lib/modules/$(uname -r)/build/include -c crash.c
# insmod ./crash.o
This will crash the system and you will see a kernel dump on the netdump server in the directory /var/crash/<IP of client><date>/. You will see the file "vmcore-incomplete" while the client is dumping data to the server. The file is renamed to "vmcore" once it is completed.

The size of "vmcore" will vary and may reach several gigs. On a system
with 512Mb of RAM, the above test created a vmcore of approximately
510Mb.

Now my application team found the exact issues and they are in the plan to upgrade the application version so that the ongoing issue will be fixed...

Thanks to gurus and google....