Latest LQ Deal: Linux Power User Bundle
Go Back > Forums > Linux Forums > Linux - Server
User Name
Linux - Server This forum is for the discussion of Linux Software used in a server related context.


  Search this Thread
Old 06-22-2009, 04:33 PM   #1
LQ Newbie
Registered: Apr 2009
Posts: 3

Rep: Reputation: 0
Issue using the crash utility on RHEL 4.x

Forced a crash dump as a test on a test machine in preparation for applying changes on a problematic server supporting an oracle database. Everything ok until I attempt to analyze the test crashdump with the drash utility. crash reports it cannot find the booted kernel. I have supplied the name of the running kernel and the fully qualified path to vmcore, as in "# crash /boot/vmlinuz-2.6.9-89.ELsmp /var/crash/"

Any insight into my issue would be greatly appreciated. Thank you.
Old 06-23-2009, 08:59 AM   #2
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 17,365

Rep: Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700
I'm not near a system setup for crash dump, but you need to have a debug kernel available. I seem to recall that message when I was playing around a year or so ago.
Old 06-23-2009, 08:17 PM   #3
LQ Newbie
Registered: Apr 2009
Posts: 3

Original Poster
Rep: Reputation: 0
Building a debug kernel vmlinux file (RHEL)

Thank you. Your answer provided just enough data for me to build a vmunix debug kernel file. For others who might follow these footsteps see the reprint below from The procedure is for RHEL2 but works just fine w/ RHEL4.

RHEL2.1 kernel rebuild procedure

To utilize crash with RHEL2.1 kernels, a vmlinux kernel must be built with the -g CFLAG. The newly-built kernel does not have to be run, but the vmlinux file with the extra debug data is required to run crash on a live system or with a dumpfile.

This is the suggested RHEL2.1 kernel build procedure:

1. Install the kernel src.rpm:

# rpm -ivh kernel-2.4.9-e.xx.src.rpm

where "xx" is the kernel version in use.

2. Go to /usr/src/redhat/SPECS, or wherever the builder's .rpmmacros file points to; /usr/src/redhat is the default build area.

3. Edit the kernel-2.4.spec file; the top few lines of which are shown below. At the top of the file, note that there are "%define buildxxx" settings, one for each kernel "flavor":

Summary: The Linux kernel (the core of the Linux operating system)

# What parts do we want to build? We must build at least one kernel.
# These are the kernels that are built IF the architecture allows and
# no contrary --with/--without arguments are given on the command line.
%define buildup 1
%define buildsmp 1
%define buildBOOT 1
%define buildenterprise 1
%define buildsummit 1
%define builddebug 1
%define buildjensen 0
%define buildtape 1
%define buildBOOTtape 1

To build only one kernel flavor, change all of the kernel "buildxxx" values to 0 -- except for the one that is to be debugged. So, for example, if you are debugging the "smp" kernel, leave buildsmp as 1, but change all of the others to be 0. Make sure that only a single instance of "1" exists, that one for the kernel type being debugged. For example:

%define buildup 0
%define buildsmp 1
%define buildBOOT 0
%define buildenterprise 0
%define buildsummit 0
%define builddebug 0
%define buildjensen 0
%define buildtape 0
%define buildBOOTtape 0

4. Then build the binary RPM:

# rpmbuild -bb --target i686 kernel-2.4.spec

This will build only one kernel RPM, that one for the kernel type specified by leaving its build value to 1.

5. Go into /usr/src/redhat/BUILD/kernel-2.4/linux (or wherever the builder's .rpmmacros file specifies), and edit the top-level Makefile there. Find the CFLAGS definition line that looks like this:

CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common -Wno-unused

Change it by adding -g, as in:

CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common -Wno-unused -g

6. Rebuild the kernel:

# make oldconfig dep bzImage

This will build the debug version of vmlinux, located in the top-level from where the above command was invoked. Copy it someplace safe, or re-name it so it won't get over-written if another build is done in this directory. For the purposes of the examples below, the debug vmlinux file has been re-named as vmlinux.dbg.

7. This debug vmlinux.dbg file can be use with crash, the vmcore file, and the /boot/ file to debug the kernel that crashed, as in:

# crash vmlinux.dbg vmcore /boot/

Or, to run crash on the live system, enter:

# crash vmlinux.dbg /boot/

In both cases above, the debug data is gathered from vmlinux.dbg file, but the actual symbol addresses are taken from the file associated with the "real" kernel. Again, also note that you don't have to actually run the debug kernel; it is only necessary that it matches the running/crashed kernel flavor, i.e., such as the "up", "smp" or "enterprise" flavors.

8. To debug a specific module with full debug data, rebuild it with -g. If the running version of the module can be unloaded with rmmod, do so, and then insmod the new debug module object file. If that is not possible, replace the module found in the appropriate /lib/modules tree with the new debug version, and reboot. Alternatively, all modules in the kernel source tree can be rebuilt with -g by running:

# make modules

after vmlinux.dbg debug kernel has been rebuilt in step 6 above. To copy them all to the appropriate /lib/modules tree, enter:

# make modules_install

Then you can load the new module's debug data while in a crash session with the mod command. Note that there is no crash utility support for loading the debug data from a module built with -g against a non-debug "stock" version of the module running in a live kernel or vmcore; the mod command will typically fail due to conflicting symbol values. However, it won't hurt to try it first.
Old 06-23-2009, 09:25 PM   #4
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 17,365

Rep: Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700Reputation: 2700
I'm surprised you needed to do a build. I was testing on Centos 5, and was able to simply download an appropriate (debug) kernel package - provided by Redhat.
Glad you got it sorted anyway.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
How to crash rhel (PSOD) amit23comp Red Hat 6 06-11-2009 12:38 PM
RHEL AS4UP2 64bit crash hunter_cao7 Linux - Enterprise 1 06-17-2008 02:55 AM
can anybody help me on taking backups using dump utility on RHEL 4 siri.siri143 Linux - Newbie 1 12-17-2007 10:12 AM
How do I setup my Red Hat ES3 system to use the 'crash' utility smur Linux - Newbie 3 07-10-2005 03:01 PM
analyzing crash dumps on rhel 3.0 u2 jazzn1doog Linux - Enterprise 0 07-28-2004 06:08 PM > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 07:49 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration