Hi,

This is about an oracle application server running on RHEL platform(Linux prod.xyz.com 2.6.18-194.el5PAE #1 SMP Mon Mar 29 20:19:03 EDT 2010 i686 i686 i386 GNU/Linux). All of a sudden all the application services shut down abnormally on weekend. On the first day of the week, business users were unable to login as a result. For root cause investigation, we were unable to findout anything in the application logs. But were trying to get reboot/shutdown information from the system.....but we are bit confused with the info in /var/log/wtmp by using 'last' command(output given bellow):

[root@prod ~]# last reboot |head -2
reboot system boot 2.6.18-194.el5PA Mon Aug 8 11:26 (1+03:10)
reboot system boot 2.6.18-194.el5PA Sun Aug 7 17:27 (17:55)


We doubt the server might have gone down on Sunday(7th Aug) but there's also an entry of 8th Aug, on which day the server was running fine, no reboot/shutdown happened.
Kindly review the above output and suggest.

Also please highlight on this, if there any other way of getting info on reboot timings

Hi,

You can try to search for syslogd restart in /var/log/messages , it happens every time when a server is rebooted.

Hi,

I've checked the /var/log/message and found the syslogd entry on both 7th and 8th August, as given below:

Aug 7 04:02:20 PROD94 syslogd 1.4.1: restart.
Aug 7 04:51:22 PROD94 xinetd[3196]: START: vnetd pid=20292 from=192.168.32.52
Aug 7 04:51:25 PROD94 xinetd[3196]: START: vnetd pid=20353 from=192.168.32.52
Aug 7 04:51:25 PROD94 xinetd[3196]: EXIT: vnetd status=0 pid=20353 duration=0(sec)
Aug 7 04:51:27 PROD94 xinetd[3196]: EXIT: vnetd status=0 pid=20292 duration=5(sec)
Aug 7 17:27:21 PROD94 syslogd 1.4.1: restart.
Aug 7 17:27:21 PROD94 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Aug 7 17:27:21 PROD94 kernel: Linux version 2.6.18-194.el5PAE (mockbuild@ca-build10.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Mon Mar 29 20:19:03 EDT 2010
Aug 7 17:27:21 PROD94 kernel: BIOS-provided physical RAM map:
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 0000000000010000 - 000000000009f400 (usable)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 0000000000100000 - 00000000df63f000 (usable)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 00000000df63f000 - 00000000df64c000 (ACPI data)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 00000000df64c000 - 00000000df64d000 (usable)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 00000000df64d000 - 00000000e4000000 (reserved)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
Aug 7 17:27:21 PROD94 kernel: BIOS-e820: 0000000100000000 - 000000041ffff000 (usable)
Aug 7 17:27:21 PROD94 kernel: 15999MB HIGHMEM available.
Aug 7 17:27:21 PROD94 kernel: 896MB LOWMEM available.
Aug 7 17:27:21 PROD94 kernel: found SMP MP-table at 000f4f80
Aug 7 17:27:21 PROD94 kernel: NX (Execute Disable) protection: active
Aug 7 17:27:21 PROD94 kernel: DMI 2.6 present.
Aug 7 17:27:21 PROD94 kernel: Using APIC driver default
Aug 7 17:27:21 PROD94 kernel: ACPI: PM-Timer IO Port: 0x908
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Aug 7 17:27:21 PROD94 kernel: Processor #0 7:10 APIC version 21
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
Aug 7 17:27:21 PROD94 kernel: Processor #4 7:10 APIC version 21
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Aug 7 17:27:21 PROD94 kernel: Processor #2 7:10 APIC version 21
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
Aug 7 17:27:21 PROD94 kernel: Processor #6 7:10 APIC version 21
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] disabled)
Aug 7 17:27:21 PROD94 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)


Also the entry on 8th(The server was running all through the day, it's start of the week) as given below:

Aug 8 11:26:24 PROD94 syslogd 1.4.1: restart.
Aug 8 11:26:24 PROD94 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Aug 8 11:26:24 PROD94 kernel: Linux version 2.6.18-194.el5PAE (mockbuild@ca-build10.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Mon Mar 29 20:19:03 EDT 2010
Aug 8 11:26:24 PROD94 kernel: BIOS-provided physical RAM map:
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 0000000000010000 - 000000000009f400 (usable)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 0000000000100000 - 00000000df63f000 (usable)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 00000000df63f000 - 00000000df64c000 (ACPI data)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 00000000df64c000 - 00000000df64d000 (usable)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 00000000df64d000 - 00000000e4000000 (reserved)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
Aug 8 11:26:24 PROD94 kernel: BIOS-e820: 0000000100000000 - 000000041ffff000 (usable)
Aug 8 11:26:24 PROD94 kernel: 15999MB HIGHMEM available.
Aug 8 11:26:24 PROD94 kernel: 896MB LOWMEM available.
Aug 8 11:26:24 PROD94 kernel: found SMP MP-table at 000f4f80
Aug 8 11:26:24 PROD94 kernel: NX (Execute Disable) protection: active
Aug 8 11:26:24 PROD94 kernel: DMI 2.6 present.
Aug 8 11:26:24 PROD94 kernel: Using APIC driver default
Aug 8 11:26:24 PROD94 kernel: ACPI: PM-Timer IO Port: 0x908
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Aug 8 11:26:24 PROD94 kernel: Processor #0 7:10 APIC version 21
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
Aug 8 11:26:24 PROD94 kernel: Processor #4 7:10 APIC version 21
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Aug 8 11:26:24 PROD94 kernel: Processor #2 7:10 APIC version 21
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
Aug 8 11:26:24 PROD94 kernel: Processor #6 7:10 APIC version 21
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] disabled)
Aug 8 11:26:24 PROD94 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])

Here I'm little puzzled as on 8th Aug the server was up and running, but still the message file is showing syslogd entry on 8th.

Kindly suggest.

Thanks and regards,
Kosys