boot log info needed!

xcr · 03-21-2006, 06:20 PM

hi all,
i am trying to get a log of when was the machine stopped and last rebooted using 'last -x' command(which helps me calculate for how long the machine was down), but here i see a reboot and again a reboot and then a crash. why is that?i am assuming that when a reboot occours then i should either get a crash or a shutdown. is this a wrong way of calculating the downtime? somebody please help me.thanks
root pts/0 ipmiha Tue Mar 21 17:07 still logged in
root pts/0 ipmiha Tue Mar 21 16:54 - 16:56 (00:01)
root pts/0 ipmiha Tue Mar 21 16:27 - 16:27 (00:00)
root pts/0 ipmiha Tue Mar 21 16:27 - 16:27 (00:00)
root pts/0 ipmiha Tue Mar 21 16:27 - 16:27 (00:00)
root pts/0 ipmiha Tue Mar 21 16:07 - 16:07 (00:00)
runlevel (to lvl 3) 2.4.20-8smp Tue Mar 21 15:59 - 17:07 (01:07)
reboot system boot 2.4.20-8smp Tue Mar 21 15:59 (01:07)
root pts/0 ipmiha Wed Mar 15 21:10 - 21:21 (00:10)
root pts/0 ipmiha Mon Mar 13 21:59 - 21:59 (00:00)
root pts/0 ipmiha Mon Mar 13 21:57 - 21:58 (00:00)
root pts/0 ipmiha Mon Mar 13 21:45 - 21:45 (00:00)
root pts/0 ipmiha Sat Mar 11 20:19 - 20:19 (00:00)
root pts/0 ipmiha Sat Mar 11 19:40 - 19:40 (00:00)
root pts/0 ipmiha Wed Mar 8 15:45 - 15:45 (00:00)
root pts/0 ipmiha Wed Mar 8 15:44 - 15:44 (00:00)
root pts/0 ipmiha Wed Mar 8 11:14 - 11:14 (00:00)
root pts/0 ipmiha Tue Mar 7 21:32 - 21:33 (00:00)
root pts/0 ipmiha Tue Mar 7 20:29 - 20:30 (00:01)
root pts/0 192.168.10.202 Tue Mar 7 19:56 - 19:56 (00:00)
root pts/0 ipmiha Tue Mar 7 19:32 - 19:33 (00:00)
root pts/0 ipmiha Tue Mar 7 14:57 - 14:58 (00:00)
root tty1 Tue Mar 7 14:56 - 14:57 (00:00)
runlevel (to lvl 3) 2.4.20-8smp Tue Mar 7 14:56 - 15:59 (14+01:02)
reboot system boot 2.4.20-8smp Tue Mar 7 14:56 (14+02:10)
root pts/0 ipmiha Tue Mar 7 14:51 - crash (00:05)
root pts/0 ipmiha Sun Mar 5 20:23 - 20:47 (00:23)
runlevel (to lvl 3) 2.4.20-8smp Thu Mar 2 17:39 - 14:56 (4+21:16)
reboot system boot 2.4.20-8smp Thu Mar 2 17:39 (18+23:27)

Simon Bridge · 03-21-2006, 06:54 PM

You're logging in only as root? Do I need to tell you how bad that is?

Anyway - looking at the log, I see:

March 2 17:39 Reboot
Runlevel changed to lvl 3
some work done until...
March 7 14:51 there was a crash.

same day, 5mins later:
March 7 14:56 - reboot (5mins downtime?)
Runlevel change to 3
working working working
your next system boot is on March 20th (well done)
the last terminal was opened March 15 - for about 10 mins. However, we cannot tell when the machine was shut down from this log. Presumably you care mostly about downtime from a crash?

Whatever, it seems every crash has a corresponding reboot process. So... what was your question? Is it possible you're reading the log top-to-bottom instead of in order of the dates?

xcr · 03-21-2006, 08:24 PM

hi,
my question is, how is that there is another reboot(Mar 21st) without a crash or a shutdown?(at the top) and it does not even show a time when the machine went down! is this a good way to find out the system downtime? or is there any other way option say system level information available that will let me know when the system was down and rebooted? i could continuously ping the machine and note down everytime the machine is unreachable but i want to avoid this technique.
thanks

Simon Bridge · 03-22-2006, 05:00 PM

I think you could be onto something there ... before you see a reboot, it should show a runlevel (to 0) change to shutdown. My (FC4) system gives me a log very like yours - OTOH: I havn't rebooted in a little over a year... so I figured I'd see what the difference is with my laptop:

Recent entries look like:

Code:

simon   pts/2
simon   pts/1
simon   :0
runlevel   (to lvl 2)
reboot   system boot
shutdown   system down
runlevel (to lvl 0)

and so on... So the question is: "Why dosn't xcr's machine log a change to runlevel 0 before a shutdown (Or runlevel 6 prior to a restart.)?"

Suggested test: shutdown via "# shutdown -r" ... immediately after a reboot, get the last -x entries and see if the runlevel change is noted. Try from runlevel 5 as well as runlevel 3 in case the bahavior is different.

xcr · 03-24-2006, 10:50 PM

you mentioned about calling scripts to maintain log. i understand we can call scripts by placing them in /etc/init.d/ when a system boots. but is there a way to call a script to make a time log when we send a system shutdown command?(graceful or abrupt)?? if yes where do i place them?

Simon Bridge · 03-25-2006, 03:44 AM

Ah good - I lost the thread a bit there.

A wee while ago I said:

Quote:

So the question is: "Why dosn't xcr's machine log a change to runlevel 0 before a shutdown (Or runlevel 6 prior to a restart.)?"

I must have been a tad dopey that night. The most likely reason (a priori) that you have restart events, in the "last -x" output, with no corresponding shutdown or crash events, is that your system did not shut down cleanly enough to even log the event.(Imagine what would happen if you just yanked the power cord?)

As far as measuring downtime is concerned:
I've been checking with some IT-business folk I know, guys who run large corporate servers, and they tell me that less -x is a pretty bad way to do things because of this.

Your best bet is to keep track of uptime, then downtime is the reciprocal. Since you are interested, I guess, in system useability as opposed to just "is the system powered up", you want to do aaplication layer health checks...

From the NZLUG mailing list...
[quote=Jim Cheetham]Look at the existing stable solutions to some of these things -
http://munin.sf.net will reveal all sorts of interesting data about his
platform from within itself, and http://www.nagios.org will help you set
up monitoring of basic IP availabiity all the way to internal
app<->database connectivity.

If you don't want to go that far ... uprecords, gives you a nice
display like:

Code:

     #               Uptime | System                               Boot up
----------------------------+-------------------------------------------------
     1   316 days, 13:25:49 | Linux 2.6.6-1-386   Fri Feb  4 01:08:41 2005
->   2    97 days, 01:46:42 | Linux 2.6.8-2-686   Sat Dec 17 14:36:21 2005
----------------------------+-------------------------------------------------
no1 in   219 days, 11:39:08 | at                  Mon Oct 30 04:02:12 2006

... this couresy of Robin Shaet, same list.

This last one looks like what you want.

You needent run the script from init ... just write a script to extract and so on the data from the general log, and run it whever you need to access this information.

You're system is crashing an aweful lot - anything you want to tell us?

xcr · 03-25-2006, 02:51 PM

hi,
thanks for that valuable information about 'uprecords'. it does work pretty much the way i want it to work. i am using a web interface to fire IPMI commands to shutdown/restart my machines everynow and then to experiment like 'failover' etc, maybe that is why my machine crashes so many times!
thanks though! will trouble you again if i need help

later