/var/log file system and monitoring health of system

drManhattan · 04-28-2011, 04:39 AM

Hi

I'm trying to understand how to monitor health of redhat system.
I know the follwoing :
-----------------------
/var/log/messages ---> error messages related system and its services
/var/log/boot ---> init error messages
/var/log/dmesg ---> boot error messages

Questions :
------------
1. So if I want to seee whether some disk have I/O errors I read /var/log/dmesg ?
2. It depends whether a service is configured to use syslog or not by using 'logger'cmd or syslog API ?
3. What is 'dmesg' cmd for ? Is'nt the same as /var/log/dmesg ?

thx forl help

sibe · 04-29-2011, 11:30 AM

Hi,

2. Not sure about it, perhaps someone else has some explanation on it.
1&3. It's the same log. Disk I/O and SMART related messages usually go to this log.

drManhattan · 04-29-2011, 01:50 PM

Quote:

1&3. It's the same log. Disk I/O and SMART related messages usually go to this log.

I agree that /var/log/messages and dmesg command contains ALMOST the same content.
Please see code below.
Output from dmesg cmd last 16 lines :

Code:

[  112.932370] wlan0: direct probe to AP 00:23:33:1f:28:a3 (try 1)
[  112.934989] wlan0: direct probe responded
[  112.934998] wlan0: authenticate with AP 00:23:33:1f:28:a3 (try 1)
[  112.937418] wlan0: authenticated
[  112.937459] wlan0: associate with AP 00:23:33:1f:28:a3 (try 1)
[  112.940553] wlan0: RX AssocResp from 00:23:33:1f:28:a3 (capab=0x421 status=0 aid=31)
[  112.940560] wlan0: associated
[  112.943204] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[  116.190102] RPC: Registered udp transport module.
[  116.190105] RPC: Registered tcp transport module.
[  116.190107] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  123.188013] wlan0: no IPv6 routers present
[  218.364100] rpcbind: server 192.168.56.127 not responding, timed out
[  938.976171] lo: Disabled Privacy Extensions
[ 4522.356206] lo: Disabled Privacy Extensions
[ 4557.259959] lo: Disabled Privacy Extensions

Output from /var/log/messages last 16 lines :

Code:

Apr 29 19:27:35 ibmr61e kernel: [   29.131575] Bluetooth: SCO socket layer initialized
Apr 29 19:27:36 ibmr61e kernel: [   29.247657] Bluetooth: RFCOMM TTY layer initialized
Apr 29 19:27:36 ibmr61e kernel: [   29.247662] Bluetooth: RFCOMM socket layer initialized
Apr 29 19:27:36 ibmr61e kernel: [   29.247664] Bluetooth: RFCOMM ver 1.11
Apr 29 19:27:37 ibmr61e kernel: [   30.752049] trackpoint.c: failed to get extended button data
Apr 29 19:27:40 ibmr61e kernel: [   33.388074] IBM TrackPoint firmware: 0x0e, buttons: 0/0
Apr 29 19:27:40 ibmr61e kernel: [   33.788272] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/input/input9
Apr 29 19:27:40 ibmr61e kernel: [   33.816674] psmouse.c: Failed to enable mouse on isa0060/serio1
Apr 29 19:28:59 ibmr61e kernel: [  112.943204] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Apr 29 19:29:03 ibmr61e kernel: [  116.190102] RPC: Registered udp transport module.
Apr 29 19:29:03 ibmr61e kernel: [  116.190105] RPC: Registered tcp transport module.
Apr 29 19:29:03 ibmr61e kernel: [  116.190107] RPC: Registered tcp NFSv4.1 backchannel transport module.
Apr 29 19:30:45 ibmr61e kernel: [  218.364100] rpcbind: server 192.168.56.127 not responding, timed out
Apr 29 19:42:45 ibmr61e kernel: [  938.976171] lo: Disabled Privacy Extensions
Apr 29 20:42:29 ibmr61e kernel: [ 4522.356206] lo: Disabled Privacy Extensions
Apr 29 20:43:04 ibmr61e kernel: [ 4557.259959] lo: Disabled Privacy Extensions

Why output from dmesg cmd differs a little bit from /var/log/messages output (differences in bold) ?
It look like both have some common messages but each one have some additional messages too.
thx for help.

sibe · 04-29-2011, 03:36 PM

--

Sorry, I think I have misread the question; dmesg and /var/log/messages (I thought it was /var/log/dmesg) are different.

/var/log/messages is written to by syslogd daemon with verbosity level defined in /etc/syslog.conf. On the other hand, dmesg is a kernel ring buffer message and not dependent on syslogd. dmesg invocation defined in /etc/rc.sysinit (usually goes to /var/log/dmesg) and its verbosity level set in /etc/sysconfig/init (in Red Hat).

I think the loglevel/verbosity level setting that made their output different.

drManhattan · 04-29-2011, 03:50 PM

Sorry it was my mistake.
I wanted to know what is the difference between output from 'dmesg'cmd and /var/log/dmesg file.
I see that there is not the same info.

output from dmesg cmd :

Code:

[ 5621.991005] [UFW BLOCK] IN=wlan0 OUT= MAC=00:1c:bf:6f:a9:57:00:23:33:1f:28:a0:08:00 SRC=193.17.41.111 DST=10.0.250.55 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=0 DF PROTO=TCP SPT=80 DPT=41444 WINDOW=0 RES=0x00 RST URGP=0 
[ 6778.252288] warning: `VirtualBox' uses 32-bit capabilities (legacy support in use)
[ 6791.541527] device vboxnet0 entered promiscuous mode
[ 6801.652387] vboxnet0: no IPv6 routers present
[ 7034.673054] CE: hpet increasing min_delta_ns to 22500 nsec
[ 7819.777406] CE: hpet increasing min_delta_ns to 33750 nsec
[ 7882.330666] svc: failed to register lockdv1 RPC service (errno 97).
[ 8107.618152] __ratelimit: 27 callbacks suppressed
[ 8107.618157] vlc[4093]: segfault at 115b78f ip 00218914 sp b73fe9e0 error 4 in libc-2.11.1.so[1ea000+153000]
[11886.356021] hrtimer: interrupt took 23327 ns

output from /var/log/dmesg :

Code:

[   26.418966] Registered led device: iwl-phy0::radio
[   26.418990] Registered led device: iwl-phy0::assoc
[   26.419012] Registered led device: iwl-phy0::RX
[   26.419032] Registered led device: iwl-phy0::TX
[   26.429927] ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   26.431518]   alloc irq_desc for 29 on node -1
[   26.431521]   alloc kstat_irqs on node -1
[   26.431555] tg3 0000:04:00.0: irq 29 for MSI/MSI-X
[   26.464349] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   26.721529] cfg80211: Found new beacon on frequency: 2467 MHz (Ch 12) on phy0

Why this output differs ?

sibe · 04-29-2011, 05:47 PM

--
/var/log/dmesg gets written when the system boots, near the end of sysinit process to enable us to see what kernel sees at booting phase. From /etc/rc.sysinit :

Code:

# Now that we have all of our basic modules loaded and the kernel going,
# let's dump the syslog ring somewhere so we can find it later
dmesg -s 131072 > /var/log/dmesg

When we execute dmesg from the prompt, it simply prints out the buffer again, adding the output to whatever has been there, while /var/log/dmesg does not get updated as it only gets written by the sysinit script.

drManhattan · 04-29-2011, 07:05 PM

I understand.
What kind of useful info admin can find with dmesg cmd ?

Because I/O errors and SMART can be found in /var/log/messages.

sibe · 04-30-2011, 05:15 PM

--
Occasionally it might be useful to have a booting-phase information from the kernel ring saved somewhere, in this case, in /var/log/dmesg.

Beyond that, since syslogd catches kernel messages from level .info all the way up to .panic, I don't think there's so much significance in differentiate between the two. After all, the default kernel loglevel as defined in /etc/sysconfig/init will be reset by syslogd configuration in /etc/syslog.conf.

From /etc/sysconfig/init :

Code:

# default kernel loglevel on boot (syslog will reset this)
LOGLEVEL=3

From /etc/syslog.conf :

Code:

*.info;mail.none;authpriv.none;cron.none                /var/log/messages