LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-30-2011, 06:02 AM   #1
bweaver
Member
 
Registered: Nov 2010
Posts: 44

Rep: Reputation: 0
What problem does this /var/log/messages/ indicate?


I am running Debian Squeeze 6.0.2.1 + Xen 4.0

I have had a difficult bug, and finally got a couple of logs on it.

My question: what problem does this /var/log/messages indicate?

Is the problem I need to debug indicated by the first line of the log or the first line--about 7 lines from the top--of the "Call Trace"?

In other words,

1. A problem with
"updatedb.mlocat:3605 blocked for more than 120 seconds" ?

( if so, this could be related to Ubuntu Bug #276476 )

or

2. "pvclock_clocksource_read+0x3a/0x8b" ?

( related to Debian Bug report logs - #603727 )


[ The first line quoted here is the first line of the error message in /var/log/messages. ]


Aug 29 06:28:53 vm2 kernel: [53400.204062] INFO: task updatedb.mlocat:3605 blocked for more than 120 seconds.
Aug 29 06:28:53 vm2 kernel: [53400.204099] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 29 06:28:53 vm2 kernel: [53400.204119] updatedb.mloc D 0000000000000000 0 3605 3601 0x00000000
Aug 29 06:28:53 vm2 kernel: [53400.204125] ffff8802ed071530 0000000000000286 0000000000000000 ffff88009babd260
Aug 29 06:28:53 vm2 kernel: [53400.204132] ffff8802e95f0000 0000000000000088 000000000000f9e0 ffff88009e399fd8
Aug 29 06:28:53 vm2 kernel: [53400.204138] 0000000000015780 0000000000015780 ffff8802e95e1530 ffff8802e95e1828
Aug 29 06:28:53 vm2 kernel: [53400.204144] Call Trace:
Aug 29 06:28:53 vm2 kernel: [53400.204154] [<ffffffff8102ddcc>] ? pvclock_clocksource_read+0x3a/0x8b
Aug 29 06:28:53 vm2 kernel: [53400.204160] [<ffffffff8110f19a>] ? sync_buffer+0x0/0x40
Aug 29 06:28:53 vm2 kernel: [53400.204166] [<ffffffff8130c16a>] ? io_schedule+0x73/0xb7
Aug 29 06:28:53 vm2 kernel: [53400.204169] [<ffffffff8110f1d5>] ? sync_buffer+0x3b/0x40
Aug 29 06:28:53 vm2 kernel: [53400.204174] [<ffffffff8130d42a>] ? _spin_unlock_irqrestore+0xd/0xe
Aug 29 06:28:53 vm2 kernel: [53400.204178] [<ffffffff8130c677>] ? __wait_on_bit+0x41/0x70
Aug 29 06:28:53 vm2 kernel: [53400.204181] [<ffffffff8110f19a>] ? sync_buffer+0x0/0x40
Aug 29 06:28:53 vm2 kernel: [53400.204185] [<ffffffff8130c711>] ? out_of_line_wait_on_bit+0x6b/0x77
Aug 29 06:28:53 vm2 kernel: [53400.204190] [<ffffffff81065f34>] ? wake_bit_function+0x0/0x23
Aug 29 06:28:53 vm2 kernel: [53400.204202] [<ffffffffa04bf824>] ? ocfs2_read_blocks+0x55d/0x6c2 [ocfs2]
Aug 29 06:28:53 vm2 kernel: [53400.204208] [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
Aug 29 06:28:53 vm2 kernel: [53400.204217] [<ffffffffa04db9a1>] ? ocfs2_validate_inode_block+0x0/0x1ab [ocfs2]
Aug 29 06:28:53 vm2 kernel: [53400.204226] [<ffffffffa04db58c>] ? ocfs2_read_inode_block_full+0x37/0x51 [ocfs2]
 
Old 08-31-2011, 05:16 AM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,790

Rep: Reputation: 650Reputation: 650Reputation: 650Reputation: 650Reputation: 650Reputation: 650
I'm guessing the ocfs2 volume is shared storage and may have gone offline at some point or been extremely busy, you probably want to exclude ocfs2 filesystems from scanning by updatedb in any case, edit /etc/updatedb.conf and add ocfs2 to the PRUNEFS line, then kill off any stuck updatedb processes - 'pkill -9 updatedb'.
 
Old 08-31-2011, 01:21 PM   #3
bweaver
Member
 
Registered: Nov 2010
Posts: 44

Original Poster
Rep: Reputation: 0
what exactly to add to prunefs

Thank you. I will definitely try this out.

But what exactly should I enter on the prunefs line? and where in the line should I put it? I entered "ocfs2" so that my prunefs line now reads:

PRUNEFS="NFS nfs nfs4 rpc_pipefs afs ocfs2 binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf fuse.glusterfs fuse.sshfs"


but I noticed /etc/mtab has:

ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0

and did not know but what I should enter these or other ocfs2 related information into prunefs as well.
 
Old 08-31-2011, 01:23 PM   #4
bweaver
Member
 
Registered: Nov 2010
Posts: 44

Original Poster
Rep: Reputation: 0
cont.

...and of course mtab has the SAN entry. In all,

ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0
/dev/sdb1 /ocfs2SAN ocfs2 rw,_netdev,heartbeat=local 0 0
 
Old 08-31-2011, 08:01 PM   #5
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,790

Rep: Reputation: 650Reputation: 650Reputation: 650Reputation: 650Reputation: 650Reputation: 650
Your PRUNEFS line looks fine, this will prevent updatedb from indexing any ocfs2 volumes
 
Old 09-01-2011, 09:41 AM   #6
bweaver
Member
 
Registered: Nov 2010
Posts: 44

Original Poster
Rep: Reputation: 0
problem likely solved

Ok, thanks for this. I added 'ocfs2' as indicated. I also added the path to our SAN, '/ocfs2SAN', to Prunepaths for good measure. While I still have problems with other processes hanging at 6:25 in the morning, updatedb is no longer one of them. Your comments were very helpful as they put me onto possibility that the suite of daily cron jobs (cron.daily), scheduled for 6:25 in the morning are either hanging up or causing ocfs2 and the SAN connection to hang up. Perhaps they are hanging for the reason you mentioned: that the SAN is overloaded at that time. (which could conceivably owe either to daily backups being run then or else perhaps a bunch of new Debian Squeeze installs on several of our boxes all running their cron.daily jobs at the default time of 6:25 a.m. I am running the cron.daily jobs on the test box every 30 seconds to test possible conflict with the ocfs2-network stress test and now, during the workday, I am not encountering any problems.

Thanks again.

Ben Weaver
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
/var/adm/messages vs /var/log/messages gomes1333 Linux - General 1 04-06-2010 04:08 AM
/var/log/messages and /var/log/cron not working sigkill Linux - Software 6 08-09-2008 01:08 PM
Hard drive udma-33 reset problem in var/log/messages v00d00101 Linux - Kernel 3 02-14-2008 09:45 PM
Are these entries a problem in my /var/log/messages jim.thornton Linux - Server 4 02-08-2008 08:16 AM
/var/log/messages problem x42bn6 Linux - Security 4 03-24-2007 07:04 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:36 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration