LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 07-01-2013, 05:18 AM   #1
mihirict
LQ Newbie
 
Registered: Jul 2013
Posts: 12

Rep: Reputation: Disabled
Question Linux Cluster


Does anyone have any experience regarding below call trace.Please share what is going on my cluster server.server is running Redhat-5.6 64 bit of os.Looking forward for your suggestion

Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff800645d3>] __down_write_nested+0x7a/0x92
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff88664167>] :gfs2:gfs2_log_flush+0x1f/0x472
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff88663b29>] :gfs2:gfs2_ail1_empty+0x36/0x95
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff88664a94>] :gfs2:gfs2_logd+0xa2/0x15c
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff886649f2>] :gfs2:gfs2_logd+0x0/0x15c
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff80032974>] kthread+0xfe/0x132
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff80032876>] kthread+0x0/0x132
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jun 30 05:35:57 SPLEPTPRDN2 kernel:
Jun 30 05:35:57 SPLEPTPRDN2 kernel: INFO: task gfs2_quotad:6094 blocked for more than 120 seconds.
Jun 30 05:35:57 SPLEPTPRDN2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 30 05:35:57 SPLEPTPRDN2 kernel: gfs2_quotad D ffffffff80153806 0 6094 371 6576 6093 (L-TLB)
Jun 30 05:35:57 SPLEPTPRDN2 kernel: ffff81046adfdcd0 0000000000000046 0000000000000018 ffffffff886334f8
Jun 30 05:35:57 SPLEPTPRDN2 kernel: 0000000000000286 000000000000000a ffff81087fa77080 ffff81087fc0b040
Jun 30 05:35:57 SPLEPTPRDN2 kernel: 0001b324cd91cf8b 00000000000040ef ffff81087fa77268 0000000c88634e5f
Jun 30 05:35:57 SPLEPTPRDN2 kernel: Call Trace:
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff886334f8>] :dlm:request_lock+0x93/0xa0
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8865ef54>] :gfs2:just_schedule+0x0/0xe
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8865ef5d>] :gfs2:just_schedule+0x9/0xe
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8865ef54>] :gfs2:just_schedule+0x0/0xe
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff800a28e2>] wake_bit_function+0x0/0x23
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8865ef4f>] :gfs2:gfs2_glock_wait+0x2b/0x30
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8867569a>] :gfs2:gfs2_statfs_sync+0x3f/0x165
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff88675692>] :gfs2:gfs2_statfs_sync+0x37/0x165
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8005b935>] del_timer_sync+0xc/0x16
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8866f4af>] :gfs2:quotad_check_timeo+0x20/0x60
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff88670fb1>] :gfs2:gfs2_quotad+0xde/0x214
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff800a28b4>] autoremove_wake_function+0x0/0x2e
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff88670ed3>] :gfs2:gfs2_quotad+0x0/0x214
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff80032974>] kthread+0xfe/0x132
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff80032876>] kthread+0x0/0x132
Jun 30 05:35:57 SPLEPTPRDN2 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jun 30 05:35:57 SPLEPTPRDN2 kernel:
Jun 30 05:35:57 SPLEPTPRDN2 kernel: INFO: task sapstartsrv:6478 blocked for more than 120 seconds.
Jun 30 05:35:57 SPLEPTPRDN2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 30 05:35:57 SPLEPTPRDN2 kernel: sapstartsrv D ffffffff80153806 0 6478 1 6479 6477 (NOTLB)
Jun 30 05:35:57 SPLEPTPRDN2 kernel: ffff810469d0db08 0000000000000086 ffff810469d0dae8 0000000000000016
Jun 30 05:35:57 SPLEPTPRDN2 kernel: ffff81010fa38a50 0000000000000007 ffff8104705f0040 ffff81048e142080
Jun 30 05:35:57 SPLEPTPRDN2 kernel: 0001b3270705c94d 00000000000383df ffff8104705
 
Old 07-02-2013, 09:04 AM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 19,179

Rep: Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403Reputation: 4403
Quote:
Originally Posted by mihirict View Post
Does anyone have any experience regarding below call trace.Please share what is going on my cluster server.server is running Redhat-5.6 64 bit of os.Looking forward for your suggestion

Jun 30 05:35:56 SPLEPTPRDN2 kernel: [<ffffffff800a269c>] keventd_create_kthread+0x0/0xc4
Yes, Red Hat has experience debugging their cluster software. Call Red Hat support, since you're using RHEL and (presumably), their clustering packages, and they can walk you through how to run the traces, and help you figure out what's going on.

But based on the VERY little information you've provided here, there's little we can do. If you're PAYING for RHEL, you are also paying for support..that's the best/quickest way to get your problem resolved. If you're NOT paying for RHEL, then you could be running in to a bug that's already been patched/fixed...but you don't have that, since you're not paying. That's why you should call Red Hat, since they can tell you these things.
 
Old 07-02-2013, 09:12 AM   #3
gdr
Member
 
Registered: Jun 2013
Distribution: Mint KDE
Posts: 44

Rep: Reputation: Disabled
That being said, use iostat to check if it isn't an I/O problem with disks not keeping up with requests. It looks like one, but then again asking Redhat is a better idea than posting it here.
 
Old 07-09-2013, 01:39 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,359
Blog Entries: 55

Rep: Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546Reputation: 3546
Quote:
Originally Posted by mihirict View Post
Does anyone have any experience regarding below call trace. Please share what is going on my cluster server. server is running Redhat-5.6 64 bit of os.
First of all this message is of the informational level (hence the "INFO") and it literally says what happens: some tasks blocked for n seconds. It also says how to get disable the message ("echo 0 > /proc/sys/kernel/hung_task_timeout_secs") should you not want to see it. Secondly use any form of SAR (Atop, Dstat, Collectl) to collect generic server statistics and get a better overview of how the server performs and its bottlenecks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
cman service stucks during booting of cluster node (RedHat Cluster Suite) zama Linux - Software 0 07-09-2012 08:50 AM
[SOLVED] Converting Windows node cluster to Linux Cent0S 5.5 cluster do I kickstart or image ? cdrolet Linux - Newbie 1 11-18-2011 10:44 AM
LXer: Linux Terminal Server Project Cluster Edition (LTSP-Cluster) LXer Syndicated Linux News 0 02-07-2010 03:50 AM
LXer: Penguin Computing's Scyld ClusterWare(TM) HPC Offers Enhanced High Availability and Most Advanced Linux Cluster Virtualization, Enabling Broader Cluster Use LXer Syndicated Linux News 0 04-06-2006 06:03 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 07:12 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration