LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
LinkBack Search this Thread
Old 04-10-2011, 01:58 PM   #1
cnmoore
Member
 
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89

Rep: Reputation: 0
CentOS - khungtaskd - is there a way to know what last hung task was?


If khungtaskd stopped a hung process is there any way to know what that task was? I.e. does khungtaskd leave any kind of log record?

Or any other way to find out if a task hung.

Last edited by cnmoore; 04-10-2011 at 02:07 PM.
 
Old 04-10-2011, 03:02 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,534
Blog Entries: 51

Rep: Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601
In /var/log/messages you find lines like "INFO: task [processname]:[PID] blocked for more than 120 seconds." The stack trace will be right beneath it. You might also want to collect server performance statistics using SAR tools, Atop, Dstat or collectl for later perusal.
 
Old 04-10-2011, 04:09 PM   #3
cnmoore
Member
 
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89

Original Poster
Rep: Reputation: 0
Thanks! Unfortunately messages shows nothing unusual for the time it spiked up today. No INFO or anything else of interest.

I already collect a lot of stuff, so far haven't solved puzzle of why CPU% suddenly shoots way up in top. I have a script that checks top every minute, and if CPU% is way high it makes a dump of top, and a copy of server-status.

The server-status often shows a lot of clients waiting to connect, but the httpd log just shows a lot of GETs.

The top dumps tend to look like this:
Code:
top - 22:42:01 up 6 days, 23:53,  0 users,  load average: 13.24, 6.93, 2.65
Tasks: 162 total,   1 running, 160 sleeping,   0 stopped,   1 zombie
Cpu0  :  8.6%us,  1.1%sy,  0.0%ni, 89.3%id,  0.8%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu1  :  1.9%us,  0.5%sy,  0.0%ni, 97.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4051448k total,  3648444k used,   403004k free,    94448k buffers
Swap:  2031608k total,      116k used,  2031492k free,  2445680k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27973 apache    15   0  166m  20m 4084 S 12.8  0.5   0:00.40 /usr/sbin/httpd -k
 2561 named     25   0  225m  11m 2076 S  1.0  0.3   6:51.92 /usr/sbin/named -u
28005 mike      20   0 12740 1036  736 R  1.0  0.0   0:00.01 /usr/bin/top -n 1 -
    1 root      15   0 10352  704  588 S  0.0  0.0   0:00.78 init [3]
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [migration/0]
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.16 [ksoftirqd/0]
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/0]
(It's obscure to me how the load average can be 13.24 with Cpu0 89.3%id, and Cpu1 97.3%id).
The server-status and httpd log didn't give any clue to that high httpd use. So I got to thinking Maybe it was hung. I just happened to be reading about khungtaskd.

This is just a puzzle - the usage spikes don't seem to do any harm at all.
 
Old 04-10-2011, 05:58 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,534
Blog Entries: 51

Rep: Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601
Quote:
Originally Posted by cnmoore View Post
I already collect a lot of stuff, so far haven't solved puzzle of why CPU% suddenly shoots way up in top. I have a script that checks top every minute, and if CPU% is way high it makes a dump of top, and a copy of server-status. (..) This is just a puzzle - the usage spikes don't seem to do any harm at all.
While a script may seem nice it doesn't work as well as having a continuous flow of data available. For instance with Atop you just let it collect information for say 24 hours. Then you can replay its binary log and step through it allowing you to see at around what time performance drops. An indication of time should help you search logs for information slash clues.
 
Old 04-10-2011, 06:28 PM   #5
cnmoore
Member
 
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89

Original Poster
Rep: Reputation: 0
Our server doesn't seem to have Atop installed. It sounds great. I found the home page http://www.atoptool.nl/index.php.
I mostly let our host do installs but I can give it a try. Do I just say wget http://www.atoptool.nl/download/atop-1.26-2.x86_64.rpm and then rpm -i the package?
 
Old 04-10-2011, 08:54 PM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 11,801

Rep: Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923
Quote:
Originally Posted by cnmoore View Post
(It's obscure to me how the load average can be 13.24 with Cpu0 89.3%id, and Cpu1 97.3%id).
Loadavg includes tasks in uninterruptible sleep - usually defined as waiting on disk I/O, but tasks like http will put threads in this state. atop looks like it counts them for you (I don't use it)
Quote:
This is just a puzzle - the usage spikes don't seem to do any harm at all.
So why do you care ?.
 
Old 04-10-2011, 10:33 PM   #7
cnmoore
Member
 
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89

Original Poster
Rep: Reputation: 0
I just like to try to understand puzzles, tackling them one at a time.

Thanks for your explanation about uninterruptible sleep. I didn't know about that - trying to read up on it now - one thing does lead to another. This is a very helpful lead.

My saved top screens do not show any processes in state D. But I realize that the thing may have woken up before the load average could get reported.
 
Old 04-13-2011, 08:59 PM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,534
Blog Entries: 51

Rep: Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601Reputation: 2601
Quote:
Originally Posted by cnmoore View Post
Do I just say wget http://www.atoptool.nl/download/atop-1.26-2.x86_64.rpm and then rpm -i the package?
Yes, that would be it. And when it no longer serves any purpose you can just uninstall the package.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
kernel panic attempted to kill the idle task! in idle task - not syncing dudutworld Linux - Newbie 2 09-16-2013 01:50 PM
launching a task from cli and "divorcing it from terminal task wufo Linux - General 6 02-11-2011 11:32 AM
Task bar no longer shows task Richard Rahl Linux - Newbie 3 04-12-2010 09:39 AM
[SOLVED] Kill a hung task when kill -9 doesn't help 10110111 Linux - General 4 04-02-2009 11:10 AM
Suse 9.3 Hung up sparton Suse/Novell 2 04-11-2006 03:41 PM


All times are GMT -5. The time now is 10:03 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration