Linux - Server This forum is for the discussion of Linux Software used in a server related context. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
04-10-2011, 02:58 PM
|
#1
|
Member
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89
Rep:
|
CentOS - khungtaskd - is there a way to know what last hung task was?
If khungtaskd stopped a hung process is there any way to know what that task was? I.e. does khungtaskd leave any kind of log record?
Or any other way to find out if a task hung.
Last edited by cnmoore; 04-10-2011 at 03:07 PM.
|
|
|
04-10-2011, 04:02 PM
|
#2
|
Moderator
Registered: May 2001
Posts: 29,415
|
In /var/log/messages you find lines like "INFO: task [processname]:[PID] blocked for more than 120 seconds." The stack trace will be right beneath it. You might also want to collect server performance statistics using SAR tools, Atop, Dstat or collectl for later perusal.
|
|
|
04-10-2011, 05:09 PM
|
#3
|
Member
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89
Original Poster
Rep:
|
Thanks! Unfortunately messages shows nothing unusual for the time it spiked up today. No INFO or anything else of interest.
I already collect a lot of stuff, so far haven't solved puzzle of why CPU% suddenly shoots way up in top. I have a script that checks top every minute, and if CPU% is way high it makes a dump of top, and a copy of server-status.
The server-status often shows a lot of clients waiting to connect, but the httpd log just shows a lot of GETs.
The top dumps tend to look like this:
Code:
top - 22:42:01 up 6 days, 23:53, 0 users, load average: 13.24, 6.93, 2.65
Tasks: 162 total, 1 running, 160 sleeping, 0 stopped, 1 zombie
Cpu0 : 8.6%us, 1.1%sy, 0.0%ni, 89.3%id, 0.8%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu1 : 1.9%us, 0.5%sy, 0.0%ni, 97.3%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4051448k total, 3648444k used, 403004k free, 94448k buffers
Swap: 2031608k total, 116k used, 2031492k free, 2445680k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27973 apache 15 0 166m 20m 4084 S 12.8 0.5 0:00.40 /usr/sbin/httpd -k
2561 named 25 0 225m 11m 2076 S 1.0 0.3 6:51.92 /usr/sbin/named -u
28005 mike 20 0 12740 1036 736 R 1.0 0.0 0:00.01 /usr/bin/top -n 1 -
1 root 15 0 10352 704 588 S 0.0 0.0 0:00.78 init [3]
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [migration/0]
3 root 34 19 0 0 0 S 0.0 0.0 0:00.16 [ksoftirqd/0]
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/0]
(It's obscure to me how the load average can be 13.24 with Cpu0 89.3%id, and Cpu1 97.3%id).
The server-status and httpd log didn't give any clue to that high httpd use. So I got to thinking Maybe it was hung. I just happened to be reading about khungtaskd.
This is just a puzzle - the usage spikes don't seem to do any harm at all.
|
|
|
04-10-2011, 06:58 PM
|
#4
|
Moderator
Registered: May 2001
Posts: 29,415
|
Quote:
Originally Posted by cnmoore
I already collect a lot of stuff, so far haven't solved puzzle of why CPU% suddenly shoots way up in top. I have a script that checks top every minute, and if CPU% is way high it makes a dump of top, and a copy of server-status. (..) This is just a puzzle - the usage spikes don't seem to do any harm at all.
|
While a script may seem nice it doesn't work as well as having a continuous flow of data available. For instance with Atop you just let it collect information for say 24 hours. Then you can replay its binary log and step through it allowing you to see at around what time performance drops. An indication of time should help you search logs for information slash clues.
|
|
|
04-10-2011, 09:54 PM
|
#6
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,340
|
Quote:
Originally Posted by cnmoore
(It's obscure to me how the load average can be 13.24 with Cpu0 89.3%id, and Cpu1 97.3%id).
|
Loadavg includes tasks in uninterruptible sleep - usually defined as waiting on disk I/O, but tasks like http will put threads in this state. atop looks like it counts them for you (I don't use it)
Quote:
This is just a puzzle - the usage spikes don't seem to do any harm at all.
|
So why do you care ?.
|
|
|
04-10-2011, 11:33 PM
|
#7
|
Member
Registered: Sep 2010
Location: Sunnyvale, CA
Distribution: CentOS 5.5
Posts: 89
Original Poster
Rep:
|
I just like to try to understand puzzles, tackling them one at a time.
Thanks for your explanation about uninterruptible sleep. I didn't know about that - trying to read up on it now - one thing does lead to another. This is a very helpful lead.
My saved top screens do not show any processes in state D. But I realize that the thing may have woken up before the load average could get reported.
|
|
|
04-13-2011, 09:59 PM
|
#8
|
Moderator
Registered: May 2001
Posts: 29,415
|
Quote:
Originally Posted by cnmoore
|
Yes, that would be it. And when it no longer serves any purpose you can just uninstall the package.
|
|
1 members found this post helpful.
|
All times are GMT -5. The time now is 10:54 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|