LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices

Reply
 
Search this Thread
Old 03-10-2013, 03:59 PM   #1
mail2vivek1
LQ Newbie
 
Registered: Sep 2011
Posts: 24
Blog Entries: 1

Rep: Reputation: Disabled
Kernel error


Hello All,

I am running Virtual machine on xen. Because of some reason VM got crashed and i am not able to find enough information in logs.


This was the kernel error message:-

INFO: task mv:16138 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88002804cf00 0 16138 25947 0x00000080
ffff880357af5d18 0000000000000286 0000000000000000 ffffffff81126737
ffff8800a476a300 ffffffff81aae4c0 ffff8800a476a6d0 0000000103347a7e
0000000057af5d28 0000000000000000 0000000000000000 ffff8803d3da26a4
Call Trace:
[<ffffffff81126737>] ? __link_path_walk+0x10d/0x5fb
[<ffffffff81456ad0>] __mutex_lock_common+0x12f/0x1a1
[<ffffffff81123ed7>] ? path_put+0x22/0x27
[<ffffffff81456b91>] __mutex_lock_slowpath+0x19/0x1b
[<ffffffff81456bfa>] mutex_lock+0x23/0x3a
[<ffffffff8112456d>] lock_rename+0x49/0xad
[<ffffffff81126f07>] sys_renameat+0xbe/0x201
[<ffffffff810f216c>] ? handle_mm_fault+0x14b/0x80f
[<ffffffff81459ec0>] ? do_page_fault+0x28a/0x299
[<ffffffff81127065>] sys_rename+0x1b/0x1d
[<ffffffff81011db2>] system_call_fastpath+0x16/0x1b
INFO: task mv:16139 blocked for more than 120 seconds.

Could any one please help me to debug it further?

Thank you
Vivek
 
Old 03-10-2013, 05:58 PM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,771

Rep: Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614
Quote:
Originally Posted by mail2vivek1 View Post
Hello All,
I am running Virtual machine on xen. Because of some reason VM got crashed and i am not able to find enough information in logs.

This was the kernel error message:- Could any one please help me to debug it further?
No, not with what you've posted, no one can. All you've told us it that it "got crashed", and it's a virtual machine on xen. Nothing about the hardware, version/distro of Linux, what runs on the machine, what was going on before it crashed, what logs you checked, or ANYTHING useful. Just looking up the error on Google brings up lots of possibilities, but since you've provided no details, we can't say which may apply to you.
 
Old 03-10-2013, 06:57 PM   #3
mail2vivek1
LQ Newbie
 
Registered: Sep 2011
Posts: 24
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Hi,

This VM is running on IBM blade server.
OS distro is :- Linux Server release 5.8
Kernel:- 2.6.32-300.39.2.el5uek

I checked /var/log/messages, sar log
and on hypervisor:- i checked /var/log/xen

=====================

root@abc:/root# sar -b -f /var/log/sa/sa05 -s 17:01:00
Linux 2.6.32-300.39.2.el5uek (abc.abc.com) 03/05/2013

05:10:01 PM tps rtps wtps bread/s bwrtn/s
05:20:01 PM 22.43 8.03 14.40 176.99 52.11
05:30:02 PM 41.99 19.58 22.41 409.49 94.87
05:40:01 PM 13.78 2.92 10.86 113.91 56.26
05:50:01 PM 9.08 1.86 7.22 76.51 30.28
06:00:01 PM 10.56 1.25 9.31 50.81 47.62
06:10:02 PM 27.00 3.08 23.92 111.46 90.71
06:20:02 PM 36.40 14.88 21.52 403.55 80.54
06:30:01 PM 40.45 28.29 12.15 664.30 56.81
06:40:02 PM 17.88 3.79 14.09 169.26 63.74
06:50:01 PM 8.06 3.17 4.89 111.35 26.71
Average: 22.94 8.74 14.19 229.72 60.30

07:20:17 PM LINUX RESTART

07:31:10 PM LINUX RESTART

07:40:01 PM tps rtps wtps bread/s bwrtn/s
07:50:01 PM 3.36 0.50 2.86 24.18 12.94
08:00:01 PM 5.20 2.16 3.04 80.10 14.61
08:10:01 PM 6.38 2.06 4.32 109.85 20.26
08:20:01 PM 49.15 9.61 39.53 219.40 124.70

==================

root@abc:/root# sar -B -f /var/log/sa/sa05 -s 17:01:00
Linux 2.6.32-300.39.2.el5uek (ab.cabc.com) 03/05/2013

05:10:01 PM pgpgin/s pgpgout/s fault/s majflt/s
05:20:01 PM 44.25 13.02 3931.86 0.05
05:30:02 PM 102.40 23.72 33503.86 0.64
05:40:01 PM 28.47 14.06 11102.83 0.72
05:50:01 PM 19.13 7.57 6025.53 0.71
06:00:01 PM 12.68 11.90 17439.82 0.37
06:10:02 PM 27.89 22.68 21842.83 0.81
06:20:02 PM 100.98 20.14 9606.81 1.30
06:30:01 PM 166.01 14.21 4571.78 1.35
06:40:02 PM 42.38 15.93 5695.02 1.26
06:50:01 PM 27.91 6.67 1255.84 0.52
Average: 57.45 15.07 11504.82 0.77

07:20:17 PM LINUX RESTART

07:31:10 PM LINUX RESTART

07:40:01 PM pgpgin/s pgpgout/s fault/s majflt/s
07:50:01 PM 6.03 3.23 413.07 0.05
08:00:01 PM 20.02 3.66 491.01 0.13
08:10:01 PM 27.46 5.06 1157.37 0.24
08:20:01 PM 54.93 31.18 44016.30 0.33
08:30:01 PM 17.55 12.78 13897.98 0.80
08:40:02 PM 149.59 20.72 6596.87 0.73
08:50:01 PM 19.33 13.25 5152.19 0.79
09:00:02 PM 26.88 12.73 7836.33 0.92


=================

/var/log/message

Mar 5 18:13:54 abc tdodbc[10129]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:15:48 abc tdodbc[18624]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:16:07 abc tdodbc[18624]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:22:13 abc tdodbc[29106]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:27:29 abc tdodbc[2933]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:27:45 abc ntpd[2053]: synchronized to 17.254.0.27, stratum 2
Mar 5 18:28:22 abc ntpd[2053]: synchronized to 17.254.0.28, stratum 2
Mar 5 18:30:42 abc tdodbc[10719]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:31:37 abc ntpd[2053]: synchronized to 17.254.0.27, stratum 2
Mar 5 18:31:43 abc ntpd[2053]: synchronized to 17.254.0.28, stratum 2
Mar 5 18:31:45 abc tdodbc[12859]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:32:50 abc tdodbc[14780]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 18:33:06 abc auditd[1738]: Audit daemon rotating log files
Mar 5 18:36:40 abc tdodbc[19395]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed
Mar 5 19:20:20 abc syslogd 1.4.1: restart.
Mar 5 19:20:20 abc kernel: klogd 1.4.1, log source = /proc/kmsg started.

Please guide me how to proceed from here
 
Old 03-11-2013, 08:59 AM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,771

Rep: Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614
Quote:
Originally Posted by mail2vivek1 View Post
Hi,
This VM is running on IBM blade server. OS distro is :- Linux Server release 5.8 Kernel:- 2.6.32-300.39.2.el5uek

I checked /var/log/messages, sar log and on hypervisor:- i checked /var/log/xen
07:20:17 PM LINUX RESTART
07:31:10 PM LINUX RESTART

/var/log/message
Mar 5 18:13:54 abc tdodbc[10129]: codbctrace.cpp EnableTraceingForDSN 7 : Could not use /bcd/fgh/gsd/Informatica/DSN.Trace.log for tracing, size exceeds maximum allowed size of 1000000 and overwrite not allowed

Please guide me how to proceed from here
And you still haven't said what runs on this machine, what it was doing before the crash, etc. And did you look at what you posted?? It's telling you that the server was RESTARTED...that means, someone rebooted it. And based on the trace log message, you have a database of some sort running...have you check the parameters of that database? Have you tried to contact Red Hat support? They can often assist you with crashes like this, by analyzing dump/trace files that they have you generate. Since you're paying for RHEL, you're paying for support...that would be my first call.
 
  


Reply

Tags
kernel panic, xenvirtualization


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Kernel Error echo 0 > /proc/sys/kernel/hung _task_timeout_secsin Redhat 5.8 LittleMaster Linux - Server 1 10-18-2012 09:32 AM
upgrading Fedora Core 3 kernel 2.6.9-1.667 to 2.6.26.3 manually+kernel panic error LinuxLovinNoob Linux - Newbie 7 12-17-2008 08:45 PM


All times are GMT -5. The time now is 10:05 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration