LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices


Reply
  Search this Thread
Old 02-28-2018, 05:10 AM   #1
barani_pdy
LQ Newbie
 
Registered: Jan 2018
Posts: 7

Rep: Reputation: Disabled
medium access timeout failure. offlining disk!


Hi,

Request your expertise views for the following errors.

When I deployed a VM on ESXi host observed the following
Please let me know on this.


Error snippets:
================
[ 283.626000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff881819539780
[ 283.626000] sd 2:0:1:0: [sdb] task abort on host 2, ffff881808d781c0
[ 283.626000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff881808d781c0
[ 283.626000] sd 2:0:1:0: [sdb] SCSI device reset on scsi2:1
[ 283.627000] sd 2:0:1:0: [sdb] Medium access timeout failure. Offlining disk!
[ 283.627000] sd 2:0:1:0: [sdb] SCSI Bus reset
[ 293.629000] sd 2:0:1:0: [sdb] Unhandled error code
[ 293.629000] sd 2:0:1:0: [sdb]
[ 293.629000] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 293.629000] sd 2:0:1:0: [sdb] CDB:
[ 293.629000] Write(16): 8a 00 00 00 00 00 0d 3d c0 01 00 00 00 80 00 00
[ 293.629000] end_request: I/O error, dev sdb, sector 222150657
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585344
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585345
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585346
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585347
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585348
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585349
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585350
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585351
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585352
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] Buffer I/O error on device sdb5, logical block 150585353
[ 293.629000] lost page write due to I/O error on sdb5
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: [sdb] killing request
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device
[ 293.629000] sd 2:0:1:0: rejecting I/O to offline device

Regards,
 
Old 03-01-2018, 02:08 AM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 8,924

Rep: Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916
Looks like whatever is writing to sdb is causing significant errors. I would recommend the entire disk in the host OS, and check behaviour of the host OS. Disk errors in a VM are unusual because virtual box passes hardware access back to the host system.
 
Old 03-06-2018, 07:17 AM   #3
barani_pdy
LQ Newbie
 
Registered: Jan 2018
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi business_kid,

Thanks for your reply.


In the host I am observing the below


log snippets:
==============
2018-03-06T11:46:22.012Z cpu26:334252)PVSCSI: 769: PVSCSICheckForFirstWrite: CID for target: 1 is not updated yet. Will return BSY.

2018-03-06T11:46:49.125Z cpu62:32925)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43aa80e5d000, 0) to dev "naa.600605b0052b76a0189a635b295ad161" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:46:49.142Z cpu62:32925)ScsiDeviceIO: 2646: Cmd(0x43aa80e5d000) 0x1a, CmdSN 0x147cf from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:46:49.240Z cpu75:32938)ScsiDeviceIO: 2646: Cmd(0x43aa89444880) 0x1a, CmdSN 0x147d4 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:46:49.718Z cpu62:32925)ScsiDeviceIO: 2646: Cmd(0x43aa8945ea00) 0x1a, CmdSN 0x147d9 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:47:22.767Z cpu76:33669)NMP: nmp_ResetDeviceLogThrottling:3345: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 3 times
2018-03-06T11:48:09.486Z cpu24:334247)PVSCSI: 2628: scsi0:1: ABORT ctx=0xf8
2018-03-06T11:48:09.486Z cpu24:334247)megasas: ABORT sn 13997731 cmd=0x8a retries=0 tmo=0
2018-03-06T11:48:09.486Z cpu24:334247)<5>0 :: megasas: RESET -13997731 cmd=8a retries=0
2018-03-06T11:48:09.486Z cpu24:334247)megaraid_sas: HBA reset handler invoked without an internal reset condition.
2018-03-06T11:48:09.756Z cpu4:33352)WARNING: LinScsi: SCSILinuxQueueCommand:1207: queuecommand failed with status = 0x1055 Host Busy vmhba2:2:0:0 (driver name: LSI Logic SAS based MegaRAID driver) - Message repeated 953 times
2018-03-06T11:48:09.756Z cpu10:33480)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x88 (0x439e8b299e40, 318302) to dev "naa.600605b0052b76a0189a635b295ad161" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:NONE
2018-03-06T11:48:09.756Z cpu10:33480)ScsiDeviceIO: 2608: Cmd(0x439e8b299e40) 0x88, CmdSN 0x80 from world 318302 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:48:09.900Z cpu4:33272)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 10 times
2018-03-06T11:48:10.144Z cpu4:33272)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 20 times
2018-03-06T11:48:10.434Z cpu4:33272)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 40 times
2018-03-06T11:48:10.926Z cpu4:33352)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 80 times
2018-03-06T11:48:11.699Z cpu4:33352)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 160 times
2018-03-06T11:48:12.628Z cpu14:33568)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 320 times
2018-03-06T11:48:12.632Z cpu38:33207)ScsiDeviceIO: 2608: Cmd(0x43a280e7d4c0) 0x2a, CmdSN 0x3cc62 from world 32854 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:48:12.705Z cpu38:33790)ScsiDeviceIO: 2608: Cmd(0x43a28a9093c0) 0x2a, CmdSN 0x3cc63 from world 32854 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:48:14.480Z cpu14:33362)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 640 times
2018-03-06T11:48:15.558Z cpu24:334247)<7>megaraid_sas: megasas_wait_for_outstanding: line 2156: AFTER HBA reset handler invoked without an internal reset condition: took 6 seconds. Max is 180.
2018-03-06T11:48:15.558Z cpu24:334247)megaraid_sas: 180 seconds wait had cleared all pending commands.
2018-03-06T11:48:15.558Z cpu24:334247)<5>megasas: reset successful

2018-03-06T11:48:15.558Z cpu24:334247)megasas: ABORT sn 13997732 cmd=0x8a retries=0 tmo=0
2018-03-06T11:48:15.558Z cpu24:334247)<5>0 :: megasas: RESET -13997732 cmd=8a retries=0
2018-03-06T11:48:15.558Z cpu24:334247)megaraid_sas: HBA reset handler invoked without an internal reset condition.
2018-03-06T11:48:15.558Z cpu24:334247)<7>megaraid_sas: megasas_wait_for_outstanding: line 2156: AFTER HBA reset handler invoked without an internal reset condition: took 0 seconds. Max is 180.
2018-03-06T11:48:15.558Z cpu24:334247)megaraid_sas: 180 seconds wait had cleared all pending commands.
2018-03-06T11:48:15.558Z cpu24:334247)<5>megasas: reset successful

2018-03-06T11:48:52.774Z cpu76:33669)NMP: nmp_ResetDeviceLogThrottling:3345: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 376 times
2018-03-06T11:49:57.638Z cpu8:32871)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x439e8e1d9d00, 0) to dev "naa.600605b0052b76a0189a635b295ad161" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:49:57.638Z cpu8:32871)ScsiDeviceIO: 2646: Cmd(0x439e8e1d9d00) 0x1a, CmdSN 0x14804 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:49:57.747Z cpu32:334353)ScsiDeviceIO: 2646: Cmd(0x43a289103380) 0x1a, CmdSN 0x14809 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:49:58.217Z cpu32:33207)ScsiDeviceIO: 2646: Cmd(0x43a28d7d4a80) 0x1a, CmdSN 0x1480e from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:50:07.628Z cpu46:334257)PVSCSI: 2628: scsi0:1: ABORT ctx=0x7d
2018-03-06T11:50:07.628Z cpu46:334257)megasas: ABORT sn 14020983 cmd=0x8a retries=0 tmo=0
2018-03-06T11:50:07.628Z cpu46:334257)<5>0 :: megasas: RESET -14020983 cmd=8a retries=0
2018-03-06T11:50:07.628Z cpu46:334257)megaraid_sas: HBA reset handler invoked without an internal reset condition.
2018-03-06T11:50:09.350Z cpu24:33372)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x2a (0x43a28b636580, 32854) to dev "naa.600605b0052b76a0189a635b295ad161" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:NONE
2018-03-06T11:50:09.350Z cpu24:33372)ScsiDeviceIO: 2608: Cmd(0x43a28b636580) 0x2a, CmdSN 0x3d21d from world 32854 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:50:09.422Z cpu24:33372)ScsiDeviceIO: 2608: Cmd(0x43a288e6c700) 0x2a, CmdSN 0x3d21e from world 32854 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:50:12.422Z cpu24:33958)NMP: nmp_ThrottleLogForDevice:3130: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 10 times
2018-03-06T11:50:12.422Z cpu24:33958)ScsiDeviceIO: 2608: Cmd(0x43a280e5d5c0) 0x2a, CmdSN 0x3d227 from world 32854 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:50:12.495Z cpu24:33958)ScsiDeviceIO: 2608: Cmd(0x43a289764180) 0x2a, CmdSN 0x3d228 from world 32854 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2018-03-06T11:50:13.702Z cpu46:334257)<7>megaraid_sas: megasas_wait_for_outstanding: line 2156: AFTER HBA reset handler invoked without an internal reset condition: took 6 seconds. Max is 180.
2018-03-06T11:50:13.702Z cpu46:334257)megaraid_sas: 180 seconds wait had cleared all pending commands.
2018-03-06T11:50:13.702Z cpu46:334257)<5>megasas: reset successful

2018-03-06T11:50:52.780Z cpu69:33669)NMP: nmp_ResetDeviceLogThrottling:3345: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 7 times
2018-03-06T11:50:52.884Z cpu28:32891)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a28a85dc00, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:50:52.986Z cpu24:32887)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a28a85dc00, 0) to dev "mpx.vmhba32:C0:T0:L1" on path "vmhba32:C0:T0:L1" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:50:52.987Z cpu24:32887)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a28a85dc00, 0) to dev "mpx.vmhba32:C0:T0:L2" on path "vmhba32:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:50:52.991Z cpu24:32887)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a28a85dc00, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2018-03-06T11:52:25.675Z cpu30:334278)Vmxnet3: 15108: Using default queue delivery for vmxnet3 for port 0x200005c
2018-03-06T11:52:25.676Z cpu30:334278)NetPort: 1573: enabled port 0x200005c with mac 00:50:56:b6:10:51
2018-03-06T11:52:25.688Z cpu30:334278)NetPort: 1780: disabled port 0x200005c
2018-03-06T11:52:25.701Z cpu30:334278)Vmxnet3: 15108: Using default queue delivery for vmxnet3 for port 0x200005c
2018-03-06T11:52:25.702Z cpu30:334278)NetPort: 1573: enabled port 0x200005c with mac 00:50:56:b6:10:51
2018-03-06T11:52:47.769Z cpu52:334261)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a68d905340, 0) to dev "naa.600605b0052b76a0189a635b295ad161" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:52:47.776Z cpu52:334261)ScsiDeviceIO: 2646: Cmd(0x43a68d905340) 0x1a, CmdSN 0x1481c from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:52:47.819Z cpu11:318303)ScsiDeviceIO: 2646: Cmd(0x439e80e9c580) 0x1a, CmdSN 0x14821 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:52:48.186Z cpu9:33752)ScsiDeviceIO: 2646: Cmd(0x439e900198c0) 0x1a, CmdSN 0x14826 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:53:22.789Z cpu69:33669)NMP: nmp_ResetDeviceLogThrottling:3345: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 3 times
2018-03-06T11:53:46.893Z cpu47:33395)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a680e6b0c0, 0) to dev "naa.600605b0052b76a0189a635b295ad161" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2018-03-06T11:53:46.893Z cpu47:33395)ScsiDeviceIO: 2646: Cmd(0x43a680e6b0c0) 0x1a, CmdSN 0x1482b from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:53:46.908Z cpu13:33281)ScsiDeviceIO: 2646: Cmd(0x439e90dfb380) 0x1a, CmdSN 0x14830 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:53:47.237Z cpu19:33287)ScsiDeviceIO: 2646: Cmd(0x439e80e83f80) 0x1a, CmdSN 0x14835 from world 0 to dev "naa.600605b0052b76a0189a635b295ad161" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-03-06T11:54:22.790Z cpu69:33669)NMP: nmp_ResetDeviceLogThrottling:3345: last error status from device naa.600605b0052b76a0189a635b295ad161 repeated 2 times
2018-03-06T11:55:52.899Z cpu32:32895)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x43a28d7c09c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
 
Old 03-07-2018, 02:49 AM   #4
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 8,924

Rep: Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916
I was thinking that I was going to see sdb errors but it appears that you have a raid system. Is sdb involved in the raid? You're cruising for a bruising if you mix raid and individual disk accesses.
 
  


Reply

Tags
scsi


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Firewalld timeout and failure on Fedora 25 Reksio Linux - Security 11 02-20-2017 03:58 PM
Script for disk timeout value of Linux VM disk in NFS during failover/failback? rajivdp Linux - Virtualization and Cloud 1 06-01-2011 03:48 AM
External hard drive failure? Sense key: medium error Mister_Abc Linux - Hardware 6 07-23-2009 10:27 AM
is this a disk failure msg - hdc: request senense failure walterj Linux - Hardware 3 03-25-2008 11:01 PM
Hard drive failure error? (sense key Medium error) Arodef Linux - Hardware 2 11-02-2004 12:39 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud

All times are GMT -5. The time now is 11:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration