Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I/O to disk sdb blocked for over 120s. lvm and sadc are victim processes that were prevented from running as a result. You need to figure out which disk is sdb and why I/O isn't proceeding to fix the underlying problem.
I/O to disk sdb blocked for over 120s. lvm and sadc are victim processes that were prevented from running as a result. You need to figure out which disk is sdb and why I/O isn't proceeding to fix the underlying problem.
You probably have a serious disk hardware or controller issue on your hands!
(But, and I am quite serious ... "check the obvious things first." Press on every cable to be sure it's firmly seated. Check the power cords, UPS boxes (are they on?), fuses. Press down on every controller card to see if it's firmly seated. And so on.)
You should be able to run s.m.a.r.t. internal-diagnostics on the drive – if it's still running – to see what it reports about its own internal status. (These facilities are built-in to the drive hardware and can also therefore be used with Windows, AIX, and so on.)
Last edited by sundialsvcs; 04-06-2018 at 10:31 AM.
Is /dev/sdb an iSCSI, or other network based disk? It is not uncommon to use a network disk of some type with ESXi. If sdb is a network device, check the NAS/SANs device and make sure it not disconnecting for some reason, or being held up by another server that may be sharing the device, as that can cause the "blocked for more than 120 seconds" message at the ESXi server.
Other possibilities are:
A VM or other process is writing a very large file to sdb that takes more than 120 seconds, causing other processes to wait.
If a VM is using a physical disk partition for its disk, as opposed to a image file, and that VM is doing lots of disk writes.
The blocked for more 120 seconds message is not necessarily an error, as the blocked operation will continue once it is no longer blocked, athough it may appear forzen while it is blocked.
Last edited by anotherlinuxuser; 04-08-2018 at 12:03 AM.
Is /dev/sdb an iSCSI, or other network based disk? It is not uncommon to use a network disk of some type with ESXi. If sdb is a network device, check the NAS/SANs device and make sure it not disconnecting for some reason, or being held up by another server that may be sharing the device, as that can cause the "blocked for more than 120 seconds" message at the ESXi server.
Other possibilities are:
A VM or other process is writing a very large file to sdb that takes more than 120 seconds, causing other processes to wait.
If a VM is using a physical disk partition for its disk, as opposed to a image file, and that VM is doing lots of disk writes.
The blocked for more 120 seconds message is not necessarily an error, as the blocked operation will continue once it is no longer blocked, athough it may appear forzen while it is blocked.
Hi,
It is not a network based disk. It is a disk plugged into the server running ESXi hypervisor, where the disk are emulated to VM as a file. Also the deployment has only one node. And like understand what made the kernel process to block for more than 120 seconds.
also anyidea on the following, can it be the cause ?
[2441143.217000] sd 2:0:1:0: [sdb] task abort on host 2, ffff880c01e0c3c0
[2441143.217000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff880c01e0c3c0
[2441143.217000] sd 2:0:1:0: [sdb] task abort on host 2, ffff88075fc79cc0
[2441143.217000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff88075fc79cc0
The impact we are seeing is the applications are not responding, believe it is because of process being blocked for more than 120 seconds.
Hi,
It is not a network based disk. It is a disk plugged into the server running ESXi hypervisor, where the disk are emulated to VM as a file. Also the deployment has only one node. And like understand what made the kernel process to block for more than 120 seconds. also anyidea on the following, can it be the cause ?
Code:
[2441143.217000] sd 2:0:1:0: [sdb] task abort on host 2, ffff880c01e0c3c0
[2441143.217000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff880c01e0c3c0
[2441143.217000] sd 2:0:1:0: [sdb] task abort on host 2, ffff88075fc79cc0
[2441143.217000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff88075fc79cc0
The impact we are seeing is the applications are not responding, believe it is because of process being blocked for more than 120 seconds.
...and, bumping your own thread and not providing ANY further info
Quote:
Originally Posted by barani_pdy
Request input for the above query.
We volunteer our time here; bumping your own thread and providing no additional information is fairly rude. You have been told clearly that you need to figure out where/what that sdb device is. Have you done ANY of the several troubleshooting steps given to you thus far? What were the results? What version/distro of Linux??
And since you appear to need a quick solution, you're using VMWare ESXi....which is a commercial, PAY-FOR product. So have you contacted VMWare support???
The problem is that 2 processes (sadc and lvm) are trying to write something to disk 'sdb'. These 2 processes are being blocked by a another process that is writing to disk, but because the process doing the blocking is not blocked itself, it does not produce warnings or errors. In other words, the process that is hogging the disk is doing its thing without any errors, so you are left having to figure out what process is hogging the disk some other way.
For example, a VM doing back up to a local VM disk image file will be writing LOTS of data as quickly as possible, so that VM will receive much more disk time than other processes that occasionally write a few bytes to disk. These other processes can end up waiting more than 120 seconds to do a write, and produce the warnings you are getting. Given enough time, and provided programs don't time out while waiting, everything will get written to disk and tasks will carry on, but of course, waiting for things for long periods of time is not optimal.
Let's breakdown what you have here. These are the most informative lines, with brief explanations.
[2441143.217000] sd 2:0:1:0: [sdb] task abort on host 2, ffff880c01e0c3c0
[2441143.217000] sd 2:0:1:0: [sdb] Failed to abort cmd ffff880c01e0c3c0
A 'task' that is trying to use disk sdb was aborted, but the task could not be aborted. A blocked process cannot be aborted until it is unblocked, so the abort failed.
The 'task' could be anything that writes anything to disk. On VM servers, most of disk writing is done by VMs, but any program or command that produces disk write will wait for a long time to complete the disk write, because not completing a committed write will cause file system issues.
The abort can come from a 'kill' command, a CTRL-C at the CLI, or another program terminating a child process, like a hypervisor that is told to force a VM to stop. Unfortunately, we cannot tell what the 'task' is.
The first blocked task's name and PID# are shown in this line.
[2441049.121000] INFO: task sadc:11031 blocked for more than 120 seconds.
Task name: sadc, PID#: 11031
This PID# can be found in the process list (see 'ps --help') but only while this instance of the process is still running. If the process becomes unblocked and finishes, or it dies for any reason, or the system is rebooted, the process will no longer exist, so you will lose the opportunity to examine the full command line.
Further down we that PID 11031 was trying to commit changes to disk
[2441049.121000] [<ffffffff8119b8ff>] log_wait_commit+0xc5/0x11b
The LVM task (PID 2528), likely for /dev/sdb, is also blocked:
[2792409.478000] INFO: task lvm:2528 blocked for more than 120 seconds.
This is not unusual, as tasks writing to a Logical Volume need to go through a LVM process.
Use the 'iotop' and 'top' commands to see what process(es) are using the disk and cpu resources, and, if possible, stop suspected blocking processes without killing the processes. Trying to force a process to die may just make things worse.
As you stop disk busy programs, use 'top' and 'ps' to see if the blocked PIDs complete and disappear from the process list.
If they do, then the last process you stopped was doing the blocking. What you do from there depends on what you find.
Last edited by anotherlinuxuser; 04-12-2018 at 10:10 PM.
Reason: Spelling corrections
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
