Task blocked for more than 120 seconds errors and crashes

sanjay87 · 06-18-2012, 05:11 AM

Hi all,
Today one of my virtual server got hangs after prompting for ssh password.When i try to restart the server application service httpd restart it get hangs.When i contact my hosting provider to rebooted the server .The issue get solved after reboot after few hour i m facing out same problem .Service provider recommonded the cause is due to

Quote:

.Im not running out of memory

.

And they recommend to exe the following command on my server after reboot

echo 0 > /proc/sys/kernel/hung_task_timeout_secs

Quote:

#fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00029fef

Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 1084 8192000 82 Linux swap / Solaris
/dev/sda3 1084 60667 478599168 83 Linux

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00029fef

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2 64 1084 8192000 82 Linux swap / Solaris
/dev/sdb3 1084 60667 478599168 83 Linux

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450: 499.0 GB, 498999492608 bytes
255 heads, 63 sectors/track, 60666 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00029fef

Device Boot Start End Blocks Id System
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p2 64 1084 8192000 82 Linux swap / Solaris
/dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p3 1084 60667 478599168 83 Linux

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1: 524 MB, 524288000 bytes
255 heads, 63 sectors/track, 63 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p1 doesn't contain a valid partition table

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p2: 8388 MB, 8388608000 bytes
255 heads, 63 sectors/track, 1019 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p2 doesn't contain a valid partition table

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p3: 490.1 GB, 490085548032 bytes
255 heads, 63 sectors/track, 59582 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/ddf1_4c53492020202020808627c3000000004711471100001450p3 doesn't contain a valid partition table
[root@SW ~]#

ssh hangs over log

Quote:

debug1: Found key in /root/.ssh/known_hosts:5

debug1: ssh_rsa_verify: signature correct

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

debug1: Next authentication method: gssapi-keyex

debug1: No valid Key exchange context

debug1: Next authentication method: gssapi-with-mic

debug1: Unspecified GSS failure. Minor code may provide more information

Credentials cache file '/tmp/krb5cc_0' not found

Quote:

debug1: Unspecified GSS failure. Minor code may provide more information

Credentials cache file '/tmp/krb5cc_0' not found

debug1: Unspecified GSS failure. Minor code may provide more information

debug1: Next authentication method: publickey

debug1: Offering public key: /root/.ssh/id_rsa

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

debug1: Trying private key: /root/.ssh/identity

debug1: Trying private key: /root/.ssh/id_dsa

debug1: Next authentication method: password

root@xx.xx's password:

debug1: Authentication succeeded (password).

debug1: channel 0: new [client-session]

debug1: Requesting no-more-sessions@openssh.com

debug1: Entering interactive session.

debug1: Sending environment.

debug1: Sending env LANG = en_US.UTF-8

robertjinx · 06-18-2012, 07:43 AM

Disable GSSApi in sshd_config "GSSAPIAuthentication no", it might help.

---------- Post added 06-18-12 at 14:44 ----------

Ohh sorry, its not sshd, or GSS might not have anything to do with it.

sanjay87 · 06-18-2012, 09:43 AM

In my ssh configuration i could find GSSAPIAuthentication yes.Whether these could hangs my server .Whether dns issue can cause my server to hangs

Currently my server hostname is not resolved with the ipaddress of my server .

Quote:

#cat /etc/ssh/sshd_config

GSSAPIAuthentication yes

Quote:

#dmesg
ata2.00: exception Emask 0x0 SAct 0x1f SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: WRITE FPDMA QUEUED
ata2.00: cmd 61/28:18:20:c9:8e/00:00:1d:00:00/40 tag 3 ncq 20480 out
res 41/10:28:26:c9:8e/00:00:1d:00:00/00 Emask 0x481 (invalid argument) <F>
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x1f SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: WRITE FPDMA QUEUED
ata2.00: cmd 61/18:00:48:c9:8e/00:00:1d:00:00/40 tag 0 ncq 12288 out
res 41/10:18:4a:c9:8e/00:00:1d:00:00/00 Emask 0x481 (invalid argument) <F>
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x10 SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: WRITE FPDMA QUEUED
ata2.00: cmd 61/18:20:48:c9:8e/00:00:1d:00:00/40 tag 4 ncq 12288 out
res 41/10:18:21:c9:8e/00:00:1d:00:00/00 Emask 0x481 (invalid argument) <F>
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: WRITE FPDMA QUEUED
ata2.00: cmd 61/18:00:48:c9:8e/00:00:1d:00:00/40 tag 0 ncq 12288 out
res 41/10:18:4f:c9:8e/00:00:1d:00:00/00 Emask 0x481 (invalid argument) <F>
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: WRITE FPDMA QUEUED
ata2.00: cmd 61/18:00:48:c9:8e/00:00:1d:00:00/40 tag 0 ncq 12288 out
res 41/10:18:49:c9:8e/00:00:1d:00:00/00 Emask 0x481 (invalid argument) <F>
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000008
ata2.00: failed command: WRITE FPDMA QUEUED
ata2.00: cmd 61/18:00:48:c9:8e/00:00:1d:00:00/40 tag 0 ncq 12288 out
res 41/10:18:4a:c9:8e/00:00:1d:00:00/00 Emask 0x481 (invalid argument) <F>
ata2.00: status: { DRDY ERR }
ata2.00: error: { IDNF }
ata2.00: configured for UDMA/133
sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 1:0:0:0: [sdb] Sense Key : Aborted Command [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 0b 14 00 00 00 00 0c 00 0a 80 00 00 00 00 00
1d 8e c9 4a
sd 1:0:0:0: [sdb] Add. Sense: Recorded entity not found
sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 1d 8e c9 48 00 00 18 00
end_request: I/O error, dev sdb, sector 495896904
ata2: EH complete

robertjinx · 06-19-2012, 01:51 AM

Looks a lot like a disk/ide issue. Try to search for the errors on google, might give you some hints.