Just when I thought things were good... the server decides to reboot. This is the first reboot since last Monday however, we receive something in the logs that looks promising.

Nov 14 11:16:49 localhost kernel: WARNING: at kernel/sched_fair.c:988 hrtick_start_fair+0x18b/0x190() (Not tainted)
Nov 14 11:16:49 localhost kernel: Hardware name: X8DTH-i/6/iF/6F
Nov 14 11:16:49 localhost kernel: Modules linked in: mpt2sas scsi_transport_sas raid_class mptctl mptbase nfsd lockd nfs_acl auth_rpcgss autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 xfs exportfs dm_mirror dm_region_hash dm_log ses enclosure ixgbe mdio igb microcode serio_raw ghes hed sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 megaraid_sas(U) sd_mod crc_t10dif ahci dm_mod [last unloaded: scsi_wait_scan]
Nov 14 11:16:49 localhost kernel: Pid: 2751, comm: nfsd Not tainted 2.6.32-131.17.1.el6.x86_64 #1
Nov 14 11:16:49 localhost kernel: Call Trace:
Nov 14 11:16:49 localhost kernel: [<ffffffff810670f7>] ? warn_slowpath_common+0x87/0xc0
Nov 14 11:16:49 localhost kernel: [<ffffffff8106714a>] ? warn_slowpath_null+0x1a/0x20
Nov 14 11:16:49 localhost kernel: [<ffffffff81050b5b>] ? hrtick_start_fair+0x18b/0x190
Nov 14 11:16:49 localhost kernel: [<ffffffff8106067b>] ? dequeue_task_fair+0x8b/0x90
Nov 14 11:16:49 localhost kernel: [<ffffffff8110d3d0>] ? sync_page+0x0/0x50
Nov 14 11:16:49 localhost kernel: [<ffffffff8105727c>] ? dequeue_task+0x9c/0xb0
Nov 14 11:16:49 localhost kernel: [<ffffffff810572be>] ? deactivate_task+0x2e/0x40
Nov 14 11:16:49 localhost kernel: [<ffffffff814db079>] ? thread_return+0x120/0x777
Nov 14 11:16:49 localhost kernel: [<ffffffff81098d19>] ? ktime_get_ts+0xa9/0xe0
Nov 14 11:16:49 localhost kernel: [<ffffffff8110d3d0>] ? sync_page+0x0/0x50
Nov 14 11:16:49 localhost kernel: [<ffffffff814db743>] ? io_schedule+0x73/0xc0
Nov 14 11:16:49 localhost kernel: [<ffffffff8110d40d>] ? sync_page+0x3d/0x50
Nov 14 11:16:49 localhost kernel: [<ffffffff814dbfaf>] ? __wait_on_bit+0x5f/0x90
Nov 14 11:16:49 localhost kernel: [<ffffffff8110d5c3>] ? wait_on_page_bit+0x73/0x80
Nov 14 11:16:49 localhost kernel: [<ffffffff8108e1c0>] ? wake_bit_function+0x0/0x50
Nov 14 11:16:49 localhost kernel: [<ffffffff811232d5>] ? pagevec_lookup_tag+0x25/0x40
Nov 14 11:16:49 localhost kernel: [<ffffffff8110d9db>] ? wait_on_page_writeback_range+0xfb/0x190
Nov 14 11:16:49 localhost kernel: [<ffffffff81122481>] ? do_writepages+0x21/0x40
Nov 14 11:16:49 localhost kernel: [<ffffffff8110db2b>] ? __filemap_fdatawrite_range+0x5b/0x60
Nov 14 11:16:49 localhost kernel: [<ffffffff8110dba8>] ? filemap_write_and_wait_range+0x78/0x90
Nov 14 11:16:49 localhost kernel: [<ffffffffa026510f>] ? xfs_write+0xa2f/0xb70 [xfs]
Nov 14 11:16:49 localhost kernel: [<ffffffff81417845>] ? memcpy_toiovec+0x55/0x80
Nov 14 11:16:49 localhost kernel: [<ffffffffa0260800>] ? xfs_file_aio_write+0x0/0x70 [xfs]
Nov 14 11:16:49 localhost kernel: [<ffffffffa0260861>] ? xfs_file_aio_write+0x61/0x70 [xfs]
Nov 14 11:16:49 localhost kernel: [<ffffffff8117241b>] ? do_sync_readv_writev+0xfb/0x140
Nov 14 11:16:49 localhost kernel: [<ffffffff8118aefd>] ? d_obtain_alias+0x4d/0x160
Nov 14 11:16:49 localhost kernel: [<ffffffff8108e180>] ? autoremove_wake_function+0x0/0x40
Nov 14 11:16:49 localhost kernel: [<ffffffff812056a6>] ? security_task_setgroups+0x16/0x20
Nov 14 11:16:49 localhost kernel: [<ffffffff81205346>] ? security_file_permission+0x16/0x20
Nov 14 11:16:49 localhost kernel: [<ffffffff811734df>] ? do_readv_writev+0xcf/0x1f0
Nov 14 11:16:49 localhost kernel: [<ffffffffa0482852>] ? nfsd_setuser_and_check_port+0x62/0xb0 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffff81173646>] ? vfs_writev+0x46/0x60
Nov 14 11:16:49 localhost kernel: [<ffffffffa04843d7>] ? nfsd_vfs_write+0x107/0x430 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffff8116fe82>] ? dentry_open+0x52/0xc0
Nov 14 11:16:49 localhost kernel: [<ffffffffa04869fe>] ? nfsd_open+0x13e/0x210 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffffa0486e87>] ? nfsd_write+0xe7/0x100 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffffa048e7df>] ? nfsd3_proc_write+0xaf/0x140 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffffa047f43e>] ? nfsd_dispatch+0xfe/0x240 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffffa03f54f4>] ? svc_process_common+0x344/0x640 [sunrpc]
Nov 14 11:16:49 localhost kernel: [<ffffffff8105dc20>] ? default_wake_function+0x0/0x20
Nov 14 11:16:49 localhost kernel: [<ffffffffa03f5b30>] ? svc_process+0x110/0x160 [sunrpc]
Nov 14 11:16:49 localhost kernel: [<ffffffffa047fb62>] ? nfsd+0xc2/0x160 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffffa047faa0>] ? nfsd+0x0/0x160 [nfsd]
Nov 14 11:16:49 localhost kernel: [<ffffffff8108de16>] ? kthread+0x96/0xa0
Nov 14 11:16:49 localhost kernel: [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
Nov 14 11:16:49 localhost kernel: [<ffffffff8108dd80>] ? kthread+0x0/0xa0
Nov 14 11:16:49 localhost kernel: [<ffffffff8100c1c0>] ? child_rip+0x0/0x20

I still think that you have some sort of hardware issue. Just my guess though. Hopefully you can get this resolved.

Hey eur0disciple did you find any solution to the problem. I am encountering the same problem on HP DL380 G7 server running RHEL 6.1 with kernel 2.6.32-131.0.15.el6.x86_64

While this is an old thread that's similar, your situation is a bit different. We'll suggest to you what we suggested to the OP...contact Red Hat support and HP support. You're paying for it with your RHEL subscription, and they can analyze what you get from the system tools.

The OP's problem was probably hardware related...have you checked your hardware, and had HP run a diagnostic?