3 machines down- please peek at my logs

LarryFrigginWachs · 04-26-2006, 09:27 AM

Hi Folks-

I previously posted about 2 out of 3 of my identical machines freezing. Now the third is sick. Random freezes.
Have to reboot daily after being idle. I think it is something to do with Samba/networking but am not sure.
One is a webserver/samba server. The other two are samba servers. All on same network.
I forgot to add: 2 fc4 machines, the third is centOS. All on identical IBM towers. It is not a heat issue.

Thanks very much. Larry is freakin out here. I wish I could post a reward for solving this problem. I promise to send good karma to whomever can help.

Log exceApr 25 19:10:16 localhost smbd[5908]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:16 localhost smbd[5920]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:24 localhost smbd[5938]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:24 localhost smbd[5976]: write_socket_data: write
failure. Error = Connection reset by peer
Apr 25 19:10:24 localhost smbd[5986]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:28 localhost smbd[5997]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:28 localhost smbd[6024]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:32 localhost smbd[6039]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:41 localhost smbd[6058]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:45 localhost smbd[6066]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:45 localhost smbd[6068]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:49 localhost smbd[6103]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:49 localhost smbd[6107]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:53 localhost smbd[6113]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:57 localhost smbd[6116]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:10:57 localhost smbd[6126]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:12:50 localhost smbd[5267]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:13:11 localhost smbd[5285]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:13:48 localhost smbd[5604]: [2006/04/25 19:13:48, 0]
lib/util_sock.c:read_socket_data(384)
Apr 25 19:15:24 localhost smbd[5712]: [2006/04/25 19:15:24, 0]
lib/util_sock.c:read_socket_data(384)
Apr 25 19:15:33 localhost smbd[5332]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:16:06 localhost smbd[5874]: [2006/04/25 19:16:06, 0]
lib/util_sock.c:write_socket(455)
Apr 25 19:16:18 localhost smbd[5898]: [2006/04/25 19:16:18, 0]
lib/util_sock.c:read_socket_data(384)
Apr 25 19:16:31 localhost smbd[5976]: [2006/04/25 19:16:31, 0]
lib/util_sock.c:write_socket(455)
Apr 25 19:17:08 localhost smbd[5604]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:17:08 localhost smbd[5712]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:17:13 localhost smbd[5874]: write_socket: Error writing 4
bytes to socket 5: ERRNO = Connection reset by peer
Apr 25 19:17:17 localhost smbd[5898]: read_socket_data: recv failure
for 4. Error = Connection reset by peer
Apr 25 19:17:17 localhost smbd[5976]: write_socket: Error writing 4
bytes to socket 5: ERRNO = Connection reset by peer
Apr 25 19:17:25 localhost smbd[5874]: [2006/04/25 19:17:25, 0]
lib/util_sock.c:send_smb(647)
Apr 25 19:17:29 localhost smbd[5976]: [2006/04/25 19:17:29, 0]
lib/util_sock.c:send_smb(647)
Apr 25 19:17:33 localhost smbd[5874]: Error writing 4 bytes to client.
-1. (Connection reset by peer)
Apr 25 19:17:38 localhost smbd[5976]: Error writing 4 bytes to client.
-1. (Connection reset by peer)
Apr 25 19:26:35 localhost smbd[6286]: [2006/04/25 19:26:35, 0]
lib/util_sock.c:write_socket_data(430)
Apr 25 19:26:35 localhost smbd[6286]: write_socket_data: write
failure. Error = Connection reset by peer
Apr 25 19:26:35 localhost smbd[6286]: [2006/04/25 19:26:35, 0]
lib/util_sock.c:write_socket(455)
Apr 25 19:26:35 localhost smbd[6286]: write_socket: Error writing 4
bytes to socket 5: ERRNO = Connection reset by peer
Apr 25 19:26:35 localhost smbd[6286]: [2006/04/25 19:26:35, 0]
lib/util_sock.c:send_smb(647)
Apr 25 19:26:35 localhost smbd[6286]: Error writing 4 bytes to client.
-1. (Connection reset by peer)
Apr 25 19:41:16 localhost nmbd[2606]: [2006/04/25 19:41:16, 0]
nmbd/nmbd_namequery.c:query_name_response(101)
Apr 25 19:41:16 localhost nmbd[2606]: query_name_response: Multiple
(2) responses received for a query on subnet 192.168.1.116 for name
MYGROUP<1d>.
Apr 25 19:41:16 localhost nmbd[2606]: This response was from IP
192.168.1.118, reporting an IP address of 192.168.1.118.
Apr 25 19:55:59 localhost smbd[6458]: [2006/04/25 19:55:59, 0]
lib/util_sock.c:write_socket_data(430)
Apr 25 19:55:59 localhost smbd[6458]: write_socket_data: write
failure. Error = Connection reset by peer
Apr 25 19:55:59 localhost smbd[6458]: [2006/04/25 19:55:59, 0]
lib/util_sock.c:write_socket(455)
Apr 25 19:55:59 localhost smbd[6458]: write_socket: Error writing 4
bytes to socket 24: ERRNO = Connection reset by peer
Apr 25 19:55:59 localhost smbd[6458]: [2006/04/25 19:55:59, 0]
lib/util_sock.c:send_smb(647)
Apr 25 19:55:59 localhost smbd[6458]: Error writing 4 bytes to client.
-1. (Connection reset by peer)
Apr 25 20:01:19 localhost nmbd[2606]: [2006/04/25 20:01:19, 0]
nmbd/nmbd_namequery.c:query_name_response(101)
Apr 25 20:01:19 localhost nmbd[2606]: query_name_response: Multiple
(2) responses received for a query on subnet 192.168.1.116 for name
MYGROUP<1d>.
Apr 25 20:01:19 localhost nmbd[2606]: This response was from IP
192.168.1.118, reporting an IP address of 192.168.1.118.
Apr 26 08:33:46 localhost syslogd 1.4.1: restart

Machine 2 excerpt 21:04:09 www smbd[4966]: write_socket_data: write failure.
> Error = Connection reset by peer
> Apr 25 21:04:09 www smbd[4966]: [2006/04/25 21:04:09, 0]
> lib/util_sock.c:write_socket(455)
> Apr 25 21:04:09 www smbd[4966]: write_socket: Error writing 4 bytes to
> socket 5: ERRNO = Connection reset by peer
> Apr 25 21:04:09 www smbd[4966]: [2006/04/25 21:04:09, 0]
> lib/util_sock.c:send_smb(647)
> Apr 25 21:04:09 www smbd[4966]: Error writing 4 bytes to client. -1.
> (Connection reset by peer)
> Apr 25 21:36:12 www smbd[4973]: [2006/04/25 21:36:12, 0]
> lib/util_sock.c:get_peer_addr(1000)
> Apr 25 21:36:12 www smbd[4973]: getpeername failed. Error was
> Transport endpoint is not connected
> Apr 25 21:36:12 www smbd[4973]: [2006/04/25 21:36:12, 0]
> lib/util_sock.c:get_peer_addr(1000)
> Apr 25 21:36:12 www smbd[4973]: getpeername failed. Error was
> Transport endpoint is not connected
> Apr 25 21:36:12 www smbd[4973]: [2006/04/25 21:36:12, 0]
> lib/util_sock.c:write_socket_data(430)
> Apr 25 21:36:12 www smbd[4973]: write_socket_data: write failure.
> Error = Connection reset by peer
> Apr 25 21:36:12 www smbd[4973]: [2006/04/25 21:36:12, 0]
> lib/util_sock.c:write_socket(455)
> Apr 25 21:36:12 www smbd[4973]: write_socket: Error writing 4 bytes to
> socket 23: ERRNO = Connection reset by peer
> Apr 25 21:36:12 www smbd[4973]: [2006/04/25 21:36:12, 0]
> lib/util_sock.c:send_smb(647)
> Apr 25 21:36:12 www smbd[4973]: Error writing 4 bytes to client. -1.
> (Connection reset by peer)
> Apr 25 22:01:01 www crond(pam_unix)[4977]: session opened for user root
> by (uid=0)
> Apr 25 22:01:01 www crond(pam_unix)[4977]: session closed for user root
> Apr 25 22:08:17 www smbd[4980]: [2006/04/25 22:08:17, 0]
> lib/util_sock.c:get_peer_addr(1000)
> Apr 25 22:08:17 www smbd[4980]: getpeername failed. Error was
> Transport endpoint is not connected
> Apr 25 22:08:17 www smbd[4980]: [2006/04/25 22:08:17, 0]
> lib/util_sock.c:write_socket_data(430)
> Apr 25 22:08:17 www smbd[4980]: write_socket_data: write failure.
> Error = Connection reset by peer
> Apr 25 22:08:17 www smbd[4980]: [2006/04/25 22:08:17, 0]
> lib/util_sock.c:write_socket(455)
> Apr 25 22:08:17 www smbd[4980]: write_socket: Error writing 4 bytes to
> socket 5: ERRNO = Connection reset by peer
> Apr 25 22:08:17 www smbd[4980]: [2006/04/25 22:08:17, 0]
> lib/util_sock.c:send_smb(647)
> Apr 25 22:08:17 www smbd[4980]: Error writing 4 bytes to client. -1.
> (Connection reset by peer)
> Apr 25 22:40:31 www smbd[4987]: [2006/04/25 22:40:31, 0]
> lib/util_sock.c:get_peer_addr(1000)
> Apr 25 22:40:31 www smbd[4987]: getpeername failed. Error was
> Transport endpoint is not connected
> Apr 25 22:40:31 www smbd[4987]: [2006/04/25 22:40:31, 0]
> lib/util_sock.c:get_peer_addr(1000)
> Apr 25 22:40:31 www smbd[4987]: getpeername failed. Error was
> Transport endpoint is not connected
> Apr 25 22:40:31 www smbd[4987]: [2006/04/25 22:40:31, 0]
> lib/util_sock.c:write_socket_data(430)
> Apr 25 22:40:31 www smbd[4987]: write_socket_data: write failure.
> Error = Connection reset by peer
> Apr 25 22:40:31 www smbd[4987]: [2006/04/25 22:40:31, 0]
> lib/util_sock.c:write_socket(455)
> Apr 25 22:40:31 www smbd[4987]: write_socket: Error writing 4 bytes to
> socket 23: ERRNO = Connection reset by peer
> Apr 25 22:40:31 www smbd[4987]: [2006/04/25 22:40:31, 0]
> lib/util_sock.c:send_smb(647)
> Apr 25 22:40:31 www smbd[4987]: Error writing 4 bytes to client. -1.
> (Connection reset by peer)
> Apr 25 23:01:03 www crond(pam_unix)[4989]: session opened for user root
> by (uid=0)
> Apr 25 23:01:03 www crond(pam_unix)[4989]: session closed for user root
> Apr 25 23:13:37 www shutdown: shutting down for system halt
> Apr 25 23:13:37 www init: Switching to runlevel: 0
> Apr 26 08:36:38 www syslogd 1.4.1: restart.
Machine 3
unsolicited break reply - dumping info.
Apr 25 21:20:09 localhost smbd[5613]: [2006/04/25 21:20:09, 0]
lib/util_sock.c:write_socket(455)
Apr 25 21:20:13 localhost smbd[5609]: [2006/04/25 21:20:13, 0]
smbd/oplock.c

rocess_local_message(435)
Apr 25 21:20:13 localhost smbd[5612]: [2006/04/25 21:20:13, 0]
smbd/oplock.c

rocess_local_message(435)
Apr 25 21:20:17 localhost smbd[5613]: write_socket: Error writing 107
bytes to socket 5: ERRNO = Broken pipe
Apr 25 21:20:17 localhost smbd[5609]: process_local_message:
unsolicited oplock break reply from pid 5609, port 32796, dev = fd00,
inode = 1966087, file_id = 7
Apr 25 21:20:21 localhost smbd[5612]: process_local_message:
unsolicited oplock break reply from pid 5612, port 32796, dev = fd00,
inode = 1966087, file_id = 7
Apr 25 21:20:21 localhost smbd[5613]: [2006/04/25 21:20:21, 0]
lib/util_sock.c:send_smb(647)
Apr 25 21:20:34 localhost smbd[5613]: Error writing 107 bytes to
client. -1. (Broken pipe)
Apr 25 21:20:34 localhost smbd[5613]: [2006/04/25 21:20:34, 0]
smbd/oplock.c

rocess_local_message(420)
Apr 25 21:20:38 localhost smbd[5613]: process_local_message: Received
unsolicited break reply - dumping info.
Apr 25 21:20:38 localhost smbd[5613]: [2006/04/25 21:20:38, 0]
smbd/oplock.c

rocess_local_message(435)
Apr 25 21:20:38 localhost smbd[5613]: process_local_message:
unsolicited oplock break reply from pid 5613, port 32796, dev = fd00,
inode = 1966087, file_id = 7
Apr 25 21:36:58 localhost nmbd[2451]: [2006/04/25 21:36:58, 0]
nmbd/nmbd_namequery.c:query_name_response(101)
Apr 25 21:36:58 localhost nmbd[2451]: query_name_response: Multiple
(2) responses received for a query on subnet 192.168.1.200 for name
MYGROUP<1d>.
Apr 25 21:36:58 localhost nmbd[2451]: This response was from IP
192.168.1.118, reporting an IP address of 192.168.1.118.
Apr 26 08:37:48 localhost syslogd 1.4.1: restart

ataraxia · 04-27-2006, 06:33 PM

How's you're confidence in the network equipment between these machines? Cables ok, switch configured properly and in good shape? I've seen dying switches cause this sort of thing.

LarryFrigginWachs · 04-28-2006, 07:40 AM

The network connections are flawless, switch, etc have never given me any trouble. The first machine I had as a Samba server for almost a year with no problems at all-
I don't know if this is relevant but I also noticed that two of the machines appear to have been hacked via samba- all kinds of weird names in the smba logs- makes sense since most of this started when I put one of the machines up as a web server.
I may start another thread in the sec. forum, but I am really not sure which direction to go in (hardware issue vs hack vs combination of these).
I know this is confusing and I apologize- is there anything else I can post that would be helpful in diagnosis? The centOS machine is not on the network but it freezes after a couple hours, here is what is weird: If I move the mouse on the centoOS machine, it seems to "help"- that is to say as long as I keep moving the mouse, the gui responds, example: I type "top" nothing happens- unless I continue to roll the mouse around.