| hsnanua |
09-19-2011 09:47 PM |
Crashed!
Dear all,
small issue came out yesterday. My server crashed and i believe was due to the network. Ok, before that, this is a loadbalancer, that is connected to another. So, this is LB-01 and there is LB-02. LB-01 is running on keepalived, ipvs, iptables. LB-02 used to run on the same, but i removed keepalived and installed redhat cluster suite (piranha). I started piranha when the server went down.
Below is the logs. It came up again when i shut down the LB-02.
Below are the logs from /var/logs/messages
Code:
Sep 19 11:31:42 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:31:42 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:31:42 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:31:42 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:31:47 lb-01 sshd[19197]: Did not receive identification string from 10.103.4.40
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: Removing slave eth0
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - 00:21:9B:FD:7A:32 - is still in use by bond0. Set the H
Waddr of eth0 to a different address to avoid conflicts.
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: releasing active interface eth0
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: making interface eth1 the new active one.
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: Removing slave eth1
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: releasing active interface eth1
Sep 19 11:34:25 lb-01 Keepalived_vrrp: Kernel is reporting: interface bond0.18 DOWN
Sep 19 11:34:25 lb-01 kernel: IPVS: stopping sync thread 29767 ...
Sep 19 11:34:25 lb-01 Keepalived_vrrp: VRRP_Instance(VI1) Entering FAULT STATE
Sep 19 11:34:25 lb-01 Keepalived_vrrp: Netlink: error:
Sep 19 11:34:25 lb-01 Keepalived_vrrp: Netlink: error: Cannot assign requested address, type=(21), seq=1316071292, pid=0
Sep 19 11:34:25 lb-01 Keepalived_vrrp: VRRP_Instance(VI1) Now in FAULT state
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: setting mode to active-backup (1).
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: Setting MII monitoring interval to 500.
Sep 19 11:34:25 lb-01 kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready
Sep 19 11:34:25 lb-01 kernel: IPVS: sync thread stopped!
Sep 19 11:34:25 lb-01 kernel: IPVS: Error joining to the multicast group
Sep 19 11:34:25 lb-01 kernel: IPVS: sync thread stopped!
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: Adding slave eth0.
Sep 19 11:34:25 lb-01 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: enslaving eth0 as a backup interface with a down link.
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: Adding slave eth1.
Sep 19 11:34:25 lb-01 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Sep 19 11:34:25 lb-01 kernel: bonding: bond0: enslaving eth1 as a backup interface with a down link.
Sep 19 11:34:28 lb-01 kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Sep 19 11:34:28 lb-01 kernel: tg3: eth0: Flow control is off for TX and off for RX.
Sep 19 11:34:28 lb-01 kernel: tg3: eth0: Flow control is off for TX and off for RX.
Sep 19 11:34:28 lb-01 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 19 11:34:28 lb-01 kernel: tg3: eth1: Link is up at 1000 Mbps, full duplex.
Sep 19 11:34:28 lb-01 kernel: tg3: eth1: Flow control is off for TX and off for RX.
Sep 19 11:34:28 lb-01 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Sep 19 11:34:28 lb-01 kernel: bonding: bond0: link status definitely up for interface eth0.
Sep 19 11:34:28 lb-01 kernel: bonding: bond0: link status definitely up for interface eth1.
Sep 19 11:34:28 lb-01 kernel: bonding: bond0: making interface eth0 the new active one.
Sep 19 11:34:28 lb-01 kernel: bonding: bond0: first active interface up!
Sep 19 11:34:28 lb-01 kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Sep 19 11:34:29 lb-01 Keepalived_vrrp: Netlink: filter function error
Sep 19 11:34:29 lb-01 Keepalived_healthcheckers: Netlink: filter function error
Sep 19 11:34:29 lb-01 Keepalived_vrrp: Netlink: filter function error
Sep 19 11:34:35 lb-01 Keepalived_healthcheckers: Netlink: filter function error
Sep 19 11:34:35 lb-01 Keepalived_vrrp: Netlink: filter function error
Sep 19 11:34:36 lb-01 Keepalived_vrrp: Kernel is reporting: interface bond0.18 UP
Sep 19 11:34:36 lb-01 Keepalived_vrrp: cant do IP_ADD_MEMBERSHIP errno=No such device (19)
Sep 19 11:34:36 lb-01 Keepalived_vrrp: cant bind to device bond0.18. errno=9. (try to run it as root)
Sep 19 11:34:36 lb-01 Keepalived_vrrp: VRRP_Instance(VI1) Transition to MASTER STATE
Sep 19 11:34:37 lb-01 Keepalived_vrrp: Netlink: error: Network is unreachable, type=(24), seq=1316071300, pid=0
Sep 19 11:34:37 lb-01 Keepalived_vrrp: Error sending gratutious ARP on bond0.5 for 172.31.13.226
Sep 19 11:34:37 lb-01 kernel: IPVS: sync thread started: state = MASTER, mcast_ifn = bond0.18, syncid = 103
Sep 19 11:34:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:53860
Sep 19 11:34:41 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:53860
Sep 19 11:34:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:53860
Sep 19 11:34:41 lb-01 last message repeated 48 times
Sep 19 11:34:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 11:34:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:53860
Sep 19 11:34:42 lb-01 Keepalived_vrrp: Error sending gratutious ARP on bond0.5 for 172.31.13.226
Sep 19 11:34:47 lb-01 sshd[20430]: Did not receive identification string from 10.103.4.40
Sep 19 11:35:05 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:35:05 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:35:05 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:35:05 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:37:47 lb-01 sshd[20595]: Did not receive identification string from 10.103.4.40
Sep 19 11:38:41 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:38:41 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:38:41 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:38:41 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:38:47 lb-01 sshd[20750]: Did not receive identification string from 10.103.4.40
Sep 19 11:39:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:42946
Sep 19 11:39:41 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:42946
Sep 19 11:39:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:42946
Sep 19 11:39:41 lb-01 last message repeated 48 times
Sep 19 11:39:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 11:39:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:42946
Sep 19 11:39:47 lb-01 sshd[20823]: Did not receive identification string from 10.103.4.40
Sep 19 11:39:48 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:39:48 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:39:48 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:39:48 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:40:33 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:40:33 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:40:33 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:40:33 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:40:43 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:40:43 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:40:43 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:40:43 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:40:47 lb-01 sshd[21220]: Did not receive identification string from 10.103.4.40
Sep 19 11:41:43 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:41:43 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:41:43 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:41:43 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:41:47 lb-01 sshd[21376]: Did not receive identification string from 10.103.4.40
Sep 19 11:42:27 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 11:42:27 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 11:42:27 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 11:42:27 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 11:42:47 lb-01 sshd[21533]: Did not receive identification string from 10.103.4.40
Sep 19 11:42:47 lb-01 sshd[21534]: Did not receive identification string from 10.103.4.40Netfilter Core Team
Sep 19 12:09:42 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:09:42 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:09:47 lb-01 sshd[23535]: Did not receive identification string from 10.103.4.40
Sep 19 12:09:47 lb-01 sshd[23536]: Did not receive identification string from 10.103.4.40
Sep 19 12:10:03 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 12:10:03 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 12:10:03 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:10:03 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:10:47 lb-01 sshd[23692]: Did not receive identification string from 10.103.4.40
Sep 19 12:12:06 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 12:12:06 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 12:12:06 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:12:06 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:12:34 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 12:12:34 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 12:12:34 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:12:34 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:12:47 lb-01 sshd[24006]: Did not receive identification string from 10.103.4.40
Sep 19 12:12:47 lb-01 sshd[24008]: Did not receive identification string from 10.103.4.40
Sep 19 12:13:05 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 12:13:05 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 12:13:05 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:13:05 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:13:47 lb-01 sshd[24164]: Did not receive identification string from 10.103.4.40
Sep 19 12:13:47 lb-01 sshd[24163]: Did not receive identification string from 10.103.4.40
Sep 19 12:14:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:38696
Sep 19 12:14:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:38696
Sep 19 12:14:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:38696
Sep 19 12:14:41 lb-01 last message repeated 48 times
Sep 19 12:14:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:14:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:38696
Sep 19 12:14:41 lb-01 last message repeated 20 times
Sep 19 12:18:47 lb-01 sshd[24180]: Did not receive identification string from 10.103.4.40
Sep 19 12:19:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:57723
Sep 19 12:19:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:57723
Sep 19 12:19:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:57723
Sep 19 12:19:41 lb-01 last message repeated 48 times
Sep 19 12:19:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:19:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:57723
Sep 19 12:19:41 lb-01 last message repeated 20 times
Sep 19 12:21:47 lb-01 sshd[24221]: Did not receive identification string from 10.103.4.40
Sep 19 12:22:08 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 12:22:08 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 12:22:08 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:22:08 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:22:47 lb-01 sshd[24400]: Did not receive identification string from 10.103.4.40
Sep 19 12:22:47 lb-01 sshd[24401]: Did not receive identification string from 10.103.4.40
Sep 19 12:24:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:56205
Sep 19 12:24:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:56205
Sep 19 12:24:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:56205
Sep 19 12:24:41 lb-01 last message repeated 48 times
Sep 19 12:24:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:24:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:56205
Sep 19 12:24:41 lb-01 last message repeated 20 times
Sep 19 12:26:47 lb-01 sshd[24423]: Did not receive identification string from 10.103.4.40
Sep 19 12:27:25 lb-01 sshd[24424]: Accepted password for root from 10.103.0.250 port 22088 ssh2
Sep 19 12:27:37 lb-01 Keepalived: Terminating on signal
Sep 19 12:27:37 lb-01 Keepalived: Stopping Keepalived v1.1.17 (03/16,2009)
Sep 19 12:27:37 lb-01 Keepalived_healthcheckers: Terminating Healthchecker child process on signal
Sep 19 12:27:37 lb-01 Keepalived_healthcheckers: IPVS: No such destination
Sep 19 12:27:37 lb-01 Keepalived_healthcheckers: IPVS: No such service
Sep 19 12:27:37 lb-01 Keepalived_healthcheckers: IPVS: No such service
Sep 19 12:27:37 lb-01 Keepalived_vrrp: Terminating VRRP child process on signal
Sep 19 12:27:37 lb-01 kernel: IPVS: stopping sync thread 20344 ...
Sep 19 12:27:37 lb-01 kernel: IPVS: sync thread stopped!
Sep 19 12:27:40 lb-01 Keepalived: Starting Keepalived v1.1.17 (03/16,2009)
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread...
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Registering Kernel netlink reflector
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Registering Kernel netlink command channel
Sep 19 12:27:40 lb-01 Keepalived: Starting Healthcheck child process, pid=24496
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Sep 19 12:27:40 lb-01 Keepalived: Starting VRRP child process, pid=24498
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Registering Kernel netlink reflector
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Registering Kernel netlink command channel
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Registering gratutious ARP shared channel
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Configuration is using : 107558 Bytes
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: ------< Global definitions >------
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Router ID = silverstreet
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Smtp server = 127.0.0.1
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Smtp server connection timeout = 30
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Email notification from = tech@xsnetworks.net
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: ------< SSL definitions >------
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Using autogen SSL context
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: ------< LVS Topology >------
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: System is compiled with LVS v1.2.1
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_1
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.8, VPORT = 80
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_3
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Configuration is using : 67394 Bytes
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.11, VPORT = 80
Sep 19 12:27:40 lb-01 Keepalived_vrrp: ------< Global definitions >------
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_4
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Router ID = silverstreet
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.11, VPORT = 443
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Smtp server = 127.0.0.1
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_5
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Smtp server connection timeout = 30
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.12, VPORT = 80
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Email notification from = tech@xsnetworks.net
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_6
Sep 19 12:27:40 lb-01 Keepalived_vrrp: ------< VRRP Topology >------
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.12, VPORT = 443
Sep 19 12:27:40 lb-01 Keepalived_vrrp: VRRP Instance = VI1
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_7
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Want State = MASTER
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.8, VPORT = 2675
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Runing on device = bond0.18
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_8
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Runing LVS sync daemon on interface = bond0.18
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: VIP = 195.88.18.8, VPORT = 443
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Virtual Router ID = 103
Sep 19 12:27:40 lb-01 Keepalived_healthcheckers: Virtual Server Group = VSG_9
Sep 19 12:27:40 lb-01 Keepalived_vrrp: Priority = 100
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: digest = c5def62ca69e4d4e5c0d6798693600b8
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.3:80
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.3:80
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = HTTP_GET
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Nb get retry = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Delay before retry = 2
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Checked url = /test.html
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: digest = c5def62ca69e4d4e5c0d6798693600b8
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.7.1:80
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.7.1:443
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.6.1:80
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.6.1:443
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.1:2677
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.2:2677
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.3:2677
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.1:443
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.2:443
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.3:443
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.1:8080
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.2:8080
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.3:8080
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.4.50:80
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.4.50:3306
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.1:8880
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.2:8880
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.3:8880
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.1:2677
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Keepalive method = TCP_CHECK
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Connection timeout = 3
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: 10.103.1.2:2677
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Activating healtchecker for service [10.103.1.2:2677]
Sep 19 12:29:48 lb-01 Keepalived_healthcheckers: Activating healtchecker for service [10.103.8.1:22]
Sep 19 12:29:49 lb-01 Keepalived_vrrp: VRRP_Instance(VI1) Transition to MASTER STATE
Sep 19 12:29:49 lb-01 Keepalived_healthcheckers: MD5 digest error to [10.103.1.1:80] url[/test.html], MD5SUM [90cc08199ea74cbdd45182c25adde59
9].
Sep 19 12:29:49 lb-01 Keepalived_healthcheckers: Removing service [10.103.1.1:80] from VS [VSG_1:0]
Sep 19 12:29:49 lb-01 Keepalived_healthcheckers: TCP connection to [10.103.1.2:2677] failed !!!
Sep 19 12:29:49 lb-01 Keepalived_healthcheckers: Removing service [10.103.1.2:2677] from VS [VSG_7:0]
Sep 19 12:29:50 lb-01 Keepalived_healthcheckers: TCP connection to [10.103.1.3:2677] failed !!!
Sep 19 12:30:03 lb-01 Keepalived: Starting Keepalived v1.1.17 (03/16,2009)
Sep 19 12:30:03 lb-01 Keepalived: daemon is already running
Sep 19 12:30:13 lb-01 Keepalived: Starting Keepalived v1.1.17 (03/16,2009)
Sep 19 12:30:13 lb-01 Keepalived: daemon is already running
Sep 19 12:30:47 lb-01 sshd[24586]: Did not receive identification string from 10.103.4.40
Sep 19 12:30:47 lb-01 sshd[24584]: Did not receive identification string from 10.103.4.40
Sep 19 12:31:46 lb-01 sshd[24596]: Did not receive identification string from 10.103.4.40
Sep 19 12:31:47 lb-01 sshd[24597]: Did not receive identification string from 10.103.4.40
Sep 19 12:31:47 lb-01 kernel: Removing netfilter NETLINK layer.
Sep 19 12:31:47 lb-01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 19 12:31:47 lb-01 kernel: Netfilter messages via NETLINK v0.30.
Sep 19 12:31:47 lb-01 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
Sep 19 12:32:46 lb-01 sshd[24758]: Did not receive identification string from 10.103.4.40
Sep 19 12:32:46 lb-01 sshd[24760]: Did not receive identification string from 10.103.4.40
Sep 19 12:33:46 lb-01 sshd[24765]: Did not receive identification string from 10.103.4.40
Sep 19 12:33:46 lb-01 sshd[24764]: Did not receive identification string from 10.103.4.40
Sep 19 12:34:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:59987
Sep 19 12:34:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:59987
Sep 19 12:34:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:59987
Sep 19 12:34:41 lb-01 last message repeated 48 times
Sep 19 12:34:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:34:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:59987
Sep 19 12:34:41 lb-01 last message repeated 20 times
Sep 19 12:38:46 lb-01 sshd[24781]: Did not receive identification string from 10.103.4.40
Sep 19 12:39:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:40777
Sep 19 12:39:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:40777
Sep 19 12:39:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:40777
Sep 19 12:39:41 lb-01 last message repeated 48 times
Sep 19 12:39:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:39:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:40777
Sep 19 12:39:41 lb-01 last message repeated 20 times
Sep 19 12:39:46 lb-01 sshd[24782]: Did not receive identification string from 10.103.4.40
Sep 19 12:39:46 lb-01 sshd[24784]: Did not receive identification string from 10.103.4.40
Sep 19 12:44:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:48412
Sep 19 12:44:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:48412
Sep 19 12:44:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:48412
Sep 19 12:44:41 lb-01 last message repeated 48 times
Sep 19 12:44:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:44:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:48412
Sep 19 12:44:41 lb-01 last message repeated 20 times
Sep 19 12:48:46 lb-01 sshd[24812]: Did not receive identification string from 10.103.4.40
Sep 19 12:49:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:35543
Sep 19 12:49:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:35543
Sep 19 12:49:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:35543
Sep 19 12:49:41 lb-01 last message repeated 48 times
Sep 19 12:49:41 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:49:41 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:35543
Sep 19 12:49:41 lb-01 last message repeated 20 times
Sep 19 12:53:46 lb-01 sshd[24827]: Did not receive identification string from 10.103.4.40
Sep 19 12:54:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:47548
Sep 19 12:54:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:47548
Sep 19 12:54:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:47548
Sep 19 12:54:40 lb-01 last message repeated 48 times
Sep 19 12:54:40 lb-01 snmpd[19821]: truncating integer value > 32 bits
Sep 19 12:54:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:47548
Sep 19 12:54:41 lb-01 last message repeated 20 times
Sep 19 12:56:46 lb-01 sshd[24838]: Did not receive identification string from 10.103.4.40
Sep 19 10:56:46 lb-01 sshd[24835]: Connection closed by 10.103.0.250
Sep 19 12:59:40 lb-01 snmpd[19821]: Connection from UDP: [10.103.4.40]:57482
Sep 19 12:59:40 lb-01 snmpd[19821]: Received SNMP packet(s) from UDP: [10.103.4.40]:57482
This could be an issue with the keepalived. Is it wose to shut the keepalived on LB-01?
how to overcome this?
Thanks |
