LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
Reload this Page Keepalived backup keeps becoming master
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 10-28-2015, 12:14 PM   #1
thaylin
LQ Newbie
 
Registered: Oct 2015
Posts: 22

Rep: Reputation: Disabled
Keepalived backup keeps becoming master

I am using a rhel 7 machine with 2 nics, one internal and one external.

For some reason even though the machines are receiving the vrrp packets, seemingly properly, the backup is not recognizing the master is back online.
Here is my backup's relevant conf code
Code:
global_defs {
   notification_email {
     xxx@xxx
   }
   notification_email_from xxx@xxx
   smtp_server localhost
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}


vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_2 {
    state BACKUP
    interface eno33559296
    virtual_router_id 52
    priority 100
    smpt_alert
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass xxxxxxxxxxxxx
    }
    virtual_ipaddress {
        10.76.10.100/24 brd 10.76.10.255
    }
    track_script {
        chk_haproxy
    }

}
static_routes{

192.168.136.0/24 dev eno16780032 tab 1
10.76.10.0/24 dev eno33559296 tab 2
default via 192.168.136.1 dev eno16780032 tab 1
add default via 10.76.10.1 dev eno33559296 tab 2
add from 192.168.136.184/32 tab 1 priority 100
add from 10.76.10.100/32 tab 2 priority 300
}

virtual_server 10.76.10.100 3306 {
    delay_loop 6
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    real_server 10.76.10.101 3306 {
        weight 1
}
    real_server 10.76.10.102 3306 {
        weight 1
}
    real_server 10.76.10.103 3306 {
        weight 1
}
Here is the tcpdump showing that the machines seem to be seeing each other with the alternating priorities.

Code:
13:03:42.791859 IP mysqlclmgmt > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 152, authtype simple, intvl 1s, length 20
13:03:42.858678 IP mysqlclmgmt > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 102, authtype simple, intvl 1s, length 20
13:03:43.792423 IP mysqlclmgmt > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 152, authtype simple, intvl 1s, length 20
13:03:43.859558 IP mysqlclmgmt > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 102, authtype simple, intvl 1s, length 20
13:03:44.792915 IP mysqlclmgmt > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 152, authtype simple, intvl 1s, length 20
13:03:44.860401 IP mysqlclmgmt > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 102, authtype simple, intvl 1s, length 20

And here is the log output

Code:
Oct 28 12:59:10 lvs2 systemd: Starting LVS and VRRP High Availability Monitor...
Oct 28 12:59:10 lvs2 Keepalived[17087]: Starting Keepalived v1.2.13 (11/18,2014)
Oct 28 12:59:10 lvs2 systemd: Started LVS and VRRP High Availability Monitor.
Oct 28 12:59:10 lvs2 Keepalived[17088]: Starting Healthcheck child process, pid=17089
Oct 28 12:59:10 lvs2 Keepalived[17088]: Starting VRRP child process, pid=17090
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Netlink reflector reports IP 192.168.136.184 added
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Netlink reflector reports IP fe80::250:56ff:fea8:77d added
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Netlink reflector reports IP fe80::250:56ff:fea8:539a added
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Registering Kernel netlink reflector
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Registering Kernel netlink command channel
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Registering gratuitous ARP shared channel
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP 192.68.136.184 added
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP fe80::250:56ff:fea8:77d added
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP fe80::250:56ff:fea8:539a added
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Registering Kernel netlink reflector
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Registering Kernel netlink command channel
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Configuration is using : 65505 Bytes
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: ------< Global definitions >------
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Router ID = LVS_DEVEL
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Smtp server = 0.0.0.0
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Smtp server connection timeout = 30
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Email notification from = 
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Email notification = 
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP IPv4 mcast group = 224.0.0.18
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP IPv6 mcast group = 224.0.0.18
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: SNMP Trap disabled
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Configuration is using : 13316 Bytes
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: ------< VRRP Topology >------
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP Instance = VI_2
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Want State = BACKUP
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Runing on device = eno33559296
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Virtual Router ID = 52
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Priority = 100
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Advert interval = 1sec
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Authentication type = SIMPLE_PASSWORD
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Password = 1111
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Tracked scripts = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: chk_haproxy weight 2
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Virtual IP = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: 10.76.10.100/24 brd 10.76.10.255 dev eno33559296 scope global
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: ------< VRRP Scripts >------
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP Script = chk_haproxy
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Command = killall -0 haproxy
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Interval = 2 sec
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Timeout = 0 sec
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Weight = 2
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Rise = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Fall = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Status = INIT
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Using LinkWatch kernel netlink reflector...
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) Entering BACKUP STATE
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: Scheduler or persistence engine not found
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: No such file or directory
Oct 28 12:59:10 lvs2 systemd: Starting LVS and VRRP High Availability Monitor...
Oct 28 12:59:10 lvs2 Keepalived[17087]: Starting Keepalived v1.2.13 (11/18,2014)
Oct 28 12:59:10 lvs2 systemd: Started LVS and VRRP High Availability Monitor.
Oct 28 12:59:10 lvs2 Keepalived[17088]: Starting Healthcheck child process, pid=17089
Oct 28 12:59:10 lvs2 Keepalived[17088]: Starting VRRP child process, pid=17090
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Netlink reflector reports IP 192.68.136.184 added
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Netlink reflector reports IP fe80::250:56ff:fea8:77d added
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Netlink reflector reports IP fe80::250:56ff:fea8:539a added
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Registering Kernel netlink reflector
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Registering Kernel netlink command channel
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Registering gratuitous ARP shared channel
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP 192.68.136.184 added
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP fe80::250:56ff:fea8:77d added
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP fe80::250:56ff:fea8:539a added
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Registering Kernel netlink reflector
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Registering Kernel netlink command channel
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Configuration is using : 65505 Bytes
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: ------< Global definitions >------
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Router ID = LVS_DEVEL
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Smtp server = 0.0.0.0
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Smtp server connection timeout = 30
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Email notification from = 
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Email notification = 
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP IPv4 mcast group = 224.0.0.18
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP IPv6 mcast group = 224.0.0.18
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: SNMP Trap disabled
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Configuration is using : 13316 Bytes
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: ------< VRRP Topology >------
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP Instance = VI_2
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Want State = BACKUP
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Runing on device = eno33559296
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Virtual Router ID = 52
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Priority = 100
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Advert interval = 1sec
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Authentication type = SIMPLE_PASSWORD
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Password = 1111
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Tracked scripts = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: chk_haproxy weight 2
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Virtual IP = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: 10.76.10.100/24 brd 10.76.10.255 dev eno33559296 scope global
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: ------< VRRP Scripts >------
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP Script = chk_haproxy
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Command = killall -0 haproxy
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Interval = 2 sec
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Timeout = 0 sec
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Weight = 2
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Rise = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Fall = 1
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Status = INIT
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: Using LinkWatch kernel netlink reflector...
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) Entering BACKUP STATE
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: Scheduler or persistence engine not found
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: No such file or directory
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: No such file or directory
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: No such file or directory
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: ------< Global definitions >------
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Router ID = LVS_DEVEL
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Smtp server = 0.0.0.0
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Smtp server connection timeout = 30
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Email notification from = 
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Email notification = 
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: VRRP IPv4 mcast group = 224.0.0.18
Oct 28 12:59:10 lvs2 kernel: IPVS: Scheduler module ip_vs_ not found
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: VRRP IPv6 mcast group = 224.0.0.18
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: SNMP Trap disabled
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: ------< SSL definitions >------
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Using autogen SSL context
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: ------< LVS Topology >------
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: System is compiled with LVS v1.2.1
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: VIP = 10.76.10.100, VPORT = 3306
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: delay_loop = 6, lb_algo =
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: persistence timeout = 50
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: protocol = TCP
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: alpha is OFF, omega is OFF
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: quorum = 1, hysteresis = 0
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: lb_kind = NAT
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: RIP = 10.76.10.101, RPORT = 3306, WEIGHT = 1
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: RIP = 10.76.10.102, RPORT = 3306, WEIGHT = 1
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: RIP = 10.76.10.103, RPORT = 3306, WEIGHT = 1
Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: Using LinkWatch kernel netlink reflector...
Oct 28 12:59:10 lvs2 Keepalived_vrrp[17090]: VRRP_Script(chk_haproxy) succeeded
Oct 28 12:59:11 lvs2 ntpd[2852]: Deleting interface #69 eno33559296, 10.76.10.100#123, interface stats: received=0, sent=0, dropped=0, active_time=391 secs
Oct 28 12:59:13 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) Transition to MASTER STATE
Oct 28 12:59:14 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) Entering MASTER STATE
Oct 28 12:59:14 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) setting protocol VIPs.
Oct 28 12:59:14 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eno33559296 for 10.76.10.100
Oct 28 12:59:14 lvs2 avahi-daemon[696]: Joining mDNS multicast group on interface eno33559296.IPv4 with address 10.76.10.100.
Oct 28 12:59:14 lvs2 avahi-daemon[696]: New relevant interface eno33559296.IPv4 for mDNS.
Oct 28 12:59:14 lvs2 avahi-daemon[696]: Registering new address record for 10.76.10.100 on eno33559296.IPv4.
Oct 28 12:59:14 lvs2 Keepalived_healthcheckers[17089]: Netlink reflector reports IP 10.76.10.100 added
Oct 28 12:59:15 lvs2 ntpd[2852]: Listen normally on 70 eno33559296 10.76.10.100 UDP 123
Oct 28 12:59:19 lvs2 Keepalived_vrrp[17090]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eno33559296 for 10.76.10.100

Any help would be greatly appreciated.
 
Old 10-29-2015, 02:42 PM   #2
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,853

Rep: Reputation: 161Reputation: 161
Health checking, Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: Scheduler or persistence engine not found, is failure. You should upgrade to 7.1.
Check following bug fix.
https://bugzilla.redhat.com/show_bug.cgi?id=1261389
 
Old 10-29-2015, 03:54 PM   #3
thaylin
LQ Newbie
 
Registered: Oct 2015
Posts: 22

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by nini09 View Post
Health checking, Oct 28 12:59:10 lvs2 Keepalived_healthcheckers[17089]: IPVS: Scheduler or persistence engine not found, is failure. You should upgrade to 7.1.
Check following bug fix.
https://bugzilla.redhat.com/show_bug.cgi?id=1261389
Thank you for responding. I am already on 7.1, I figured out that problem, it was that the secondary nic did not have an IP address already and so it seemed to fail. With another IP assigned it works, however then mysql connections fail with:

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 "Internal error/check (Not system error)"
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Keepalived + ipvsadm showing connections for both master and backup VRRP campi01 Linux - Networking 3 11-11-2013 05:16 AM
LXer: Setting Up Master-Master Replication On Four Nodes With MySQL 5 On Debian Etch LXer Syndicated Linux News 0 11-20-2008 07:30 PM
master hared disk s.m.a.r.t backup and replace perathprabha Linux - Newbie 1 03-13-2008 10:40 PM
whats a good program to backup my master boot record FocusedWolf Linux - General 11 03-15-2005 09:55 PM
Backup/Restore Master Boot Record marktin Linux - General 8 04-05-2004 02:35 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 02:54 AM.

Contact Us - Advertising Info - Rules - Privacy - LQ Merchandise - Donations - Contributing Member - LQ Sitemap -
Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration