iptables rules with a VIP

I am looking at how I can implement an iptables rule to tackle the following

App ServerX --> VIP --> database

the VIP is a loadbalancer that round robins to 3 servers that have the database.

I want only AppServerX to be able to access the database.

I've added firewall rules iptables to all 3 servers running the database, to only allow traffic from server.

this works when the VIP is turned off. As soon as the VIP is enabled, any host is able to access the database.

How can I restrict this so that only App ServerX has access to the database. removing the VIP is not an option

What do your rules look like now?

well its currently empty. But here's one version we tried

target prot opt source destination
DROP tcp -- !abc.xx.xx.net anywhere tcp dpt:irtrans
DROP tcp -- !abc.xx.xx.net anywhere tcp dpt:21050


so we only want server abc to be able to access port 21050 and 21000 (irtrans)

however the 3 servers are round-robin to a vip. Soon as this is turned on . the rules are ignored and anyone can connect.

What version of RH/Centos is this?
Using firewalld not iptables?