At this point I'm thinking about abandoning my original plan of dedicated NICs for database traffic and everything else and thinking about create a bonded interface. At least that way i'll get load balancing and failover support. But what still bugs me is why does two separate IP addresses, assigned to two different NICs would resolve to the same MAC address(eth0) ? I just need to know the reason why the current setup is not working and i'll be happy.

Try this and see if it works.

http://serverfault.com/questions/580...-interfaces-on

Basically the interface is responding to the ARP request since the server owns the MAC. Try using sysctl to tell it to reply only if that interface has the MAC.

I added the entry "net.ipv4.conf.all.arp_ignore = 1" to my sysctl.conf and ran sysctl -p. Then tried running arping from another box to both IPs. It worked for eth0 and I got unicast replies. But for eth1 arping couldn't reach the host at all. I tried running ping -I eth1 from the target box to my desktop and it failed too. When I set the value of arp_ignore to 0 again, ping from the target box to my desktop via both eth0 and eth1 started working and from the old behavior of returning eth0's MAC for both IPs returned. Then I tried "ifconfig eth0 down" and then when I brought it back up again, arping started showing eth1's MAC for both IPs.