You've got an asymmetric routing situation (i.e. packets come in one way and go out another). I would suggest checking the reverse path filtering (rp_filter) setting.
To check the rp_filter setting:
Code:
sysctl -a | grep -e '\.rp_filter'
If it's set to 1, then try setting it to 2:
Code:
echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
When rp_filter is set to 1, the system does strict reverse path filtering, which doesn't work with asymmetric routing. In a simple configuration (without a bunch of static routes), it basically means that the IP address of incoming packets is compared to the IP address(es) on the NIC it comes in on and if it doesn't match, the packet is discarded. Since there can only be one default route (without policy based routing), all packets from outside the subnet will come in through one NIC (the one with the default route), so anything destined to the other IP addresses will be discarded.
Setting the rp_filter value to 2 allows "loose" reverse path filtering, which basically means that if the packet is destined for any of the IP addresses on the system, it will be allowed through. So, even though the packet still comes in through the NIC with default route, as long as it is destined for one of the NICs on the system, the packet will get through.
To make the change permanent, you need to add the settings to /etc/sysctl.conf.
