I have a question about asymmetric routing - I've been searching the web and reading docs for the past couple of days, but still haven't found satisfactory answers.

We have a 100 megabit load balancer. One of my servers is connected to it via a gigabit ethernet NIC (eth0). Unfortunately, we really need faster gigabit speeds, since the load balancer is limiting us to 100 megabits/second. We still need the device plugged into the load balancer because there are other machines coming online that we will need to load balance between.

However, for certain network operations we can live without the redundancy of a load balancer in order to garner higher speeds. So we configured a second gigabit ethernet NIC (eth1) to be plugged into a layer 2 gigabit switch. It has a different IP than eth0 and is on a different subnet. By doing this, all traffic going through eth1 will uses gigabit speeds.

When we activate eth1, suddenly we cannot hit the URL configured on the load balancer. It hangs in the browser. One of our support folks said the https connection is failing because it's routing traffic asymmetrically - traffic coming into eth0 via the load balancer is going out through eth1 via the layer 2 switch.

But what's so bad about this? More importantly, why does this make the URL configured on the load balancer inaccessible?

I've read elsewhere on the internet that asymmetrical routing can be implemented. In fact, I've modified Linux's routing table to add 2 host entries to our most network-intensive servers, so that the gigabit eth1 NIC is ONLY communicating with these 2 servers and no other servers. Now everything is working again, but I don't quite understand how it was failing in the first place.

Thanks for your responses in advance.

The kernel picks the shortest route when sending a response to an incoming packet. What you did caused the following:
This doesn't work because the original request from (source address, call it 1.2.3.4) went to the IP of eth0 (well, the VIP on the load balancer actually), call it 5.6.7.8. However the box receiving that packet decided that eth1 was a shorter route back to 1.2.3.4, so it composes a response packet out eth1 and gives the response a source address of 6.7.8.9 (the IP of eth1). When the response arrives at 1.2.3.4 that host looks at the packet and sees that it came from 6.7.8.9, but 1.2.3.4 never made a request to 6.7.8.9, so it throws away the packet and waits for a response from 5.6.7.8 (which never comes).

It is possible to make asymmetric routing work, but you have to perform NAT or some other trickery to make sure the source of the return packets gets rewritten as the destination of the original request, rather than the IP of the interface that sends the response.

The situation you describe is usually called "triangular routing" because it takes the form: A sends to B, B sends packet through C, A gets response from C.