Hi Folks,

I've gotten myself into a bit of a pickle and I don't know if there's a way out or not. Here's the setup:

I've mostly switched to my new Fiber-to-the-Home connection on which I have a little 8-IP subnet set up. These are public IPs and all the 5 usable IPs are functional. It all works really nicely (and wonderfully rapidly).

But, I still have my old, reliable, but slower DSL modem. I'd sure like it if, during a hopefully short switch-over period, people could hit port 80 on the Linux server from either the new fiber external IP address or the old DSL address. For reasons which are historical and kinda dumb, some people hit the web server via an IP address, and it'll take a while to get everyone switched over to the new one.

I've got 3 NICs in the Linux box. eth0 & eth1 have fixed, public addresses and are hooked to the Internet through a Cisco PIX firewall (do NOT buy one of these unless you have a serious investment in Bayer stock)--but it's working now.... The firewall does forwarding nicely for the Linux box's public address which are in the public subnet range (2x.x.x.130). So, is there any way to hook that other NIC I just put into the Linux box today to the DSL modem (which does NAT so the NIC has a 10.0.0.0-type address) and have it continue to work?

Here's the nub of the matter: packets seem to arrive via the DSL modem just fine, but is there a way to have the Linux box send the return traffic out the same NIC it came in from? (That is, the outbound routing somehow has to be related to the NIC it arrived via, rather than the source it's intended for--which will always be a public IP address and won't fall into any given range or subnet.)

I don't have anything special set up in the routing tables at this point, and although I find sections of Linux Advanced Routing & Traffic Control HOWTO (http://lartc.org/howto/index.html) that seem to address vaguely similar situations, nothing is quite the same. If I know it's possible, I don't mind digging... But I can't see how it can work at this point (seems like outbound traffic's going to get routed based on the destination, not where it came in from). Am I missing something in the section on separate routing tables for the separate devices? General guidance most gratefully accepted; specific directions if you feel so inclined would be just super.

Some details:

Fedora 3 Linux host with 3 NICs
(two are on the 20x.x.x.128/29 subnet and they work fine)
eth0
fixed IP visible to the Internet (ip 20x.x.x.130).
The firewall allows traffic to permit serving the web site and JNLP uploads on its port 80. This guy's IP's got a DNS listing to my main site and it all works just peachily.

eth1
fixed IP visible to the Internet (ip 20x.x.x.134). This IP has its own DNS entry and will someday host its own site; for now it's functionally identical to eth0.

eth2
Local IP address (10.0.0.6) using the old DSL modem's NAT capability to be reachable from the internet only when someone hits the DSL modem's external IP address (166.x.x.159) and port 80 (this is how the web site was hosted until the upgraded gear was installed and the DNS name/IP associations got changed). It doesn't work--certainly because I don't have things set up right; perhaps because what I'm trying to do is misguided or worse... :-`

If I try to hit the old 166.x.x.159 address from outside via a web browser, it just times out (no surprise--the default route on the Linux webserver is associated with the eth0 device so the answer--if any, is coming out of a different NIC and the browser wouldn't, presumably, get it). Here's what I see of the three SYN packets the browser sends when I run ethereal on the NIC hooked to the DSL modem (10.0.0.6)--I've omitted the Frame and EthernetII levels and generified the IP addresses:

Internet Protocol, Src: 2x.8x.x.118 ( 2x.8x.x.118 ), Dst: 10.0.0.6 (10.0.0.6)
Transmission Control Protocol, Src Port: 4855 (4855), Dst Port: 80 (80), Seq: 0, Ack: 0, Len: 0

So, the packets are arriving, but how to get them to go back out the 10.0.0.6 NIC? There doesn't seem to me to be any way to set up the routing to say, "If it came in via 10.0.0.6, send the reply back out that same NIC"--am I right? That is, the IP stack is going to think it terms of the incoming source, rather than the incoming destination of the packet to determine outbound routing, isn't it? Or is there a level of complexity there that would allow this to work?

(I haven't checked to see if any ACK is going out the default NIC--I decided to ask first.)

I saw the sections on separate routing tables for each device, but that seems to be related to situations where you're bridging between networks (send all traffice for 192.168.1.x out eth0; all traffic to 192.168.2.x out eth1)--which isn't the situation I'm dealing with here....

Do I need another box to handle the bridging somehow?

Any info would be a help.

Thanks,

John

Could it be as easy as adding a route to the 10.0.0.0 net through the eth2 interface?

If I am right, man route will have the details.

When your webserver replies to packets that came in from 10.0.0.*, the replies will be addressed to 10.0.0.* (won't they?). So all you must do is route them to the correct interface.

Or am I missing something here, this seems too easy.

Hi,

Thanks for replying, but I don't believe it's quite that simple. The route it seems like you're suggesting is already in place:

[root@hostbox ~]# ip route
2x.x.x.128/29 dev eth0 proto kernel scope link src 2x.x.x.130
2x.x.x.128/29 dev eth1 proto kernel scope link src 2x.x.x.134
10.0.0.0/24 dev eth2 proto kernel scope link src 10.0.0.6
default via 204.228.148.129 dev eth0

Routing replies to packets which identify their source as part of the 10.0.0.0 subnet is no problem; that works just by virtue of configuring the NIC. If the packets' IP source is the 10.0.0.0 subnet, there's no issue with them--replies to such packets already go back out the NIC in question (a SYN packet from the DSL modem itself results in an ACK being sent out the NIC that's on the 10.0.0.0 subnet--all too easy). Here's the rub: the source IP of packets from the Internet received via the eth2 NIC isn't the 10.0.0.0 subnet. That is to say, the DSL modem doesn't rewrite the source IP as the packet passes through it to the eth2 NIC to indicate that the packet came via the private net--the source address is unchanged (see the example of the ethereal captured SYN packet in my original post). (And maybe there's a way to configure the DSL modem to essentially do NAT on the outbound port too, but I'm not aware of it--in case anyone's a DSL or Cisco wizard, it's a Cisco 678.)

So, to sum up, the problem I'm wrestling with is this: the source IP of anything interesting is the public net, not the private subnet. The problem is determining whether it's possible to configure Linx to send something back out the NIC it came in through--when the reply packet's target address isn't distinguishable from the target addresses of traffic that should go via the default route. It doesn't seem like there's any way to use "routing" to solve this problem--but if there is, I'd like to know it....

Again, thanks for taking the time to reply.

John

Indeed, I was assuming that you already had NAT in place.

I did a quick Google on "Cisco 678" (w/ the quotes), and got the following:

• Quick Start for the Cisco 678 ADSL Router
• Release Notes for the Cisco 678 ADSL Router
• Cisco 678 Overview

The last one says, in part: Sounds like you have the feature you need, maybe the other 2 refs. will help you get it working.

Or did you have these already?

Thanks, I have the docs on the 678. Since I've been using it for years and doing PAT with it essentially from day-1, I didn't actually figure there was anything I could do with it, but I thought I'd mention it.

If it was to work, in any obvious way, in the arrangement that I've got, it would need to do translation of the network address as well as the port. That is, it would need to take a packet from the Internet (it's WAN port), and reroute it to the private (LAN) port after changing the packet's source IP address to be in the 10.0.0.0 subnet. It does not do this.

It appears to me that what I'd need is a separate box that could serve a bridging function between the packets sent to the Cisco 678's LAN port and the NIC on the web server--that way, packets that arrived via the 10.0.0.0 subnet would identify that as their return address and the Cisco wouldn't have to try to do the whole job. There is a bridge mode on the Cisco, but I've never used it. Maybe that's what I need--so mentioning the docs may help after all!

John

Nope, misinterpreted what was meant by "bridging mode":

Definitely not what the doctor ordered.... Oh, well, the transition period is essentially over now anyway.....

John

I should have known you already have the docs.

BTW, how does it do the promised connection sharing, if not by NAT?

Well, there's NAT and then there's NAT. What it does is strictly called PAT (Port Address Translation) these days (it's still called NAT in the docs for the 678 although it does say that it uses port translation in the explanatory notes). That is to say, suppose you go to hit a web site like www.linuxquestions.org from a PC on the LAN of the 678, the exchange from PC to the web server and back goes like this (I apologize if this is too basic):

PC (LAN IP address 10.0.0.6) sends a SYN packet via its NIC to the 10.0.0.0 subnet via its default gateway (10.0.0.1--the 678's LAN port's IP address) to the IP of www.linuxquestions.org:

Source IP:
10.0.0.6
Source Port:
80
Destination IP:
64.179.4.149
Destination Port:
80

At the DSL modem (Cisco 678), this gets picked up on the LAN port (10.0.0.1--the PC's default gateway) and retransmitted via the WAN port (166.79.29.156)--but it changes the source port as it goes out:

Soucre IP:
166.79.29.156
Source Port:
10976 -- this port is internally associated with 10.0.0.6/80
Destination IP:
64.179.4.149
Destination Port:
80

www.linuxquestions.com's web server receives the packet and sends an ACK back:
Source IP:
64.179.4.149
Source Port:
80
Destination IP:
166.79.29.156
Destination Port:
10976

That packet hits the WAN port of the Cisco 678 (166.76.29.156) and is retransmitted by the LAN port onto the 10.0.0.0 subnet as follows (because it finds in its NAT table that port 10976 is associated with 10.0.0.6; port 80 [strictly speaking, that's a PAT situation even though it calls it "NAT" in the docs--it does not fully translate the IP address]):

Source IP:
64.179.4.149
Source Port:
80
Destination IP:
10.0.0.6
Destination Port:
80

That packet hits the NIC on the PC and the exchange has done the full circuit. But, in order for this to do what I need it to do in my setup, the Cisco 678 would need to not only adjust the port (which it uses to route the return packet to port 10976 to the right LAN IP and port), but it would also need to translate the IP address itself. That is, it'd have to do this to the ACK packet coming back from the web server:

Source IP:
10.0.0.1 -- note that this is the LAN, not the Internet address
Source Port:
80
Destination IP:
10.0.0.6
Destination Port:
80

If it did that, then the routing table entry on the Linux box for eth2 that associates 10.0.0.0 with that NIC could return the subsequent packet via the right NIC (rather than the default eth0 which is on the Internet itself). But, in the first place, the 678 doesn't do that. In the second, it wouldn't be the whole story if it did: the web browser is expecting a reply from www.linuxquestions.org (64.179.4.149), not from the IP of the DSL modem's LAN port (10.0.0.1)--so the translation would have to be done in a more complex fashion anyway.

Now, my situation is also turned around: I have the web server receiving packets from the public net via two independent networks. But the source IP could be exactly the same. For instance, suppose I hit my web server with two Firefox sessions: one via the 204.x.x.x address and one via the old 166.x.x.x address. The server's IP stack can't tell the difference once the packet is inside the IP stack because the source address is the same; there's no way to use routing (that I can see) to reply to the same exact IP address via two different NICs based on what NIC the packets arrived by--the IP stack no longer cares what the MAC address of the packets was; that's not important at the IP layer. Right?

As I see it, there's not a way to do what I need to do--unless there's a magic way to do it I don't know about. I can't see any way to tell the IP stack: Hey, remember this packet that arrived via the 10.0.0.0 NIC, okay? And when it's time to reply, make sure it goes back out via the NIC the originating packet came in by--even though there's no way to distinguish the target IP address (e.g. 64.179.4.149 in the example above) from that of other packets that should go out via the default port on the other NIC.

John

This would be a bit of a resource hog I imagine, but couldn't it be done by running an additional ( or few additional servers) bound to the eth2 interface and unbind the main server from eth2, then just set the root directory for your additional servers to the same as for the original server then these new servers will serve only things that come in on eth2 and send them out via eth2? This would entail some additional resources temorarily, but would solve the issue until the entire switch was finished?

Alex

(PS I think you can bind to an interface with apache, but I am not entirely sure)