Ok, here's the scenario.

Private LAN using 172.16.x.x addressing behind a linux (FC1) firewall doing SNAT. The lan consists of 3 buildings, each of which comes into the firewall on a different physical interface, so we have 172.16.0.1, 172.16.4.1 and 172.16.6.1 as the addresses of the 3 interfaces on the internal side of the firewall (eth1-3).

External interface on the firewall (eth0) has 4 IP addresses. One primary address represents the firewall itself, and does not handle outgoing traffic from the lan. The 3 IP aliases, eth0:0-2, are assigned using the ifcfg-eth0-range format in /etc/sysconfig/network-scripts, and are assigned to each of the three buildings respectively via SNAT, thus:

iptables -t nat -A POSTROUTING -s $SH_LAN -o $DMZ_IF -j SNAT --to-source $SH_EXT
iptables -t nat -A POSTROUTING -s $TH_LAN -o $DMZ_IF -j SNAT --to-source $TH_EXT
iptables -t nat -A POSTROUTING -s $MS_LAN -o $DMZ_IF -j SNAT --to-source $MS_EXT

Where $xx_LAN specifies the private IP range for a specific building, and $xx_EXT specifies one of the 3 IP aliases.

All this is working fine up to the point where packets leave the firewall on their happy way to the internet. When SNAT'd packets leave eth0, they never come back. The problem seems to be related to arp. I've tested with a cisco 2600 router. Packets leave the firewall and go to the router. That's the last I see of them on the firewall. If I then check the arp table on the cisco, it lists the source IP that SNAT is assigning, but has no MAC address associated with it, instead saying "(incomplete)" in the MAC address column of the arp table. No matter what settings I tweak on the linux box, this remains true. If, however, I manually add an arp entry on the cisco, including the MAC address for eth0 for each of the 3 IP aliases I'm using for SNAT, everything works beautifully.

The problem is, I won't be using the cisco as a gateway once this goes "live," I'll be using a content filtering appliance, and I do not have shell access or means to manipulate the arp table on the appliance. I've tested with it already, and it behaves as expected; traffic leaves the firewall, goes to the appliance, and never returns to the firewall.

I've been all over Google and IRC and some other forums with no enlightenment so far. I'm stumped.

Thanks for taking the time to read this.

B-man

Are *_EXT on the same subnet as the cisco and eth0?

Or does the cisco have to route the different LANs to the "real" eth0?

What does the arp traffic look like, does the cisco ever ask for any of the virtual IP's?

I'm pretty clueless when i comes to IOS, but this hopefully a generic networking problem.

Good Luck,
chris

*_EXT, eth0 and the cisco all have addresses in the same subnet, so no routing is necessary between these hosts.

I don't see any arp requests coming from the cisco, or from the web filtering appliance (which is basically an x86 running some version of redhat), so I suspect it is a "generic" networking problem.

If, however, I manually add an arp entry on the cisco, including the MAC address for eth0 for each of the 3 IP aliases I'm using for SNAT, everything works beautifully.
Well it definately sounds like the cisco is seeing the packets, but is never trying to arp for them. It really sounds like the cisco doesn't know that they are available via one of it's interfaces. Could you post it's route relating to the firewall? Linux's route -n would be helpful.

How are you talking to the cisco (over the IP of eth0, telnet, ssh)? Or over the "console"?

I googled this: http://puck.nether.net/lists/cisco-nsp/1408.html which describes enabling arp debuging. (arp debug). This should let you see why the cisco isn't doing arp to your firewall.

Can eth0 ping the cisco/web filter?
# ping $CISCO
Can eth0:0 ping the cisco
# ping -I $MS_EXT $CISCO
Do you get arp answers from the cisco? Can the cisco ping $MS_EXT?

What I'm getting at, is if eth0 works and eth0:0 doesn't, then either eth0:0's IP isn't getting off your interface, link is borked, or the cisco doesn't think that IP address should be coming for it, over that interface.

Do you get replies from the cisco when broadcasting? (-b may not be necessary)
ping -b 169.154.1.255

Do the broadcast address and netmask match, on both host's interfaces? (ifconfig still uses the A/B/C classes for this, I think)

Do your routes match for the local (eth0) network on the cisco and linux?

I assume you are doing some packet sniffing to see the arp traffic, which side of the link are you sniffing?

Hope this helps,
chris

edit: slightly premature post.

Thanks for all the input on this. As it turns out, I may have found the answer where I didn't expect it to be: DNS. The nameserver is on a different subnet, and packets have to be routed by the cisco to get to it. Still testing, but will post what I learn. If it turns out not to be DNS related, I'll provide more arp details.

Thanks again

It did turn out to be DNS related. The DNS server is on a different subnet but the same physical network, so packets were going into the cisco on eth0, being routed back out eth0 from the cisco to the DNS server, which then sent the replies back to the cisco, etc.. Somewhere along this path the packets were getting lost. As it happens, we're migrating to a different Class C range, so it was simple enough to move the DNS server to the same subnet as the firewall, and then everything worked as it should.

Thanks again for your time and suggestions. I probably should have seen the DNS issue sooner, but I was confused by the (incomplete) arp entries on the cisco and chased the wrong goose.