[SOLVED] Access firewall public IP from internal NATed DMZ machine

rsantos · 05-13-2010, 01:16 PM

Hi all,

I'm experiencing a problem I could no handle/circumvent on the past week, despite of the several "googling" and documents reading. I will try to mention all needed bits...

I'm managing a network with the following structure:
eth0: internal net
eth1: DMZ
eth2: 10 mbit/sec sync line with eight public /29 IP's + a /30 interconnection network. All public IP's must use the interconnection's network gateway.
eth3: ADSL with ppoe with fixed IP (ppp0).

Now, a few extra info:
- All internal traffic is routed through ppp0 except when it's destined for DMZ and public IP's range.
- DMZ traffic is routed either through ppp0 or eth2. This is done by source IP.
- eth2 has on IP from the interconnection network, and six public IP's are also defined on eth2. Command: "ip addr add ...."
- One public IP is routed directly do the DMZ. Both eth2 and eth1 have proxy_arp activated in order to achieve this.
- Several NAT Prerouting rules are defined in order, for example, direct all call's from public IP 3 port 80, to internal DMZ server port 80. There are many other rules defined but, let's just stick with 80 and IP 3.

Almost everything works like a charm, except that I can't access the DMZ Server at port 80 when I try connect to Public IP3 at port 80, and that connection originates from that same DMZ Server. resuming:
- I'm able to access the DMZ Server at port 80 anywhere in the world by opening a connection to public IP3 at Port 80. This includes opening a connection originating from the internal network at eth0.
- I cannot access it through that same Public IP3 at Port 80 from the DMZ Server. It should open a connection to "itself".

Can anyone help me on this matter ? Anything will do: Solution, link to another "Linux Networking how-to", etc...

Thanks in advance for all your help,
Rui Santos

SuperJediWombat! · 05-14-2010, 10:04 AM

You need to use SNAT.

Right now you probably have a rule that looks something like this:

Code:

iptables -t nat -A PREROUTING -p tcp --dport 80 --destination 203.0.0.3 -j DNAT --to-destination 10.0.0.3

Where 203.0.0.3 is the public IP and 10.0.0.3 is the web server inside the DMZ.

This works great for traffic coming outside of the DMZ, as the router picks it up, changes the destination and forwards it on to the the DMZ server. When the DMZ server receives that packet, it sees that is addressed to itself and that the reply is external so it sends the packet back out through the router. The router then does its DNAT magic, and reverses the original info.

However, if the traffic source is internal, the DMZ server 10.0.0.3 will reply to the internal source IP. This means that the router never receives the reply and cannot reverse the DNAT.

You need to make a rule which changes the source address of any packets originating from the DMZ which have been DNAT'd from the router.

Something like:

Code:

iptables -t nat -A POSTROUTING -p tcp --dport 80 -s 10.0.0.0/24 -j SNAT --to-source 10.0.0.250

Where 10.0.0.0/24 is the DMZ subnet, and 10.0.0.250 is the IP of the router.

rsantos · 05-24-2010, 02:32 PM

Hi SuperJediWombat!,

Thank you for your reply. First of all I'm sorry for the long reply. I had to take a week of vacations because a household problem.

You are correct about the first rule.

About your suggestion, this is what I had:
iptables -t nat -A POSTROUTING -s 10.0.0.144/29 -j SNAT --to-source 203.0.0.3
I also inserted the rule you told me to before this one. It looks like this:
iptables -t nat -A POSTROUTING -p tcp --dport 22 -s 10.0.0.144/29 -j SNAT --to-source 10.0.0.254

10.0.0.144/29 is the DMZ subnet
10.0.0.144 is the DMZ server
203.0.0.3 is public IP
10.0.0.254 is the DMZ gateway (One of the NICs on the linux router)

if I issue the command 'iptables -t nat -n -v', after I run the ssh command 'ssh -l root 203.0.0.3' several times, I get this output on the POSTROUTING chain:
0 0 SNAT tcp -- * * 10.0.0.144/29 0.0.0.0/0 tcp dpt:22 to:10.0.0.254
The first field is the number of packets processed by that specific rule, which indices that no packet is ever processed by that rule.

However with the next PREROUTING chain will always process that packet:
5 288 DNAT tcp -- * * 0.0.0.0/0 203.0.0.3 tcp dpt:22 to:10.0.0.144

So the PREROUTING chain always processes the packet. I loose the packets track after that.

Bottom line, the line you posted will never get processed.

Any idea why this should be ?

Thanks again for all your help,
Rui

SuperJediWombat! · 05-24-2010, 07:06 PM

Please post the full output of these commands:

Code:

ifconfig
iptables-save
ip route
cat /proc/sys/net/ipv4/ip_forward

You are tracing an ssh connection (which uses port 22) rather than a http connection (using port 80) my rule only covers port 80. Does web traffic get through?

rsantos · 05-25-2010, 05:47 AM

Hi SuperJediWombat!,

Thanks again for your quick reply.

Quote:

Originally Posted by SuperJediWombat!

Please post the full output of these commands:

Code:

ifconfig
iptables-save
ip route
cat /proc/sys/net/ipv4/ip_forward

Of course. Here it is. Also a few other commands:

Code:

ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:04:f4:e8:c0
          inet addr:192.168.0.254  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:98654938 errors:10 dropped:0 overruns:337 frame:10
          TX packets:143101490 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2835422255 (2.6 GiB)  TX bytes:4075724259 (3.7 GiB)
          Interrupt:9 Base address:0x6000

eth1      Link encap:Ethernet  HWaddr 00:50:04:f4:e7:29
          inet addr:10.0.0.254  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41161421 errors:28 dropped:0 overruns:25 frame:28
          TX packets:40791203 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3302250697 (3.0 GiB)  TX bytes:557998102 (532.1 MiB)
          Interrupt:9 Base address:0xe400

eth2      Link encap:Ethernet  HWaddr 00:50:04:2f:3f:c0
          inet addr:200.0.0.102  Bcast:200.0.0.255  Mask:255.255.255.252
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:34898669 errors:3 dropped:0 overruns:0 frame:3
          TX packets:29170719 errors:0 dropped:0 overruns:0 carrier:0
          collisions:163446 txqueuelen:1000
          RX bytes:2816394061 (2.6 GiB)  TX bytes:2914654165 (2.7 GiB)
          Interrupt:9 Base address:0xc800

eth3      Link encap:Ethernet  HWaddr 00:4f:4c:04:3f:00
          inet addr:192.168.6.254  Bcast:192.168.6.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:115345315 errors:0 dropped:46 overruns:0 frame:1
          TX packets:78078103 errors:0 dropped:0 overruns:0 carrier:0
          collisions:4102602 txqueuelen:1000
          RX bytes:3615122005 (3.3 GiB)  TX bytes:4044726966 (3.7 GiB)
          Interrupt:5 Base address:0x220

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:11 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1120 (1.0 KiB)  TX bytes:1120 (1.0 KiB)

ppp0      Link encap:Point-to-Point Protocol
          inet addr:200.0.0.50  P-t-P:200.0.0.51  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1000  Metric:1
          RX packets:2034011 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1577902 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:30
          RX bytes:2226718320 (2.0 GiB)  TX bytes:392476724 (374.2 MiB)

Code:

ip addr show eth2
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:50:04:2f:3f:c0 brd ff:ff:ff:ff:ff:ff
    inet 200.0.0.102/30 brd 200.0.0.255 scope global eth2
    inet 200.0.0.1/32 brd 200.0.0.1 scope global eth2
    inet 200.0.0.2/32 brd 200.0.0.2 scope global eth2
    inet 200.0.0.3/32 brd 200.0.0.3 scope global eth2
    inet 200.0.0.4/32 brd 200.0.0.4 scope global eth2
    inet 200.0.0.5/32 brd 200.0.0.5 scope global eth2
    inet 200.0.0.6/32 brd 200.0.0.6 scope global eth2

Code:

ip route
200.0.0.8 dev eth1  scope link
200.0.0.50 dev ppp0  scope link  src 200.0.0.50
200.0.0.51 dev ppp0  proto kernel  scope link  src 200.0.0.50
200.0.0.100/30 dev eth2  proto kernel  scope link  src 200.0.0.102
192.168.6.0/24 dev eth3  proto kernel  scope link  src 192.168.6.254
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.254
10.0.0.0/24 dev eth1  proto kernel  scope link  src 10.0.0.254
127.0.0.0/8 via 127.0.0.1 dev lo
default via 200.0.0.51 dev ppp0  metric 4007

Code:

ip rule show
32761:  from 192.168.0.242 lookup APPLICATIONS
32762:  from 10.0.0.128/26 lookup DMZ_TO_EFM
32763:  from 200.0.0.3/29 lookup FIXED_TO_EFM
32764:  from 200.0.0.101 lookup INTERCONNECTION
32765:  from 200.0.0.50 lookup ADSL
32766:  from all lookup main
32767:  from all lookup default

Code:

ip route show table APPLICATIONS
default via 200.0.0.101 dev eth2

Code:

ip route show table DMZ_TO_EFM
192.168.0.0/24 dev eth0  scope link
default via 200.0.0.101 dev eth2

Code:

ip route show table FIXED_TO_EFM
192.168.0.0/24 dev eth0  scope link
default via 200.0.0.101 dev eth2

Code:

ip route show table INTERCONNECTION
200.0.0.100/30 dev eth2  scope link  src 200.0.0.102
default via 200.0.0.101 dev eth2

Code:

ip route show table ADSL
200.0.0.50 dev ppp0  scope link  src 200.0.0.50
default via 200.0.0.51 dev ppp0

Code:

cat /proc/sys/net/ipv4/ip_forward
1

Code:

iptables-save
I've removed a lot of unneded rules and changed to public/DMZ IP's to fake ones. However, previous assumptions regarding DMZ server/router/IP's should remain the same.
# Generated by iptables-save v1.4.7 on Tue May 25 11:04:38 2010
*nat
:PREROUTING ACCEPT [100879:10560516]
:POSTROUTING ACCEPT [32917:3974660]
:OUTPUT ACCEPT [394:25140]
-A PREROUTING -s 10.0.0.0/24 -d 192.168.0.0/24 -j ACCEPT
-A PREROUTING -s 10.0.0.128/32 -d 200.0.0.1/32 -j ACCEPT
-A PREROUTING -d 200.0.0.1/32 -p tcp -m tcp -m multiport --dports 22,80,443 -j DNAT --to-destination 10.0.0.128
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.0.0.144
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp -m multiport --dports 80,443,53 -j DNAT --to-destination 10.0.0.144
-A PREROUTING -d 200.0.0.3/32 -p udp -m udp -m multiport --dports 53,9815 -j DNAT --to-destination 10.0.0.144
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 85 -j DNAT --to-destination 10.0.0.145:80
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 86 -j DNAT --to-destination 10.0.0.145:22
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 89 -j DNAT --to-destination 10.0.0.146:80
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 90 -j DNAT --to-destination 10.0.0.146:22
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 87 -j DNAT --to-destination 10.0.0.147:80
-A PREROUTING -d 200.0.0.3/32 -p tcp -m tcp --dport 88 -j DNAT --to-destination 10.0.0.147:22
-A PREROUTING -d 200.0.0.4/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.152
-A POSTROUTING -s 10.0.0.0/32 -d 192.168.0.0/24 -j ACCEPT
-A POSTROUTING -s 10.0.0.128/32 -d 200.0.0.1/32 -j ACCEPT
-A POSTROUTING -s 10.0.0.1/32 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.2/31 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.4/30 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.8/29 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.16/28 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.32/27 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.64/26 -o ppp0 -j MASQUERADE
-A POSTROUTING -s 10.0.0.128/29 -o eth2 -j SNAT --to-source 200.0.0.1
-A POSTROUTING -s 10.0.0.144/29 -o eth2 -j SNAT --to-source 200.0.0.3
-A POSTROUTING -s 10.0.0.152/29 -o eth2 -j SNAT --to-source 200.0.0.4
COMMIT
# Completed on Tue May 25 11:04:38 2010
# Generated by iptables-save v1.4.7 on Tue May 25 11:04:38 2010
*filter
:INPUT DROP [1071:43972]
:FORWARD DROP [20059:669124]
:OUTPUT DROP [250:15640]
:Cid4859560817886.0 - [0:0]
:Cid5994X3254.0 - [0:0]
:In_RULE_0 - [0:0]
:In_RULE_20 - [0:0]
:RULE_1 - [0:0]
:RULE_32 - [0:0]
:RULE_34 - [0:0]
:RULE_37 - [0:0]
:RULE_8 - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -m state --state NEW -j ACCEPT
-A INPUT -s 195.23.13.0/24 -d 200.0.0.100/32 -i eth2 -p icmp -m icmp --icmp-type 8/0 -m state --state NEW -j ACCEPT
-A INPUT -m state --state NEW -j RULE_8
-A INPUT -s 10.0.0.0/32 -m iprange --dst-range 200.0.0.1-200.0.0.6 -m state --state NEW -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 200.0.0.100/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.1/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.2/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.3/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.4/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.5/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.6/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 10.0.0.254/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 200.0.0.50/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -s 10.0.0.254/32 -i ppp0 -m state --state NEW -j In_RULE_0
-A FORWARD -p tcp -m tcp --dport 22 -m state --state NEW -j RULE_1
-A FORWARD -s 200.0.0.7/32 -p tcp -m tcp --dport 53 -m state --state NEW -j ACCEPT
-A FORWARD -s 200.0.0.7/32 -p udp -m udp --dport 53 -m state --state NEW -j ACCEPT
-A FORWARD -s 200.0.0.7/32 ! -d 192.168.0.0/24 -m state --state NEW -j ACCEPT
-A FORWARD -d 200.0.0.7/32 -m state --state NEW -j ACCEPT
-A FORWARD ! -d 192.168.0.0/24 -m state --state NEW -j Cid4859560817886.0
-A FORWARD -d 10.0.0.152/32 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A FORWARD -p tcp -m tcp -m multiport --dports 80,443,53 -m state --state NEW -j Cid5994X3254.0
-A FORWARD -p udp -m udp -m multiport --dports 53,9815 -m state --state NEW -j Cid5994X3254.0
-A FORWARD -s 192.168.0.0/24 -m state --state NEW -j ACCEPT
-A FORWARD -m state --state NEW -j RULE_37
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -o lo -m state --state NEW -j ACCEPT
-A Cid4859560817886.0 -s 10.0.0.152/32 -j ACCEPT
-A Cid4859560817886.0 -s 10.0.0.0/24 -j ACCEPT
-A Cid5994X3254.0 -d 10.0.0.128/32 -j RULE_32
-A Cid5994X3254.0 -d 10.0.0.144/32 -j RULE_32
-A Cid5994X3254.0 -d 10.0.0.145/32 -j RULE_32
-A Cid5994X3254.0 -d 10.0.0.146/32 -j RULE_32
-A Cid5994X3254.0 -d 10.0.0.147/32 -j RULE_32
-A In_RULE_0 -j LOG --log-prefix "RULE 0 -- DENY " --log-level 6
-A In_RULE_0 -j DROP
-A In_RULE_20 -j LOG --log-prefix "License" --log-level 6
-A In_RULE_20 -j ACCEPT
-A RULE_1 -j LOG --log-prefix "SSH " --log-level 6
-A RULE_1 -j ACCEPT
-A RULE_32 -j LOG --log-prefix "ACCEPT " --log-level 6
-A RULE_32 -j ACCEPT
-A RULE_34 -j LOG --log-prefix "LOGMEIN - " --log-level 6
-A RULE_34 -j DROP
-A RULE_37 -j LOG --log-prefix "ALL " --log-level 6
-A RULE_37 -j DROP
-A RULE_8 -j LOG --log-prefix "RULE 8 -- DENY " --log-level 6
-A RULE_8 -j DROP
COMMIT
# Completed on Tue May 25 11:04:38 2010
# Generated by iptables-save v1.4.7 on Tue May 25 11:04:38 2010
*mangle
:PREROUTING ACCEPT [6526969:3894436881]
:INPUT ACCEPT [631057:35150968]
:FORWARD ACCEPT [5863165:3852178969]
:OUTPUT ACCEPT [528794:1599704425]
:POSTROUTING ACCEPT [6370382:5451090787]
:MYSHAPER-OUT - [0:0]
-A POSTROUTING -o ppp0 -j MYSHAPER-OUT
-A MYSHAPER-OUT -p tcp -m tcp --sport 0:1024 -j MARK --set-xmark 0x17/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 0:1024 -j MARK --set-xmark 0x17/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 20 -j MARK --set-xmark 0x1a/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 443 -j MARK --set-xmark 0x15/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 5190 -j MARK --set-xmark 0x17/0xffffffff
-A MYSHAPER-OUT -p icmp -j MARK --set-xmark 0x14/0xffffffff
-A MYSHAPER-OUT -p udp -j MARK --set-xmark 0x15/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x15/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --sport 22 -j MARK --set-xmark 0x16/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --sport 80 -j MARK --set-xmark 0x16/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x16/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --dport 23 -j MARK --set-xmark 0x15/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --sport 23 -j MARK --set-xmark 0x15/0xffffffff
-A MYSHAPER-OUT -p tcp -m tcp --sport 80 -j MARK --set-xmark 0x19/0xffffffff
-A MYSHAPER-OUT -p tcp -m length --length 0:64 -j MARK --set-xmark 0x14/0xffffffff
-A MYSHAPER-OUT -m mark --mark 0x0 -j MARK --set-xmark 0x1a/0xffffffff
COMMIT
# Completed on Tue May 25 11:04:38 2010

Quote:

Originally Posted by SuperJediWombat!

You are tracing an ssh connection (which uses port 22) rather than a http connection (using port 80) my rule only covers port 80.

Yes, I was aware of that. I've changed your rule accordingly. Thanks for the "heads-up" anyway.

Quote:

Originally Posted by SuperJediWombat!

Does web traffic get through?

Yes, it does. SSH also gets through.

Once again, thanks for your help SuperJediWombat!
Regards,
Rui

SuperJediWombat! · 05-25-2010, 10:22 AM

What is the default gateway set to, from the web server?

Add this rule to iptables, replacing 10.0.0.254 with the gateway ip of the firewall/router from 10.0.0.144:

Code:

iptables -t nat -A POSTROUTING -s 10.0.0.144/24 -d 10.0.0.144/32  -o eth1 -p tcp --dport 80 -j SNAT --to-source 10.0.0.254

By the way, your network setup is pretty impressive

rsantos · 05-25-2010, 11:40 AM

Hi again SuperJediWombat!,

Quote:

Originally Posted by SuperJediWombat!

What is the default gateway set to, from the web server?

Add this rule to iptables, replacing 10.0.0.254 with the gateway ip of the firewall/router from 10.0.0.144:

Code:

iptables -t nat -A POSTROUTING -s 10.0.0.144/24 -d 10.0.0.144/32  -o eth1 -p tcp --dport 80 -j SNAT --to-source 10.0.0.254

That was the rule I applied earlier. However that rule alone is not enough. I found out the problem though

If you take a look at rule:
32762: from 10.0.0.128/26 lookup DMZ_TO_EFM
which contains:
192.168.0.0/24 dev eth0 scope link
default via 200.0.0.101 dev eth2
the 10.0.0.0/24 destination is missing, so it seems that the kernel is sending those TCP/22 packets thought dev eth2. It needs to go out though eth1. So I added:

Code:

ip route add 10.0.0.0/24 dev eth1 table DMZ_TO_EFM

and it all worked, finally...

I couldn't have done without your help. My huge thanks to you SuperJediWombat!

Quote:

Originally Posted by SuperJediWombat!

By the way, your network setup is pretty impressive

It has been tough to keep two Internet Connections, accepting connections to DMZ on both of them, creating DMZ chunks to be able to translate/reply through 8 different IP addresses, plus an extra proxy-arp'ed IP.
But I've learned a lot with this... and a little bit more with you

. Thanks a lot for that.

Best Regards,
Rui