CentOS Router/Firewall. Routing?

Gangrif · 11-22-2013, 10:40 AM

I am trying to build a Router/Firewall on CentOS.

Yes, i know there are a ton of firewall/nat gateway distros out there. The problem is, they are all home use/nat centric.

In my case, i want something more similar to an enterprise firewall. Inline, NAT/MASQ. I can build a nat gateway in my sleep.

I have a block of public ip's from my ISP, and i want to put a firewall between my router and the servers i build on this block. True firewall appliances, like cisco and juniper devices, are just too expensive for my small shop. I want something in between a home nat gateway, and an ASA.

I gave PFSense a go, and it got me close but i just couldnt get it going. I'm an RHCE, and am very familiar with RHEL, so i'd like to do this on CentOS.

I have a system setup, very similar to what's described here:
http://www.itadmintools.com/2012/10/...ewall-and.html

I have two vm's on my laptop, one connected to an isolated network, the other conencted to the default KVM network accessible to the laptop, and the isolated network.

I have things damn close to working.

With my isolated vm, I have the default gateway set to the isolated-side interface on the router machine. From the isolated machine, I can ping both interfaces on the router machine. but anything beyond that, dies at the router machine.

As far as I can tell, everying is configured properly.

ip_forward is enabled, i have iptable's forward table set to wide open for now, it does not seem to be getting stopped there. Yet it dies at the router machine.

What else can I try?

Thanks!

Ser Olmy · 11-22-2013, 11:14 AM

Quote:

Originally Posted by Gangrif

With my isolated vm, I have the default gateway set to the isolated-side interface on the router machine. From the isolated machine, I can ping both interfaces on the router machine. but anything beyond that, dies at the router machine.

Are you NATing the isolated KVM network, or does the next-hop router have a route to the KVM network?

Also, is IP forwarding enabled on the CentOS router? (With IP routing disabled you'll still be able to reach IP addresses assigned to other interfaces on the router itself.)

Gangrif · 11-22-2013, 11:34 AM

Quote:

Originally Posted by Ser Olmy

Are you NATing the isolated KVM network, or does the next-hop router have a route to the KVM network?

Also, is IP forwarding enabled on the CentOS router? (With IP routing disabled you'll still be able to reach IP addresses assigned to other interfaces on the router itself.)

The route from isolated client to internet looks like this:

Code:

isolated virtual network       libvirt "Default" network            Host Network       -> Internet
    192.168.100.0/24      ->      10.69.69.0/24    (natted) ->   10.1.1.0/24 (natted)       ISP

Basically, my laptop has wifi, to my home router, my laptop provides a nat gateway to my "Default" libvirt network, and the second libvirt network is isolated (no routing or dhcp enabled).

My router machine has forwarding enabled in sysctl, if thats what you're asking.

Code:

[Undr root@router ~]# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

Route table on my router machine looks like:

Code:

[Undr root@router ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.100.0   0.0.0.0         255.255.255.0   U         0 0          0 eth1
10.69.69.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
0.0.0.0         10.69.69.1      0.0.0.0         UG        0 0          0 eth0
[Undr root@router ~]#

And on my laptop:

Code:

[root@kenshin ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.1.1.1        0.0.0.0         UG        0 0          0 wlan0
10.1.1.0        0.0.0.0         255.255.255.0   U         0 0          0 wlan0
10.69.69.0      0.0.0.0         255.255.255.0   U         0 0          0 virbr0
[root@kenshin ~]#

I can get to the internet from the router machine, so its path to the internet appears clear.

Code:

[Undr root@router ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=44 time=48.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=44 time=34.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=44 time=36.9 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2162ms
rtt min/avg/max/mdev = 34.337/39.962/48.637/6.225 ms

Ser Olmy · 11-22-2013, 12:17 PM

Quote:

Originally Posted by Gangrif

The route from isolated client to internet looks like this:

Code:

isolated virtual network       libvirt "Default" network            Host Network       -> Internet
    192.168.100.0/24      ->      10.69.69.0/24    (natted) ->   10.1.1.0/24 (natted)       ISP

If I understand you correctly, the CentOS router sits between 192.168.100.0/24 and 10.69.69.0/24 and does not NAT outbound traffic. In that case, you will need to add a route for the 192.168.100.0/24 network on your laptop, pointing to the IP address of the CentOS router in the 10.69.69.0/24 network. Alternatively, you could have the CentOS router perform NAT overloading.

Try running tcpdump -i eth0 icmp on the CentOS router while pinging an Internet host from the isolated network. You should see packets leaving the eth0 interface with a source address of 192.168.100.<something>, but no return traffic since the laptop is missing a route to 192.168.100.0/24.

You can confirm this by pinging a host on the isolated network from your laptop; you probably won't see any packets arriving at the CentOS router, as the laptop will instead send these packets to its default gateway on the wireless LAN.

There could even be yet another issue here. Depending on the iptables rules on your laptop, the OS may or may not NAT traffic from 192.168.100.0/24. Run iptables -t nat -L POSTROUTING to have a look at the relevant rules.

Gangrif · 11-22-2013, 07:22 PM

Looks like postrouting is empty.

Code:

[Undr root@router ~]# iptables -t nat -L POSTROUTING
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
[Undr root@router ~]#

So, in case its not clear, the router vm is a test machine, virtual, on my laptop. These rfc19818 ip's are all just for local testing before I try to build this in production.

I'll try the other suggestions as well.

Thanks for all the help!

Gangrif · 11-22-2013, 07:38 PM

Ok, added a route to my laptop.

Code:

[root@kenshin ~]# ip route add 192.168.100.0/24 via 10.69.69.2
[root@kenshin ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.1.1.1        0.0.0.0         UG        0 0          0 wlan0
10.1.1.0        0.0.0.0         255.255.255.0   U         0 0          0 wlan0
10.69.69.0      0.0.0.0         255.255.255.0   U         0 0          0 virbr0
192.168.100.0   10.69.69.2      255.255.255.0   UG        0 0          0 virbr0
[root@kenshin ~]#

Now, if i try to ping from the client vm (on the isolated network through the router vm) to 8.8.8.8, i get destination port unreachable, where before i seemed to get nothing at all.

If i try to ping to the client from my laptop, i get the same issue that i was from the client out.

Code:

[root@kenshin ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
^C
--- 192.168.100.2 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 2999ms

Pinging the isolated side interface on the router from my laptop, i get replies.

Code:

[root@kenshin ~]# ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=0.360 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=0.459 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=0.472 ms
^C
--- 192.168.100.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.360/0.430/0.472/0.052 ms

Seems that routing is somehow dieing on the router?

Code:

[root@kenshin ~]# traceroute 192.168.100.2
traceroute to 192.168.100.2 (192.168.100.2), 30 hops max, 60 byte packets
 1  10.69.69.2 (10.69.69.2)  0.432 ms  0.306 ms  0.180 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  *^C
[root@kenshin ~]#

Gangrif · 11-22-2013, 07:50 PM

Here's the results of the tcpdump.

Code:

01:44:34.274886 IP 192.168.100.2 > google-public-dns-a.google.com: ICMP echo request, id 41988, seq 4, length 64
01:44:34.275112 IP 10.69.69.1 > 192.168.100.2: ICMP google-public-dns-a.google.com protocol 1 port 12607 unreachable, length 92

Just for fun, i ran the same dump on my laptop, first on wlan0 (my uplink to the wifi network) and then on virbr0, my virtual interface serving the router vm.

wlan0 saw none of the icmp traffic.

virbr0 showed me the same result as the router.

Code:

20:47:21.322972 IP 192.168.100.2 > google-public-dns-a.google.com: ICMP echo request, id 42500, seq 6, length 64
20:47:21.323056 IP 10.69.69.1 > 192.168.100.2: ICMP google-public-dns-a.google.com protocol 1 port 32482 unreachable, length 92

Something i noticed, is the time difference. I believe the time zone is incorrect on my router vm, this shouldnt affect anything though, should it?

Gangrif · 11-22-2013, 11:35 PM

Seems like my issues were vm related. I have things setup in my production IP space now, and things appear to be working.

Thanks!