iptables ignores ens160 interface name.

hombibi · 09-17-2017, 10:50 AM

Hi

This question was first posted (now deleted) under "Security". It realized that it is better suited for the networking forum here.

I am building a firewall on a dedicated vm with iptables (ESXi, Ubuntu 16.04.3 LTS, two network interfaces).

One of the objectives is to only accept SSH on one network interface.

But when I load a rule to allow ssh input on an interface I instead lose all connections to the vm.
It seems as if iptables does not accept (silently) the interface name.

I load the firewall with the following script:

Code:

#!/bin/bash

# root check
if [ "$(id -u)" != "0" ]; then
   printf "\033[40m\033[1;31mERROR: Root check FAILED (you MUST be root to use this script)! Quitting...\033[0m\n\n" >&2
   exit 1
fi

# start flushing all rules.
iptables -t filter -F
iptables -t nat -F

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

# Set access to and from the loopback interface
iptables -A INPUT -i lo -m comment --comment "Loopback" -j ACCEPT
iptables -A OUTPUT -o lo -m comment --comment "Loopback" -j ACCEPT

# Allow ssh connections on tcp port 22 to the lan interface,
# only from local networks:
iptables -A INPUT -i ens160 -p tcp -s 192.168.154.0/24 --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o ens160 -p tcp -s 192.168.154.0/24 --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT

If I remove -i ens160 than it works fine. Anyone understand what's going on, is iptables struggeling with interface names other than eth0 and the like? Or is it a misunderstanding on my side?

Mind you, for the time being both interfaces are connected to the same local sub net.

Thanks Hombibi

lazydog · 09-18-2017, 03:57 PM

First things first. Once you start this firewall it applies to all interfaces, so if you want certain things to be accessible over one of the other interfaces you are going to have to set rules for that interface also.

Code:

iptables -A INPUT -i ens160 -p tcp -s 192.168.154.0/24 --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o ens160 -p tcp -s 192.168.154.0/24 --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT

Your output would need to be changed to -d 192.168.154.0/24 as the source of outgoing packets are going to be the ip address of your server and the destination is going to be to where the traffic is going.

brebs · 09-18-2017, 04:10 PM

Also - by not allowing ICMP, you are preventing Path MTU Discovery from working, setting yourself up for future intermittent connection problems.

hombibi · 09-18-2017, 05:01 PM

Thanks LazyDog, i missed that one.

So I changed to

iptables -A INPUT -i ens160 -p tcp --source 192.168.154.0/24 --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o ens160 -p tcp --dest 192.168.154.0/24 --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT

But no change, exact same behavior. Nevertheless this change should be without question, I'll keep it of course.

After that I upgraded the kernel from 4.4.0.93-generic to 4.4.0.96-generic. ( was actually looking for an iptables upgrade. )

This lead to even stranger effects:
1) I can reach the VM via SSH, with the firewall applied. So that is a step forward. For now I have to subscribe that to the new kernel.
2) However I can reach the VM via either interface while only ens160 should be accepting.
3) If I change back the --dest to --source in the output rule (thus put back the error), stop and flush the firewall and reload the script, I can access the firewall over SSH fine: the --source and --dest do not seem to have any effect.

Not sure I understand what is happening, could this be due to both interfaces (currently) connected to the same subnet?

hombibi · 09-18-2017, 05:31 PM

Brebs

No specific reason to block ICMP, I had come across it, but not considered it yet so thanks for pointing it out.

Given that my default policy is drop to each chain I'd need to allow new, established and related INPUT and OUTPUT on each interface. So this will be:

Code:

# Allow icmp connections in and out via the lan interface
iptables -A OUTPUT -o ens160 -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT  -i ens160 -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Allow icmp connections in and out via the wan interface
iptables -A INPUT  -i ens192 -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -o ens192 -p icmp -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

lazydog · 09-19-2017, 08:41 AM

Quote:

Originally Posted by hombibi

Not sure I understand what is happening, could this be due to both interfaces (currently) connected to the same subnet?

You are aware that you shouldn't have 2 interfaces connected to the same network without them being setup in some sort of port-channel.

OK can you give the following output from both the VM and the host:

Code:

ip addr
route -n

We'll take this is steps.

hombibi · 09-19-2017, 04:40 PM

The host machine is a HP Micro Gen 8. Both NIC's are connected the same network and are enabled, however only one has an IP address. I thought the second was bridged but I can't find any evidence for that yet. I'll have to take the box out of the cabinet and connect it to a screen to have a look in the bios/smart provisioning setup.

As per ILO4

Port MAC Address IPv4 Address IPv6 Address Status Team/Bridge
1 xx:xx:xx:xx:b1:84 192.168.154.5 xxx:xxxx:xxxx:2060/64 OK N/A
2 xx:xx:xx:xx:b1:85 N/A N/A OK N/A

ESXi shows the same physical nics / Mac Adressess
Each physical nic is attached to one virtual switch.
Each Virtual switch is attached to one virtual NIC of the virtual machine.
The virtual machine has two virtual nics, ens160 and ens192:

I can't do ip addr or route -n on the bare metal ESXi server, or at least have not found how.

On the virtual machine however:

Code:

~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.154.127/24 brd 192.168.154.255 scope global ens160
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.154.122/24 brd 192.168.154.255 scope global ens192
       valid_lft forever preferred_lft forever

Code:

~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.154.51  0.0.0.0         UG    0      0        0 ens160
192.168.154.0   0.0.0.0         255.255.255.0   U     0      0        0 ens160
192.168.154.0   0.0.0.0         255.255.255.0   U     0      0        0 ens192

So the second nic of the host does not have an IP address, or not a different IP address. Nevertheless this one is connected to a virtual switch, which is connected to a virtual nic of a virtual machine. I guess that's not all right. However both virtual nics get IP addresses from the local dhcp server.
Thanks.

lazydog · 09-20-2017, 02:19 PM

Quote:

Originally Posted by hombibi

The host machine is a HP Micro Gen 8. Both NIC's are connected the same network and are enabled, however only one has an IP address. I thought the second was bridged but I can't find any evidence for that yet. I'll have to take the box out of the cabinet and connect it to a screen to have a look in the bios/smart provisioning setup.

As per ILO4

Port MAC Address IPv4 Address IPv6 Address Status Team/Bridge
1 xx:xx:xx:xx:b1:84 192.168.154.5 xxx:xxxx:xxxx:2060/64 OK N/A
2 xx:xx:xx:xx:b1:85 N/A N/A OK N/A

ESXi shows the same physical nics / Mac Adressess
Each physical nic is attached to one virtual switch.
Each Virtual switch is attached to one virtual NIC of the virtual machine.
The virtual machine has two virtual nics, ens160 and ens192:

I can't do ip addr or route -n on the bare metal ESXi server, or at least have not found how.

On the virtual machine however:

Code:

~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.154.127/24 brd 192.168.154.255 scope global ens160
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.154.122/24 brd 192.168.154.255 scope global ens192
       valid_lft forever preferred_lft forever

Code:

~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.154.51  0.0.0.0         UG    0      0        0 ens160
192.168.154.0   0.0.0.0         255.255.255.0   U     0      0        0 ens160
192.168.154.0   0.0.0.0         255.255.255.0   U     0      0        0 ens192

So the second nic of the host does not have an IP address, or not a different IP address. Nevertheless this one is connected to a virtual switch, which is connected to a virtual nic of a virtual machine. I guess that's not all right. However both virtual nics get IP addresses from the local dhcp server.
Thanks.

Per your output of ip addr the second nic does have an ip address of 192.168.154.112 and has an entry in the routing table.
I'm trying to figure out exactly what nic is doing what job. Why are you not bonding them together since they are both on the same network?

what does the following give you:

Code:

netstat -plnt

I know you believe that the first nic should be used but it could well be that the second nic is the one that is really being used. Some of the issues you run into with more than 1 nic on a network that is not teamed or bonded. Is it possible to shut down the nic that shouldn't be used?

hombibi · 09-20-2017, 05:32 PM

Hi LazyDog,

The Virtual server is intended to become the main router/firewall for the network.
I can either bond the two nics and increase capacity and provide redundancy, or I can physically( With a VM ?) separate Wan and Lan if I do not. The latter sounds safer, but as the server is a VM I don't think that that still matters. I don't have a preference yet, and I agree that those assumption must now be properly tested. I'll probably go for bonded, but as the server is a virtual machine I think/thought it does not - should not matter for setting up and testing the firewall/router.

I also think that the second physical nic, although not showing up in ILO4, is properly active, given that ESXi allows me to assign the physical nic to a virtual switch. The ESXi server only allows me to assign available physical nics to virtual switches.
So I'll have to do some more digging to find out why I don't see an IP address on the second nic in the ILO4 interface. I found that same behavior on a few forums, one explaining that that would be caused by the userinterface, and another that the network interface is actually working fine. So not seeing the nic in ILO4 might be a red herring.

SSH to either ens160 or ens192:

~$ sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1338/sshd
tcp6 0 0 :::22 :::* LISTEN 1338/sshd

Update:
As disabling the interfaces is concerned: it gets tricky/interesting now. I hope not to make mistakes describing what I see:

I pulled out the cable to mac add 85: mac address 84 still connected:
ILo states network degraded.
Interface 84 has 192.168.154.5 which is the management interface for ESXI's embedded vSphere webclient.
ESXi embedded vSphere webclient still up and reachable.
Interface mac addr 85 is now disconnected:
Neither ip address 122 or 127 of the virtual machine can be reached anymore.

In the vsphere client I see that vmnic0 is still active. vmnic0 is mac addr 85, vmnic0 is connected to vm mac address 1b:56, ip 192.168.154.122. However I can not reach this ip address from outside the server.
Also vmnic1 is now disabled: vmnic1 is mac 84 and connected to mac addr 14:4c -ip- 192.168.154.127 and can't be reached.

What is strange is that one interface is up, but I can't reach the interface connected to it.

Once I replug the cable and unplug the other one I can reach both VM network interfaces again. But I can't reach ESXi's embedded vSphere webclient anymore. So the webclient claims one interface, and the virtual nics the other. Not what we want, not what is represented in the vSphere web client.

I think I should be able to correct this in VMware ESXi, need to search though.

However I still think this should not have anything to do with the behavior of IPTables within the Virtual machine: currently the vm appears to have two virtual nics allocated to one physical nic, (even though the webclient does not comply with that) and that would be the same for Iptables as if both interfaces would be bonded.

So I think we have three issues now: one the setup of ESXi, the representation of virtual networks in ESXi webclient, and the behaviour of iptables. I'll focus on the first two first.

I think we are making progress, albeit in the wrong direction :-), thanks for the support so far. (might be a day or two until I can work on this further.)

Thanks, hombibi