[SOLVED] iptables rules ignored when trying to apply filter for lxc container

crts · 01-08-2020, 11:51 AM

I have the following lxc container setup on my host machine:

Code:

+--------------------------------+
|                                |
|         192.168.2.1            |
|      lxcbr (virtual bridge)    |
|            /    \              |
|           /      \             |
|          /        \            |
|  192.168.2.200     \           |
|   container1        \          |
|                192.168.2.10    |
|                 container2     |
|                                |
+--------------------------------+

I am trying to restrict access to container 192.168.2.10 from other containers. I am using the following ruleset with iptables:

Code:

echo "1" > /proc/sys/net/ipv4/ip_forward
modprobe iptable_nat
modprobe ipt_LOG

iptables --table filter -P INPUT DROP
iptables --table filter -P FORWARD DROP
iptables --table filter -P OUTPUT DROP

iptables -t filter -F
iptables -t raw -F

iptables -t raw -I PREROUTING -p tcp --dport 10025 -j TRACE
iptables -t raw -I PREROUTING -p tcp --sport 10025 -j TRACE
iptables -t raw -I PREROUTING -s 192.168.2.10 -j TRACE
iptables -t raw -I PREROUTING -d 192.168.2.10 -j TRACE

iptables -F restrict
iptables -X restrict
iptables -N restrict

iptables -A restrict -j LOG --log-level 6 --log-prefix "[FW] [restrict] ENTERED "
iptables -A restrict -j DROP

iptables -I OUTPUT -s 192.168.2.10 -j restrict
iptables -I OUTPUT -d 192.168.2.10 -j restrict

iptables -I FORWARD -s 192.168.2.10 -j restrict
iptables -I FORWARD -d 192.168.2.10 -j restrict

iptables -I INPUT -d 192.168.2.10 -j restrict
iptables -I INPUT -s 192.168.2.10 -j restrict

When a packet's destination is 192.168.2.10 it shall jump to the chain restrict. However, the rules are ignored. At first I thought that lxcbr might be bypassed but I can run tcpdump on interface lxcbr and I do see the packets arriving there, both ways:

Code:

# tcpdump -ni lxcbr
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxcbr, link-type EN10MB (Ethernet), capture size 262144 bytes
18:08:28.918571 IP 192.168.2.200.58884 > 192.168.2.10.10025: ...
18:08:28.918631 IP 192.168.2.10.10025 > 192.168.2.200.58884: ...
...

I never see a corresponding log entry in /var/log/messages. I tried to trace the packets with the trace rules in table raw but I never get any output in /var/log/syslog.

Why do the rules never match? Links to documentation that explains the behaviour would be much appreciated.

Additional info:

Code:

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.1.1     0.0.0.0         UG    1      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0
192.168.2.0     *               255.255.255.0   U     0      0        0 lxcbr


OS: Slackware 14.2
iptables: v1.6.0
lxc: v2.0.9

I have verified that the tracing works by tracing every packet with:

Code:

iptables -t raw -I PREROUTING -j TRACE

Even with this rule all packets get traced except the ones I am interested in.

PS: I am not looking for a workaround, e.g., putting the containers on separate networks. I want to understand what is going on.

TenTenths · 01-09-2020, 10:03 AM

From a pure network topology with your 192.168.2.0 interfaces with a 255.255.255.0 hostmask then packets on that network will NOT pass through your 192.168.2.1 interface they will be sent directly to the other server.

You could change the hostmask on your source machine to 255.255.255.255 and you'll see the packets being routed via 192.168.2.1

crts · 01-09-2020, 11:57 AM

Quote:

Originally Posted by TenTenths

From a pure network topology with your 192.168.2.0 interfaces with a 255.255.255.0 hostmask then packets on that network will NOT pass through your 192.168.2.1 interface they will be sent directly to the other server.

That did cross my mind which is why I ran tcpdump on the bridge. I did see those packets on that interface. Then I thought maybe the decision to forward the packet is made on a Layer on which netfilter does not operate. However, according to below flow chart it still should have passed through raw/PREROUTING, so I should have seen at least some traces of those packets in /var/log/syslog.

TenTenths · 01-09-2020, 01:11 PM

If the interface is in promiscuous mode you'll see all the traffic on the (virtual) segment regardless of whether it's passing "through" the gateway or not.

crts · 01-11-2020, 08:22 AM

Ok, I did some further testing. I put lxcbr in non-promiscuous mode and ran tcpdump on it again. After making a request I chould still see the packets arriving on lxcbr. I verified that the device was indeed in non-promiscuous mode by:

Code:

$ cat /sys/class/net/lxcbr/flags
0x1003
$ printf "%016d\n" $(echo 'ibase=16;obase=2;1003' | bc)
0001000000000011
       ^ indicates promiscuous mode state (1: true, 0: false)

The meaning of the device flags is documented in /usr/include/linux/if.h:

Code:

enum net_device_flags {
/* for compatibility with glibc net/if.h */
#if __UAPI_DEF_IF_NET_DEVICE_FLAGS
        IFF_UP                          = 1<<0,  /* sysfs */
        IFF_BROADCAST                   = 1<<1,  /* __volatile__ */
        IFF_DEBUG                       = 1<<2,  /* sysfs */
        IFF_LOOPBACK                    = 1<<3,  /* __volatile__ */
        IFF_POINTOPOINT                 = 1<<4,  /* __volatile__ */
        IFF_NOTRAILERS                  = 1<<5,  /* sysfs */
        IFF_RUNNING                     = 1<<6,  /* __volatile__ */
        IFF_NOARP                       = 1<<7,  /* sysfs */
        IFF_PROMISC                     = 1<<8,  /* sysfs */
        IFF_ALLMULTI                    = 1<<9,  /* sysfs */
        IFF_MASTER                      = 1<<10, /* __volatile__ */
        IFF_SLAVE                       = 1<<11, /* __volatile__ */
        IFF_MULTICAST                   = 1<<12, /* sysfs */
        IFF_PORTSEL                     = 1<<13, /* sysfs */
        IFF_AUTOMEDIA                   = 1<<14, /* sysfs */
        IFF_DYNAMIC                     = 1<<15, /* sysfs */
#endif /* __UAPI_DEF_IF_NET_DEVICE_FLAGS */

So the entry in /sys/class/net/lxcbr/flags confirmed that the device was in non-promiscuous mode. However, I also checked the log in /var/log/messages and found the following entries:

Code:

...
Jan 11 14:27:41 <machine> kernel: [ 1497.855109] device lxcbr entered promiscuous mode
Jan 11 14:27:48 <machine> kernel: [ 1504.860244] device lxcbr left promiscuous mode
...

As it turns out, every time I make a request lxcbr is automatically put in promiscuous mode and leaves it after a few seconds. I am not sure if this happens with all virtual NICs or if it has to do with LXC. Anyway, I now know why it behaves the way it does, so this case is finally solved.

Thanks.