LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Networking (https://www.linuxquestions.org/questions/linux-networking-3/)
-   -   OpenVPN in bridge mode cutting access to web (Server with 3 NICs acting as router) (https://www.linuxquestions.org/questions/linux-networking-3/openvpn-in-bridge-mode-cutting-access-to-web-server-with-3-nics-acting-as-router-4175426260/)

Joaquim Almeida 09-08-2012 11:33 AM

OpenVPN in bridge mode cutting access to web (Server with 3 NICs acting as router)
 
[ Scroll down to see the solution ]

Hi,

I'm facing a problem that I believe to be a simple one; nevertheless I'm missing it.

So I have this server running CentOS 5.8 with 3 NICs installed, acting as a router. The NIC-A connects to a LANswitch, NIC-B connects to the internet and NIC-C to a dedicated line.

My objective is to create a VPN in order to reach LANswitch from the Internet, and still everyone inside LANswitch can connect to the internet (like it always worked). For that I am using OpenVPN in bridge-mode.

As I start OpenVPN and bridge-start, it creates a tap0 and br0 virtual interface; and then everyone inside the LANswitch loses connection to the internet as it bridges NIC-B with the virtually created by OpenVPN.

I'm not very used to work with bridges so some doubts arise. I think it's possible to have NIC-B working with OpenVPN and still operate as a forward'er... right? If so, what am I missing?

Below is the information regarding configurations:

NIC-A = eth0 = LAN
NIC-B = eth1 = INTERNET
NIC-C = eth2 = DEDICATED LINE
(Forgot to mention before): net.ipv4.ip_forward = 1

[IFCONFIG without bridge - working well]
Code:

[root@AVNAS ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:6E:9C:9D:D2
          inet addr:10.6.0.100  Bcast:10.6.255.255  Mask:255.255.0.0
          inet6 addr: fe80::20c:6eff:fe9c:9dd2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16987582 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26694619 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2136762940 (1.9 GiB)  TX bytes:1894479874 (1.7 GiB)
          Interrupt:201

eth1      Link encap:Ethernet  HWaddr B0:48:7A:86:F7:13
          inet addr:1.2.3.4  Bcast:1.2.3.255  Mask:255.255.255.0
          inet6 addr: fe80::b248:7aff:fe86:f713/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:23260235 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14434761 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2765350936 (2.5 GiB)  TX bytes:1608018365 (1.4 GiB)
          Interrupt:209 Base address:0x2400

eth2      Link encap:Ethernet  HWaddr B0:48:7A:86:EA:8D
          inet addr:1.1.1.100  Bcast:1.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::b248:7aff:fe86:ea8d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:603729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:564718 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:216148511 (206.1 MiB)  TX bytes:95629247 (91.1 MiB)
          Interrupt:201 Base address:0x4000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:225852 errors:0 dropped:0 overruns:0 frame:0
          TX packets:225852 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:204327954 (194.8 MiB)  TX bytes:204327954 (194.8 MiB)

[IPTABLES without bridge - working well]
Code:

# Seleccionar a tabela 'filter'
*filter

# Definir a accao por defeito para a cadeia INPUT para ACCEPT e colocar os contadores a zero
:INPUT ACCEPT [0:0]

# Definir a accao por defeito para a cadeia FORWARD para ACCEPT e colocar os contadores a zero
:FORWARD ACCEPT [0:0]

# Definir a accao por defeito para a cadeia OUTPUT para ACCEPT e colocar os contadores a zero
:OUTPUT ACCEPT [0:0]

# Criar uma nova cadeia chamada RH-Firewall-1-INPUT e colocar os contadores a zero
:RH-Firewall-1-INPUT - [0:0]

# Fazer com que todos os pacotes do INPUT saltem para a cadeia RH-Firewall-1-INPUT
-A INPUT -j RH-Firewall-1-INPUT

# Fazer com que todos os pacotes do FORWARD saltem para a cadeia RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT

# Aceitar todo o trafego local no loopback
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

# Aceitar trafego ICMP como necessidade para operacao correcta do TCP/IP
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT

# Permitir ligacoes que estao a ser controladas por inspeccao do STATE
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Permitir livre transito a partir de eth0 para eth1 e eth2
-A RH-Firewall-1-INPUT -i eth0 -o eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -o eth2 -j ACCEPT

# httpd (apache - paginas web)
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

# ntp
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT

# netbios NAME
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 137 -j ACCEPT

# netbios DATAGRAM
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 138 -j ACCEPT

# netbios SESSION
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT

# samba
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT

# webmin
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 10000 -j ACCEPT

# ssh
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 20000 -j ACCEPT

# Rejeitar todos os pacotes que nao encontraram correspondencia
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# IGNORAR todos os pacotes que nao encontraram correspondencia
#-A RH-Firewall-1-INPUT -j DROP

# Activar configuracao para esta tabela
COMMIT

# Seleccionar a tabela 'nat'
*nat

# Mascarar o IP que sai de eth1 e eth2
-A POSTROUTING -o eth1 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE

# Activar configuracao para esta tabela
COMMIT

[--------- SEPERATOR ---------]

[CONFIG bridge-start]
Code:

#!/bin/sh

#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################

# Define Bridge Interface
br="br0"

# Define list of TAP interfaces to be bridged,
# for example tap="tap0 tap1 tap2".
tap="tap0"

# Define physical ethernet interface to be bridged
# with TAP interface(s) above.
eth="eth1"
eth_ip="1.2.3.4"
eth_netmask="255.255.255.0"
eth_broadcast="1.2.3.255"

for t in $tap; do
    openvpn --mktun --dev $t
done

brctl addbr $br
brctl addif $br $eth

for t in $tap; do
    brctl addif $br $t
done

for t in $tap; do
    ifconfig $t 0.0.0.0 promisc up
done

ifconfig $eth 0.0.0.0 promisc up

ifconfig $br $eth_ip netmask $eth_netmask broadcast $eth_broadcast

[CONFIG server.conf]
Code:

port 41532
proto udp
dev tap0
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem
server-bridge 1.2.3.254 255.255.255.0 1.2.3.51 1.2.3.60
push "route 1.2.3.0 255.255.255.0"
keepalive 10 120
comp-lzo
persist-key
persist-tun
status /var/log/openvpn-status.log
log /var/log/openvpn.log
log-append /var/log/openvpn.log
verb 3

[IFCONFIG with bridge]
Code:

# Seleccionar a tabela 'filter'
*filter

# Definir a accao por defeito para a cadeia INPUT para ACCEPT e colocar os contadores a zero
:INPUT ACCEPT [0:0]

# Definir a accao por defeito para a cadeia FORWARD para ACCEPT e colocar os contadores a zero
:FORWARD ACCEPT [0:0]

# Definir a accao por defeito para a cadeia OUTPUT para ACCEPT e colocar os contadores a zero
:OUTPUT ACCEPT [0:0]

# Criar uma nova cadeia chamada RH-Firewall-1-INPUT e colocar os contadores a zero
:RH-Firewall-1-INPUT - [0:0]

# Fazer com que todos os pacotes do INPUT saltem para a cadeia RH-Firewall-1-INPUT
-A INPUT -j RH-Firewall-1-INPUT

# Fazer com que todos os pacotes do FORWARD saltem para a cadeia RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT

# Aceitar todo o trafego local no loopback
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

# Aceitar trafego ICMP como necessidade para operacao correcta do TCP/IP
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT

# Permitir ligacoes que estao a ser controladas por inspeccao do STATE
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Permitir livre transito a partir de eth0 para eth1 e eth2
-A RH-Firewall-1-INPUT -i eth0 -o eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -o eth2 -j ACCEPT
-A RH-Firewall-1-INPUT -i tap0 -j ACCEPT
-A RH-Firewall-1-INPUT -i br0 -j ACCEPT


# httpd (apache - paginas web)
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

# ntp
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT

# netbios NAME
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 137 -j ACCEPT

# netbios DATAGRAM
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 138 -j ACCEPT

# netbios SESSION
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT

# samba
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT

# webmin
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 10000 -j ACCEPT

# ssh
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 20000 -j ACCEPT

## openvpn
-A RH-Firewall-1-INPUT -i eth1 -m state --state NEW -m udp -p udp --dport 41532 -j ACCEPT


# Rejeitar todos os pacotes que nao encontraram correspondencia
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# IGNORAR todos os pacotes que nao encontraram correspondencia
#-A RH-Firewall-1-INPUT -j DROP

# Activar configuracao para esta tabela
COMMIT

# Seleccionar a tabela 'nat'
*nat

# Mascarar o IP que sai de eth1
-A POSTROUTING -o eth1 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o tap0 -j MASQUERADE
-A POSTROUTING -o br0 -j MASQUERADE


# Activar configuracao para esta tabela
COMMIT

[IFCONFIG with bridge]
Code:

br0      Link encap:Ethernet  HWaddr 6E:35:9A:C8:47:E8
          inet addr:1.2.3.4  Bcast:1.2.3.255  Mask:255.255.255.0
          inet6 addr: fe80::6c35:9aff:fec8:47e8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)


eth0      Link encap:Ethernet  HWaddr 00:0C:6E:9C:9D:D2
          inet addr:10.6.0.100  Bcast:10.6.255.255  Mask:255.255.0.0
          inet6 addr: fe80::20c:6eff:fe9c:9dd2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:17012738 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26723234 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2139539184 (1.9 GiB)  TX bytes:1915589119 (1.7 GiB)
          Interrupt:201

eth1      Link encap:Ethernet  HWaddr B0:48:7A:86:F7:13
          inet6 addr: fe80::b248:7aff:fe86:f713/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:23267436 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14440939 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2770209713 (2.5 GiB)  TX bytes:1608833715 (1.4 GiB)
          Interrupt:209 Base address:0x2400


eth2      Link encap:Ethernet  HWaddr B0:48:7A:86:EA:8D
          inet addr:1.1.1.100  Bcast:1.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::b248:7aff:fe86:ea8d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:603729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:564736 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:216148511 (206.1 MiB)  TX bytes:95634008 (91.2 MiB)
          Interrupt:201 Base address:0x4000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:245226 errors:0 dropped:0 overruns:0 frame:0
          TX packets:245226 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:220140783 (209.9 MiB)  TX bytes:220140783 (209.9 MiB)

tap0      Link encap:Ethernet  HWaddr 6E:35:9A:C8:47:E8
          inet6 addr: fe80::6c35:9aff:fec8:47e8/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:0 (0.0 b)  TX bytes:148 (148.0 b)

Thanks in advance.

[--------- SOLUTION ---------]

So, it turns out that OpenVPN messes up the routing table that is associated with targeted NIC, after bridging.
I dunno why I haven't find topics pointing this out; I guess normally there aren't much PC's acting as routers these days...
My conclusions were made on a trial/error basis. Didn't lose much time after understanding what was happening.

I've issued the command "route" and got the answer:
[Before starting OpenVPN]
Code:

[root@AVNAS openvpn]# route
Kernel IP routing table
Destination    Gateway        Genmask        Flags Metric Ref    Use Iface
192.168.120.253 1.1.1.111      255.255.255.255 UGH  0      0        0 eth2
1.2.3.0        *              255.255.255.0  U    0      0        0 eth1
1.1.1.0        *              255.255.255.0  U    0      0        0 eth2
10.6.0.0        *              255.255.0.0    U    0      0        0 eth0
169.254.0.0    *              255.255.0.0    U    0      0        0 eth2
default        1.2.3.100      0.0.0.0        UG    0      0        0 eth1

[After starting OpenVPN]
Code:

[root@AVNAS openvpn]# route
Kernel IP routing table
Destination    Gateway        Genmask        Flags Metric Ref    Use Iface
192.168.120.253 1.1.1.111      255.255.255.255 UGH  0      0        0 eth2
1.2.3.0        *              255.255.255.0  U    0      0        0 br0
1.1.1.0        *              255.255.255.0  U    0      0        0 eth2
10.6.0.0        *              255.255.0.0    U    0      0        0 eth0
169.254.0.0    *              255.255.0.0    U    0      0        0 eth2
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

It needs a default gateway so I edited bridge-start and added:
Code:

#!/bin/sh

#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################

# Define Bridge Interface
br="br0"

# Define list of TAP interfaces to be bridged,
# for example tap="tap0 tap1 tap2".
tap="tap0"

# Define physical ethernet interface to be bridged
# with TAP interface(s) above.
eth="eth1"
eth_ip="1.2.3.4"
eth_gw="1.2.3.100"
eth_netmask="255.255.255.0"
eth_broadcast="1.2.3.255"

for t in $tap; do
    openvpn --mktun --dev $t
done

brctl addbr $br
brctl addif $br $eth

for t in $tap; do
    brctl addif $br $t
done

for t in $tap; do
    ifconfig $t 0.0.0.0 promisc up
done

ifconfig $eth 0.0.0.0 promisc up

ifconfig $br $eth_ip netmask $eth_netmask broadcast $eth_broadcast
route add default gw $eth_gw dev $br

And now I got:
Code:

[root@AVNAS openvpn]# route
Kernel IP routing table
Destination    Gateway        Genmask        Flags Metric Ref    Use Iface
192.168.120.253 1.1.1.111      255.255.255.255 UGH  0      0        0 eth2
1.2.3.0        *              255.255.255.0  U    0      0        0 br0
1.1.1.0        *              255.255.255.0  U    0      0        0 eth2
10.6.0.0        *              255.255.0.0    U    0      0        0 eth0
169.254.0.0    *              255.255.0.0    U    0      0        0 eth2
default        1.2.3.100      0.0.0.0        UG    0      0        0 br0

Not only that, it needs some tunning inside iptables (this is not new); what I found strange was what was needed inside it. According to OpenVPN Ethernet Bridging manual it needs:
Quote:

Now set up the Linux firewall to permit packets to flow freely over the newly created tap0 and br0 interfaces:

iptables -A INPUT -i tap0 -j ACCEPT
iptables -A INPUT -i br0 -j ACCEPT
iptables -A FORWARD -i br0 -j ACCEPT

I only have added:
Code:

# Seleccionar a tabela 'filter'
*filter

# Definir a accao por defeito para a cadeia INPUT para ACCEPT e colocar os contadores a zero
:INPUT ACCEPT [0:0]

# Definir a accao por defeito para a cadeia FORWARD para ACCEPT e colocar os contadores a zero
:FORWARD ACCEPT [0:0]

# Definir a accao por defeito para a cadeia OUTPUT para ACCEPT e colocar os contadores a zero
:OUTPUT ACCEPT [0:0]

# Criar uma nova cadeia chamada RH-Firewall-1-INPUT e colocar os contadores a zero
:RH-Firewall-1-INPUT - [0:0]

# Fazer com que todos os pacotes do INPUT saltem para a cadeia RH-Firewall-1-INPUT
-A INPUT -j RH-Firewall-1-INPUT

# Fazer com que todos os pacotes do FORWARD saltem para a cadeia RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT

# Aceitar todo o trafego local no loopback
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

# Aceitar trafego ICMP como necessidade para operacao correcta do TCP/IP
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT

# Permitir ligacoes que estao a ser controladas por inspeccao do STATE
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Permitir trafego de eth0 para br0 e eth2
-A RH-Firewall-1-INPUT -i eth0 -o br0 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -o eth2 -j ACCEPT

# httpd (apache - paginas web)
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

# ntp
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT

# netbios NAME
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 137 -j ACCEPT

# netbios DATAGRAM
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 138 -j ACCEPT

# netbios SESSION
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT

# samba
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT

# webmin
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 10000 -j ACCEPT

# ssh
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 20000 -j ACCEPT

# openvpn
-A RH-Firewall-1-INPUT -i br0 -m state --state NEW -m udp -p udp --dport 41532 -j ACCEPT

# Rejeitar todos os pacotes que nao encontraram correspondencia
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# IGNORAR todos os pacotes que nao encontraram correspondencia
#-A RH-Firewall-1-INPUT -j DROP

# Activar configuracao para esta tabela
COMMIT

# Seleccionar a tabela 'nat'
*nat

# Mascarar o IP que sai de br0 e eth2
-A POSTROUTING -o br0 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE

# Activar configuracao para esta tabela
COMMIT

And voila, done.

To be honest I still haven't tested OpenVPN client to connect, so I believe I could need to add the tap0 to iptables. I'll post here afterwards...

Compliments.

deadeyes 09-10-2012 12:12 AM

I think the output of brctl show might be interesting as well.


All times are GMT -5. The time now is 03:16 AM.