[ Scroll down to see the solution ]
Hi,
I'm facing a problem that I believe to be a simple one; nevertheless I'm missing it.
So I have this server running CentOS 5.8 with 3 NICs installed, acting as a router. The NIC-A connects to a LANswitch, NIC-B connects to the internet and NIC-C to a dedicated line.
My objective is to create a VPN in order to reach LANswitch from the Internet, and still everyone inside LANswitch can connect to the internet (like it always worked). For that I am using OpenVPN in bridge-mode.
As I start OpenVPN and bridge-start, it creates a tap0 and br0 virtual interface; and then everyone inside the LANswitch loses connection to the internet as it bridges NIC-B with the virtually created by OpenVPN.
I'm not very used to work with bridges so some doubts arise. I think it's possible to have NIC-B working with OpenVPN and still operate as a forward'er... right? If so, what am I missing?
Below is the information regarding configurations:
NIC-A = eth0 = LAN
NIC-B = eth1 = INTERNET
NIC-C = eth2 = DEDICATED LINE
(Forgot to mention before): net.ipv4.ip_forward = 1
[IFCONFIG without bridge - working well]
Code:
[root@AVNAS ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:6E:9C:9D:D2
inet addr:10.6.0.100 Bcast:10.6.255.255 Mask:255.255.0.0
inet6 addr: fe80::20c:6eff:fe9c:9dd2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16987582 errors:0 dropped:0 overruns:0 frame:0
TX packets:26694619 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2136762940 (1.9 GiB) TX bytes:1894479874 (1.7 GiB)
Interrupt:201
eth1 Link encap:Ethernet HWaddr B0:48:7A:86:F7:13
inet addr:1.2.3.4 Bcast:1.2.3.255 Mask:255.255.255.0
inet6 addr: fe80::b248:7aff:fe86:f713/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:23260235 errors:0 dropped:0 overruns:0 frame:0
TX packets:14434761 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2765350936 (2.5 GiB) TX bytes:1608018365 (1.4 GiB)
Interrupt:209 Base address:0x2400
eth2 Link encap:Ethernet HWaddr B0:48:7A:86:EA:8D
inet addr:1.1.1.100 Bcast:1.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::b248:7aff:fe86:ea8d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:603729 errors:0 dropped:0 overruns:0 frame:0
TX packets:564718 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:216148511 (206.1 MiB) TX bytes:95629247 (91.1 MiB)
Interrupt:201 Base address:0x4000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:225852 errors:0 dropped:0 overruns:0 frame:0
TX packets:225852 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:204327954 (194.8 MiB) TX bytes:204327954 (194.8 MiB)
[IPTABLES without bridge - working well]
Code:
# Seleccionar a tabela 'filter'
*filter
# Definir a accao por defeito para a cadeia INPUT para ACCEPT e colocar os contadores a zero
:INPUT ACCEPT [0:0]
# Definir a accao por defeito para a cadeia FORWARD para ACCEPT e colocar os contadores a zero
:FORWARD ACCEPT [0:0]
# Definir a accao por defeito para a cadeia OUTPUT para ACCEPT e colocar os contadores a zero
:OUTPUT ACCEPT [0:0]
# Criar uma nova cadeia chamada RH-Firewall-1-INPUT e colocar os contadores a zero
:RH-Firewall-1-INPUT - [0:0]
# Fazer com que todos os pacotes do INPUT saltem para a cadeia RH-Firewall-1-INPUT
-A INPUT -j RH-Firewall-1-INPUT
# Fazer com que todos os pacotes do FORWARD saltem para a cadeia RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
# Aceitar todo o trafego local no loopback
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
# Aceitar trafego ICMP como necessidade para operacao correcta do TCP/IP
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
# Permitir ligacoes que estao a ser controladas por inspeccao do STATE
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Permitir livre transito a partir de eth0 para eth1 e eth2
-A RH-Firewall-1-INPUT -i eth0 -o eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -o eth2 -j ACCEPT
# httpd (apache - paginas web)
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
# ntp
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT
# netbios NAME
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 137 -j ACCEPT
# netbios DATAGRAM
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 138 -j ACCEPT
# netbios SESSION
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT
# samba
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT
# webmin
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 10000 -j ACCEPT
# ssh
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 20000 -j ACCEPT
# Rejeitar todos os pacotes que nao encontraram correspondencia
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# IGNORAR todos os pacotes que nao encontraram correspondencia
#-A RH-Firewall-1-INPUT -j DROP
# Activar configuracao para esta tabela
COMMIT
# Seleccionar a tabela 'nat'
*nat
# Mascarar o IP que sai de eth1 e eth2
-A POSTROUTING -o eth1 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE
# Activar configuracao para esta tabela
COMMIT
[--------- SEPERATOR ---------]
[CONFIG bridge-start]
Code:
#!/bin/sh
#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################
# Define Bridge Interface
br="br0"
# Define list of TAP interfaces to be bridged,
# for example tap="tap0 tap1 tap2".
tap="tap0"
# Define physical ethernet interface to be bridged
# with TAP interface(s) above.
eth="eth1"
eth_ip="1.2.3.4"
eth_netmask="255.255.255.0"
eth_broadcast="1.2.3.255"
for t in $tap; do
openvpn --mktun --dev $t
done
brctl addbr $br
brctl addif $br $eth
for t in $tap; do
brctl addif $br $t
done
for t in $tap; do
ifconfig $t 0.0.0.0 promisc up
done
ifconfig $eth 0.0.0.0 promisc up
ifconfig $br $eth_ip netmask $eth_netmask broadcast $eth_broadcast
[CONFIG server.conf]
Code:
port 41532
proto udp
dev tap0
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem
server-bridge 1.2.3.254 255.255.255.0 1.2.3.51 1.2.3.60
push "route 1.2.3.0 255.255.255.0"
keepalive 10 120
comp-lzo
persist-key
persist-tun
status /var/log/openvpn-status.log
log /var/log/openvpn.log
log-append /var/log/openvpn.log
verb 3
[IFCONFIG
with bridge]
Code:
# Seleccionar a tabela 'filter'
*filter
# Definir a accao por defeito para a cadeia INPUT para ACCEPT e colocar os contadores a zero
:INPUT ACCEPT [0:0]
# Definir a accao por defeito para a cadeia FORWARD para ACCEPT e colocar os contadores a zero
:FORWARD ACCEPT [0:0]
# Definir a accao por defeito para a cadeia OUTPUT para ACCEPT e colocar os contadores a zero
:OUTPUT ACCEPT [0:0]
# Criar uma nova cadeia chamada RH-Firewall-1-INPUT e colocar os contadores a zero
:RH-Firewall-1-INPUT - [0:0]
# Fazer com que todos os pacotes do INPUT saltem para a cadeia RH-Firewall-1-INPUT
-A INPUT -j RH-Firewall-1-INPUT
# Fazer com que todos os pacotes do FORWARD saltem para a cadeia RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
# Aceitar todo o trafego local no loopback
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
# Aceitar trafego ICMP como necessidade para operacao correcta do TCP/IP
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
# Permitir ligacoes que estao a ser controladas por inspeccao do STATE
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Permitir livre transito a partir de eth0 para eth1 e eth2
-A RH-Firewall-1-INPUT -i eth0 -o eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -o eth2 -j ACCEPT
-A RH-Firewall-1-INPUT -i tap0 -j ACCEPT
-A RH-Firewall-1-INPUT -i br0 -j ACCEPT
# httpd (apache - paginas web)
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
# ntp
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT
# netbios NAME
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 137 -j ACCEPT
# netbios DATAGRAM
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 138 -j ACCEPT
# netbios SESSION
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT
# samba
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT
# webmin
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 10000 -j ACCEPT
# ssh
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 20000 -j ACCEPT
## openvpn
-A RH-Firewall-1-INPUT -i eth1 -m state --state NEW -m udp -p udp --dport 41532 -j ACCEPT
# Rejeitar todos os pacotes que nao encontraram correspondencia
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# IGNORAR todos os pacotes que nao encontraram correspondencia
#-A RH-Firewall-1-INPUT -j DROP
# Activar configuracao para esta tabela
COMMIT
# Seleccionar a tabela 'nat'
*nat
# Mascarar o IP que sai de eth1
-A POSTROUTING -o eth1 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o tap0 -j MASQUERADE
-A POSTROUTING -o br0 -j MASQUERADE
# Activar configuracao para esta tabela
COMMIT
[IFCONFIG
with bridge]
Code:
br0 Link encap:Ethernet HWaddr 6E:35:9A:C8:47:E8
inet addr:1.2.3.4 Bcast:1.2.3.255 Mask:255.255.255.0
inet6 addr: fe80::6c35:9aff:fec8:47e8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:468 (468.0 b)
eth0 Link encap:Ethernet HWaddr 00:0C:6E:9C:9D:D2
inet addr:10.6.0.100 Bcast:10.6.255.255 Mask:255.255.0.0
inet6 addr: fe80::20c:6eff:fe9c:9dd2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17012738 errors:0 dropped:0 overruns:0 frame:0
TX packets:26723234 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2139539184 (1.9 GiB) TX bytes:1915589119 (1.7 GiB)
Interrupt:201
eth1 Link encap:Ethernet HWaddr B0:48:7A:86:F7:13
inet6 addr: fe80::b248:7aff:fe86:f713/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:23267436 errors:0 dropped:0 overruns:0 frame:0
TX packets:14440939 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2770209713 (2.5 GiB) TX bytes:1608833715 (1.4 GiB)
Interrupt:209 Base address:0x2400
eth2 Link encap:Ethernet HWaddr B0:48:7A:86:EA:8D
inet addr:1.1.1.100 Bcast:1.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::b248:7aff:fe86:ea8d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:603729 errors:0 dropped:0 overruns:0 frame:0
TX packets:564736 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:216148511 (206.1 MiB) TX bytes:95634008 (91.2 MiB)
Interrupt:201 Base address:0x4000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:245226 errors:0 dropped:0 overruns:0 frame:0
TX packets:245226 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:220140783 (209.9 MiB) TX bytes:220140783 (209.9 MiB)
tap0 Link encap:Ethernet HWaddr 6E:35:9A:C8:47:E8
inet6 addr: fe80::6c35:9aff:fec8:47e8/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:0 (0.0 b) TX bytes:148 (148.0 b)
Thanks in advance.
[--------- SOLUTION ---------]
So, it turns out that OpenVPN messes up the routing table that is associated with targeted NIC, after
bridging.
I dunno why I haven't find topics pointing this out; I guess normally there aren't much PC's acting as routers these days...
My conclusions were made on a trial/error basis. Didn't lose much time after understanding what was happening.
I've issued the command "route" and got the answer:
[Before starting OpenVPN]
Code:
[root@AVNAS openvpn]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.120.253 1.1.1.111 255.255.255.255 UGH 0 0 0 eth2
1.2.3.0 * 255.255.255.0 U 0 0 0 eth1
1.1.1.0 * 255.255.255.0 U 0 0 0 eth2
10.6.0.0 * 255.255.0.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth2
default 1.2.3.100 0.0.0.0 UG 0 0 0 eth1
[After starting OpenVPN]
Code:
[root@AVNAS openvpn]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.120.253 1.1.1.111 255.255.255.255 UGH 0 0 0 eth2
1.2.3.0 * 255.255.255.0 U 0 0 0 br0
1.1.1.0 * 255.255.255.0 U 0 0 0 eth2
10.6.0.0 * 255.255.0.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth2
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
It needs a default gateway so I edited bridge-start and
added:
Code:
#!/bin/sh
#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################
# Define Bridge Interface
br="br0"
# Define list of TAP interfaces to be bridged,
# for example tap="tap0 tap1 tap2".
tap="tap0"
# Define physical ethernet interface to be bridged
# with TAP interface(s) above.
eth="eth1"
eth_ip="1.2.3.4"
eth_gw="1.2.3.100"
eth_netmask="255.255.255.0"
eth_broadcast="1.2.3.255"
for t in $tap; do
openvpn --mktun --dev $t
done
brctl addbr $br
brctl addif $br $eth
for t in $tap; do
brctl addif $br $t
done
for t in $tap; do
ifconfig $t 0.0.0.0 promisc up
done
ifconfig $eth 0.0.0.0 promisc up
ifconfig $br $eth_ip netmask $eth_netmask broadcast $eth_broadcast
route add default gw $eth_gw dev $br
And now I got:
Code:
[root@AVNAS openvpn]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.120.253 1.1.1.111 255.255.255.255 UGH 0 0 0 eth2
1.2.3.0 * 255.255.255.0 U 0 0 0 br0
1.1.1.0 * 255.255.255.0 U 0 0 0 eth2
10.6.0.0 * 255.255.0.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth2
default 1.2.3.100 0.0.0.0 UG 0 0 0 br0
Not only that, it needs some tunning inside iptables (this is not new); what I found strange was what was needed inside it. According to
OpenVPN Ethernet Bridging manual it needs:
Quote:
Now set up the Linux firewall to permit packets to flow freely over the newly created tap0 and br0 interfaces:
iptables -A INPUT -i tap0 -j ACCEPT
iptables -A INPUT -i br0 -j ACCEPT
iptables -A FORWARD -i br0 -j ACCEPT
|
I only have
added:
Code:
# Seleccionar a tabela 'filter'
*filter
# Definir a accao por defeito para a cadeia INPUT para ACCEPT e colocar os contadores a zero
:INPUT ACCEPT [0:0]
# Definir a accao por defeito para a cadeia FORWARD para ACCEPT e colocar os contadores a zero
:FORWARD ACCEPT [0:0]
# Definir a accao por defeito para a cadeia OUTPUT para ACCEPT e colocar os contadores a zero
:OUTPUT ACCEPT [0:0]
# Criar uma nova cadeia chamada RH-Firewall-1-INPUT e colocar os contadores a zero
:RH-Firewall-1-INPUT - [0:0]
# Fazer com que todos os pacotes do INPUT saltem para a cadeia RH-Firewall-1-INPUT
-A INPUT -j RH-Firewall-1-INPUT
# Fazer com que todos os pacotes do FORWARD saltem para a cadeia RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
# Aceitar todo o trafego local no loopback
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
# Aceitar trafego ICMP como necessidade para operacao correcta do TCP/IP
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
# Permitir ligacoes que estao a ser controladas por inspeccao do STATE
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Permitir trafego de eth0 para br0 e eth2
-A RH-Firewall-1-INPUT -i eth0 -o br0 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -o eth2 -j ACCEPT
# httpd (apache - paginas web)
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
# ntp
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT
# netbios NAME
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 137 -j ACCEPT
# netbios DATAGRAM
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m udp -p udp --dport 138 -j ACCEPT
# netbios SESSION
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT
# samba
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT
# webmin
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 10000 -j ACCEPT
# ssh
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -m tcp -p tcp --dport 20000 -j ACCEPT
# openvpn
-A RH-Firewall-1-INPUT -i br0 -m state --state NEW -m udp -p udp --dport 41532 -j ACCEPT
# Rejeitar todos os pacotes que nao encontraram correspondencia
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# IGNORAR todos os pacotes que nao encontraram correspondencia
#-A RH-Firewall-1-INPUT -j DROP
# Activar configuracao para esta tabela
COMMIT
# Seleccionar a tabela 'nat'
*nat
# Mascarar o IP que sai de br0 e eth2
-A POSTROUTING -o br0 -j MASQUERADE
-A POSTROUTING -o eth2 -j MASQUERADE
# Activar configuracao para esta tabela
COMMIT
And
voila, done.
To be honest I still haven't tested OpenVPN client to connect, so I believe I could need to add the tap0 to iptables. I'll post here afterwards...
Compliments.