LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 01-14-2016, 09:07 AM   #1
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Rep: Reputation: 15
Linux Router.


I am at a loss here. I've done this before, with success, but I'm having a problem on this system. So I'm looking for another set of eyes.


I have a system which is connected to a VLAN trunk port, and the system has two VLAN tagged interfaces. One of which has a public IP, the other of which is using private IP space.

My VM's are running on the private IP space, and the public IP is used mainly for management of the system.

I have a second system, my laptop, which I'd like to access the virtual systems with, over TCP/IP.

This should be simple. I have the VM host setup to so IP MASQ for the private IP space, the VM's can get out to the internet using that. I have IP_Forward enabled on the VM host. I have IPTables setup to allow the traffic.

If i add a route on my laptop, to point my private address space, 192.168.100.0/24, to an alternate gateway, the public IP of my VM host, I can ping the private IP on the host, but NOT my VM's.

From the VM host, I can connect to the VM's ip's all day long.

If i try to ping, or trace to the private IP's on my VM Host, the request gets redirected back to the default gateway for the network that both my laptop and VM host live on.

It's like the VM host is setup to refuse to route requests, anything beyone it's local TCP stack is sent back to its default gateway.

The local "private" interface of the VM system, from my laptop:
Code:
[lagern@starwind ~]$ ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=0.315 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=0.271 ms
the IP of one of the VM's on my VM host:
Code:
[lagern@starwind ~]$ ping 192.168.100.10
PING 192.168.100.10 (192.168.100.10) 56(84) bytes of data.
From xx.147.1.50 icmp_seq=1 Time to live exceeded
From xx.147.60.112: icmp_seq=2 Redirect Host(New nexthop: xx.147.60.1)
If i add a rule to the NAT/Prerouting table in iptables on my vm host, to log traffic, it shows that it's been redirected before it even gets to iptables. showing both the input and output interfaces as the public interface (if it had accepted them to route, the output interface should have been the internal private interface).

This is a RHEL7 system, I have a similar router setup on a CentOS6 system which is working well, it routes both public and private/natted ip's without a problem.

I am not using firewalld on RHEL7, rather iptables directly.

I have ip_forward enabled in sysctl.conf.

Code:
[root@gollum ~]# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0
Routing table on my workstation:
Code:
[lagern@starwind ~]$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         xx.147.60.1    0.0.0.0         UG        0 0          0 enp14s0
0.0.0.0         172.27.33.1     0.0.0.0         UG        0 0          0 wlp4s0
10.1.1.0        10.250.0.5      255.255.255.0   UG        0 0          0 tun0
10.250.0.1      10.250.0.5      255.255.255.255 UGH       0 0          0 tun0
10.250.0.5      0.0.0.0         255.255.255.255 UH        0 0          0 tun0
xx.147.2.131   172.27.33.1     255.255.255.255 UGH       0 0          0 wlp4s0
xx.147.4.188   xx.147.60.1    255.255.255.255 UGH       0 0          0 enp14s0
xx.147.60.0    0.0.0.0         255.255.255.128 U         0 0          0 enp14s0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
172.27.33.0     0.0.0.0         255.255.255.0   U         0 0          0 wlp4s0
192.168.1.0     10.250.0.5      255.255.255.0   UG        0 0          0 tun0
192.168.2.0     10.250.0.5      255.255.255.0   UG        0 0          0 tun0
192.168.3.0     10.250.0.5      255.255.255.0   UG        0 0          0 tun0
192.168.80.0    10.250.0.5      255.255.255.0   UG        0 0          0 tun0
192.168.100.0   xx.147.60.112  255.255.255.0   UG        0 0          0 enp14s0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr0
There are several routes in there for things like docker, libvirt, and vpn that are all in place in my workstation. That i'm aware of, 192.168.100.0 conflicts with none of these.


Is there some other flag that I need to set in order to tell the kernel that it's allowed to route traffic?


Sorry if this is rambling and all over the place, if you need more info, just ask.

Thanks!
 
Old 01-14-2016, 02:15 PM   #2
tlowk
Member
 
Registered: Nov 2003
Location: Belgium
Distribution: Slackware
Posts: 184

Rep: Reputation: 36
Hi,

Are you using any firewall rules?

iptables -t nat -L -n
iptables -L -n

my first guess would be that there is some unexpected rule

best regards,
 
Old 01-14-2016, 03:47 PM   #3
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
Here's the full output of iptables-save

Code:
# Generated by iptables-save v1.4.21 on Thu Jan 14 16:45:24 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [129:8491]
:OUTPUT ACCEPT [274925:79193831]
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5432 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2222:2223 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5900:6923 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 49152:49216 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 7410 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6100 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 111 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 111 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 662 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 662 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 875 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 875 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 892 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 892 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2049 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 32769 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 32803 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 67 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i ovirtmgmt -o Evil -j ACCEPT
-A FORWARD -j LOG
COMMIT
# Completed on Thu Jan 14 16:45:24 2016
# Generated by iptables-save v1.4.21 on Thu Jan 14 16:45:24 2016
*nat
:PREROUTING ACCEPT [12892:1587274]
:INPUT ACCEPT [262:52462]
:OUTPUT ACCEPT [952:59134]
:POSTROUTING ACCEPT [955:59334]
-A PREROUTING -j LOG --log-prefix "PreRouting: "
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE
COMMIT
# Completed on Thu Jan 14 16:45:24 2016
The interface that holds the public ip is called "ovirtmgmt" (this is an ovirt host), and the internet network is called Evil. These happen to be the network names ovirt manages.
 
Old 01-14-2016, 03:52 PM   #4
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
Also, youll notice my log rule there. That's to try to see whats happening as packets are entering iptables.

a ping from my workstation to the .10 address in my private space generates the following log:

Code:
Jan 14 16:50:28 gollum kernel: IN=ovirtmgmt OUT=ovirtmgmt MAC=bc:30:5b:e0:2f:1b:38:c9:86:04:8e:0a:08:00 SRC=xx.147.60.99 DST=192.168.100.10 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=41749 DF PROTO=ICMP TYPE=8 CODE=0 ID=27758 SEQ=1
 
Old 01-14-2016, 04:21 PM   #5
tlowk
Member
 
Registered: Nov 2003
Location: Belgium
Distribution: Slackware
Posts: 184

Rep: Reputation: 36
I see this rule
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE

If your laptop is not in 192.168.100.0/24 but it sends a packet to a host in that range
the reply will come from the masqueraded addres, which could be a problem sinze I guess this
could be something like xx.147.60.99
 
Old 01-14-2016, 04:21 PM   #6
tlowk
Member
 
Registered: Nov 2003
Location: Belgium
Distribution: Slackware
Posts: 184

Rep: Reputation: 36
I see this rule
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE

If your laptop is not in 192.168.100.0/24 but it sends a packet to a host in that range
the reply will come from the masqueraded addres, which could be a problem sinze I guess this
could be something like xx.147.60.99
 
Old 01-14-2016, 06:42 PM   #7
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by tlowk View Post
I see this rule
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE

If your laptop is not in 192.168.100.0/24 but it sends a packet to a host in that range
the reply will come from the masqueraded addres, which could be a problem sinze I guess this
could be something like xx.147.60.99
Are you suggesting I modify this rule? Add a -s for the range of my laptop?
 
Old 01-14-2016, 06:43 PM   #8
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
also, 60.99 is my laptop, 60.112 is the ovirt/gateway box.
 
Old 01-15-2016, 01:34 AM   #9
tlowk
Member
 
Registered: Nov 2003
Location: Belgium
Distribution: Slackware
Posts: 184

Rep: Reputation: 36
I would insert a rule before the general masquerading that makes an exception for your laptop

-A POSTROUTING -s 192.168.100.0/24 -d xx.147.60.99 -j ACCEPT // the new rule as an exception for the laptop
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE // the existing rule
 
Old 01-15-2016, 06:05 AM   #10
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by tlowk View Post
I would insert a rule before the general masquerading that makes an exception for your laptop

-A POSTROUTING -s 192.168.100.0/24 -d xx.147.60.99 -j ACCEPT // the new rule as an exception for the laptop
-A POSTROUTING -s 192.168.100.0/24 ! -d 192.168.100.0/24 -j MASQUERADE // the existing rule
I'll give it a try. Thanks!
 
Old 01-15-2016, 07:55 AM   #11
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Gangrif View Post
I'll give it a try. Thanks!
This didn't help. Same behavior.
 
Old 01-15-2016, 08:19 AM   #12
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
I've done a little reading on iptables, and the order in which a packet traverses its tables/chains.

http://www.faqs.org/docs/iptables/tr...goftables.html

It looks like, in the case of a forwarded packet (which this qualifies as), we hit mangle/prerouting -> nat/prerouting, then the kernel makes a routing decision, and hands the packet back to iptables at mangle/forward -> filter/forward, then mangle/postrouting -> nat/postrouting.

So, I added log entries at each step in that process.

My logs look like this:


Packet hits PREROUTING with an input intarface of ovirtmgmt, which is proper. thats where 60.112 is set. It's the input interface. Source of my workstation, dest of the private ip of the vm.
Code:
Jan 15 09:10:00 gollum kernel: PreRouting: IN=ovirtmgmt OUT= MAC=bc:30:5b:e0:2f:1b:38:c9:86:04:8e:0a:08:00 SRC=xx.147.60.99 DST=192.168.100.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=42653 DF PROTO=ICMP TYPE=8 CODE=0 ID=6243 SEQ=1
Here is where the kernel makes a routing decision on where to send the packet. If you do a packet capture, you see a REDIRECT message here, sending the packet back to the default route of ovirtmgmt, which tells me that this system says, I don't have a static route for this packet, get it out of here.

Here, iptables backs me up, and says its sending the packet which came in ovirtmgmt, out ovirtmgmt, which is how it would get to the default public route.

Code:
Jan 15 09:10:00 gollum kernel: Forward: IN=ovirtmgmt OUT=ovirtmgmt MAC=bc:30:5b:e0:2f:1b:38:c9:86:04:8e:0a:08:00 SRC=xx.147.60.99 DST=192.168.100.10 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=42653 DF PROTO=ICMP TYPE=8 CODE=0 ID=6243 SEQ=1
And then postrouting shows it leaving through ovirtmgmt.

Code:
Jan 15 09:10:00 gollum kernel: PostRouting: IN= OUT=ovirtmgmt SRC=xx.147.60.99 DST=192.168.100.10 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=42653 DF PROTO=ICMP TYPE=8 CODE=0 ID=6243 SEQ=1

I do not belive this is an iptables problem. Unless iptables is supposed to be doing something before that routing decision, that informs the kernel that it's supposed to send it elsewhere.

IP routing should be able to occur without iptables at all, unless I misunderstand, so I doubt that's the case.
 
Old 01-15-2016, 08:41 AM   #13
Gangrif
Member
 
Registered: Feb 2004
Distribution: Fedora 19
Posts: 73

Original Poster
Rep: Reputation: 15
If i outright disable iptables, and try to ping my 192 address, i get the same redirect message.


Code:
[lagern@starwind ~]$ ping 192.168.100.10
PING 192.168.100.10 (192.168.100.10) 56(84) bytes of data.
From xx.147.60.112: icmp_seq=1 Redirect Host(New nexthop: xx.147.60.1)
From xx.147.1.50 icmp_seq=1 Time to live exceeded
From xx.147.60.112: icmp_seq=2 Redirect Host(New nexthop: xx.147.60.1)
From xx.147.1.50 icmp_seq=2 Time to live exceeded
From xx.147.60.112: icmp_seq=3 Redirect Host(New nexthop: xx.147.60.1)
Of course nat is now broken, but I'll deal with that once the routing thing is worked out.


Code:
[root@gollum ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         xx.147.60.1    0.0.0.0         UG        0 0          0 ovirtmgmt
xx.147.60.0    0.0.0.0         255.255.255.128 U         0 0          0 ovirtmgmt
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 enp8s0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ovirtmgmt
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 Evil
192.168.100.0   0.0.0.0         255.255.255.0   U         0 0          0 Evil
The ovirt box is clearly configured to route 192.168.100.0/24 through the proper bridge interface.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Router... gateway... VPN need some help with setting up a linux router i think.. crehop Linux - Newbie 1 01-12-2016 06:54 PM
Setting up Linux box as IPv6 router to replace Netgear WNR1000 wireless router samcan Linux - Networking 1 06-19-2011 05:30 AM
Adding new Linux firewall/router on network with pre-existing gateway/router grittyminder Linux - Networking 4 08-13-2008 02:17 AM
Small Linux Router/firewall behind D-Link Hardware router dleidlein Linux - Networking 6 04-30-2007 05:12 AM
Linux Router & Netgear Wireless Router DMaCATO Linux - Wireless Networking 1 04-30-2004 09:16 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 07:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration