[SOLVED] Reply on same network interface (UDP)

SBond87 · 02-02-2018, 03:52 PM

Quote:

Originally Posted by jlinkels

Do I then understand it correctly that you access your Pi from home using nts1-e.ostfalia.de but you are not on one of the addresses between 141.41.241.65 through 141.41.241.78? You have a normal public address at home and you access the Pi from there?

Yes, I access my Pi from home using nts1-e.ostfalia.de. The domain 'nts1-e.ostfalia.de' is the IP "141.41.241.68" (https://db-ip.com/141.41.241.68). This IP allow the direct access on eth0. At home I have a normal IP address (now 77.23.214.176). So... everyone can reach my Pi.

short overview:

Code:

                                     +---------------------------------+
                                     |          Raspberry Pi           |
                                     |                                 |
                       public        |                |                |
+----------------+     access        |[NTPv4 server]  |  [NTPv4 client]|       +--------------+
| My PC at home  |     +-----+       |                |                |       |   TP-Link    |       +-----+
|(77.23.214.176) +<--->+ WWW +<----->+ eth0                       eth1 +-------+   Router     +<----->+ WWW |
+----------------+     +-----+       |(141.41.241.68)   (192.168.0.101)|       +--------------+       +-----+
                                     +---------------------------------+

jlinkels · 02-03-2018, 09:40 AM

Very clear now.

So you have to set you default gateway to 192.168.0.1 in order for you Pi to access the internet without restrictions. At the same time you have to make sure that a packet which is received on 141.41.241.68 goes out there as well.

First, change the default gateway:

Code:

auto eth0
allow-hotplug eth0
iface eth0 inet static
        address 141.41.241.68
        netmask 255.255.255.240
        network 141.41.241.64
        gateway 141.41.241.65 #<< remove this one 
        broadcast 141.41.241.79
        # nameserver 8.8.8.8
	nameserver 141.41.1.150 141.41.1.250 8.8.8.8  --> added research DNS server (but no success)


#This interface only allows local access from 192.168.0.xxx
#Check if 192.168.0.101 is available for you
auto eth1
allow-hotplug eth1
iface eth1 inet static
        address 192.168.0.101
        netmask 255.255.255.0
        network 192.168.0.0
        gateway 192.168.0.1 #<< add this one 
        broadcast 192.168.0.255

Leave the other lines in /etc/network interfaces as is.

Now you have to add a route which send back the packet on eth0 if they were received at eth0. This article specifies how: https://unix.stackexchange.com/quest...ce-as-incoming.

I remember I did this hundred years ago, and I remember eventually I got it working. But I do not remember the details, so that is why I send you this article and leave the details to you. And I cannot try it because all my Pi-s are still the old model with 1 NIC.

Needless to say you should try this only when you have physical access to the Pi. As long as you do not change things on eth1 (192.168.0.xxx) and you do not set any custom routes on it, it is unlikley you will get locked out. If things really get sour on, take the SD card out, mount it on another computer and edit /etc/iproute2/rt_tables. Maybe it is even a good idea to backup this directory.

Once you get it to work, you can proceed and get DHCP addresses again. But I think really you only have to set DHCP client addresses on eth1 and wlan0. If you have a DHCP server somewhere on the LAN. You don't need dhcpd on the Pi. Unless you want to make the Pi DHCP server for other hosts on the LAN

jlinkels

SBond87 · 02-04-2018, 06:54 AM

thanks

I try it on monday and post the results

SBond

SBond87 · 02-05-2018, 05:58 AM

my steps:

I disabled dhcpd to avoid side effects (if they exist. I don't know):

Code:

systemctl stop dhcpcd
systemctl disable dhcpcd

The file '/etc/dhcpcd.conf' is clear (no content). Next I changed '/etc/network/interfaces':

Code:

auto eth0
allow-hotplug eth0
iface eth0 inet static
        address 141.41.241.68
        netmask 255.255.255.240
        network 141.41.241.64
        broadcast 141.41.241.79
        nameserver 141.41.1.150 141.41.1.250 8.8.8.8


#This interface only allows local access from 192.168.0.xxx
#Check if 192.168.0.101 is available for you
auto eth1
allow-hotplug eth1
iface eth1 inet static
        address 192.168.0.101
        netmask 255.255.255.0
        network 192.168.0.0
        gateway 192.168.0.1
        broadcast 192.168.0.255

...and reboot...

result:

SSH access over local network (eth1): works fine
Internet on the Raspberry Pi (eth1): doesn't work
Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): doesn't work
Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): doesn't work
Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): doesn't work

route -n

Code:

Destination     Router          Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth1
141.41.241.64   0.0.0.0         255.255.255.240 U     0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

ifconfig

Code:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 141.41.241.68  netmask 255.255.255.240  broadcast 141.41.241.79
        inet6 fe80::ba27:ebff:fe54:8b92  prefixlen 64  scopeid 0x20<link>
        [...]

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.101  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::224:9bff:fe1b:4947  prefixlen 64  scopeid 0x20<link>
        [...]

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        [...]

ok, looks fine. IPs are right.
Next, I defined the routing table to reply incoming packets on same network interface...
(from: https://unix.stackexchange.com/quest...ce-as-incoming)

Code:

echo 100 public >> /etc/iproute2/rt_tables
ip rule add from 141.41.241.68/32 table public
ip route add default via 141.41.241.65 dev eth0 table public

result:

SSH access over local network (eth1): works fine
Internet on the Raspberry Pi (eth1): doesn't work
Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): works fine
Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): works fine
Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): doesn't work

route -n

Code:

Destination     Router          Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth1
141.41.241.64   0.0.0.0         255.255.255.240 U     0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

ip route list table public

Code:

default via 141.41.241.65 dev eth0

ip rule show

Code:

0:      from all lookup local
32765:  from 141.41.241.68 lookup public
32766:  from all lookup main
32767:  from all lookup default

route show

Code:

default via 192.168.0.1 dev eth1 onlink
141.41.241.64/28 dev eth0 proto kernel scope link src 141.41.241.68
192.168.0.0/24 dev eth1 proto kernel scope link src 192.168.0.101

and 'iptraf-ng' shows:

Code:

TCP Connections (Source Host:Port)  Iface
--------------------------------------------------
141.41.241.68:80                    eth0
80.187.108.126:52083                eth0
141.41.241.68:80                    eth0
80.187.108.126:52084                eth0
141.41.241.68:80                    eth0
80.187.108.126:52085                eth0
141.41.241.68:80                    eth0
80.187.108.126:52086                eth0
141.41.241.68:80                    eth0
80.187.108.126:52087                eth0

UDP Connections
--------------------------------------------------
UDP (76 bytes) from 80.187.108.126:28599 to 141.41.241.68:123 on eth0
UDP (76 bytes) from 192.168.0.101:123 to 80.187.108.126:28599 on eth1

....so... same problem. TCP works, UDP fails (response going over the wrong interface again)

for SSH access over eth0 (after reboot) I changed /etc/network/interfaces again:

Code:

auto eth0
allow-hotplug eth0
iface eth0 inet static
        address 141.41.241.68
        netmask 255.255.255.240
        network 141.41.241.64
        broadcast 141.41.241.79
        nameserver 141.41.1.150 141.41.1.250 8.8.8.8


#This interface only allows local access from 192.168.0.xxx
#Check if 192.168.0.101 is available for you
auto eth1
allow-hotplug eth1
iface eth1 inet static
        address 192.168.0.101
        netmask 255.255.255.0
        network 192.168.0.0
        gateway 192.168.0.1
        broadcast 192.168.0.255
        post-up ip rule add from 141.41.241.68/32 table public #added to make this rule persistent
        post-up ip route add default via 141.41.241.65 dev eth0 table public #added to make this rule persistent

oh man... this is really tricky

jlinkels · 02-05-2018, 06:42 PM

I wonder why internet access fails over eth1. It should work. Is your internet gateway on the LAN really 192.168.0.1? Maybe the name servers should be mentioned under eth1 instead of eth0. Or try

Code:

traceroute -n 8.8.8.8

and see how far it comes.

And it looks like your situation is the same as 43 posts ago. NTP replies still do not go out on eth0 if they were received there.

See, UDP is a connectionless protocol, as opposed to TCP. So a UDP packet comes in, is routed to the application which handles it, and the application composes the reply. And the reply is sent to the originating address. I am beginning to think (and fear) that because UDP is a connectionless protocol, it is not tracked where the UDP packet was received. And the application (NTP) simply retrieves the originating address from the UDP message and composes a complete new reply message and delivers that to the IP stack. And the IP stack only knows one thing: it should use the route which is know to be able to reach the internet. Which is through eth1.

And as soon as you set the default route through eth0, you are not able to reach the internet through eth1. Which is needed because eth0 has very limited functionality.

That is where I am stuck now, and maybe you as well.

jlinkels

SBond87 · 02-06-2018, 10:27 AM

Quote:

Originally Posted by jlinkels

Is your internet gateway on the LAN really 192.168.0.1?

yes, I'm sure

Quote:

Originally Posted by jlinkels

Maybe the name servers should be mentioned under eth1 instead of eth0. Or try

Code:

traceroute -n 8.8.8.8

and see how far it comes.

this is the result:

Code:

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.0.101  44.834 ms !H  44.771 ms !H  44.736 ms !H

Quote:

Originally Posted by jlinkels

See, UDP is a connectionless protocol, as opposed to TCP. So a UDP packet comes in, is routed to the application which handles it, and the application composes the reply. And the reply is sent to the originating address. I am beginning to think (and fear) that because UDP is a connectionless protocol, it is not tracked where the UDP packet was received.

yes, I've already thought of that and I'm afraid it really is because of it.
A way to find it out is a simple tcp/udp client and server. I can write a small test program in c++, but I hoped it is not necessary.
Such a small problem is causing a great headache.

Maybe I find another way or a workaround for that. I ask a person in my university and I have Internet connectivity (incomming/outgoing) on port 80 (HTTP) and 443 (TLS). I only used TCP on port 80 for my web server (http://nts1-e.ostfalia.de/). Port 443 is unused. It looks like the NTP client have a random port at start (unlike the server with the fixed port 123). Maybe I can redirect the NTP traffic to the port 443. ...but a clear solution is better.

..is my problem really that unique?

regards,
SBond

jlinkels · 02-06-2018, 06:20 PM

It is weird that internet connectivity on eth1 is gone. The static route looks right. You might want to disable eth0 and remove the rule public from the ip rules to see at what point it goes wrong.

For NTP traffic the originating port in the client is random. The destination port in the server is always 123. It cannot change because you have to address a known port.

If you have to make test setups, check nc (or netcat) it might save you work.

Your problem is unique and then it is not. It is unusual to have set the default gw on one interface and have the internet connection (that is for you NTP clients from the internet) on another interface. But the world is full of NTP servers and DNS servers which use UDP. And these servers can be really large and must have multiple NICs for load sharing. I have no idea how that works, but there mist be some mechanism behind. And it must be documented.

Also there could be something in IPtables where a packet receives a mark when it passes through. Even it is UDP, the IP packet will contain not only the source address of the remote client, but also the remote port of that client. A unique combination. Maybe something can be done with that. IPtables is present on the Pi because it is part of the kernel. But it is complicated.

jlinkels

SBond87 · 02-07-2018, 10:27 AM

I'll meet someone tomorrow. Maybe I get additional information. I'll let you know if I know more.

SBond87 · 02-13-2018, 08:32 AM

Hello,

um... at first... I correct my last post. The command traceroute is functional, but my cable on eth1 was disconnectet (only at the last test).
here is the result:

traceroute -n 8.8.8.8

Code:

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.0.1  0.689 ms  0.540 ms  0.475 ms
 2  141.41.39.2  1.222 ms  1.430 ms  1.367 ms
 3  141.41.238.249  0.723 ms  0.665 ms  0.604 ms
 4  141.41.238.245  0.528 ms  0.768 ms  0.705 ms
 5  141.41.238.235  1.232 ms  0.739 ms  1.086 ms
 6  * * *
[...]

The IP 141.41.238.249 looks very similar to the address 141.41.241.68 (eth0), but as you see the networks are different (this is correct). But same problems as before.
I asked a friend and tried another configuration:

with ip route add 192.53.102.0/24 via 192.168.0.1 dev eth1 metric 100

route -n

Code:

Ziel            Router          Genmask         Flags Metric Ref    Use Iface
0.0.0.0         141.41.241.65   0.0.0.0         UG    202    0        0 eth0
0.0.0.0         192.168.0.1     0.0.0.0         UG    204    0        0 eth1
141.41.241.64   0.0.0.0         255.255.255.240 U     202    0        0 eth0
192.53.102.0    192.168.0.1     255.255.255.0   UG    100    0        0 eth1
192.168.0.0     0.0.0.0         255.255.255.0   U     204    0        0 eth1

OK, we added '192.53.103.0'. My NTP client uses 192.53.103.104 as external time server and the hope was we can route the traffic to eth1.

ip rule show

Code:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

ip route show

Code:

default via 141.41.241.65 dev eth0 src 141.41.241.68 metric 202
default via 192.168.0.1 dev eth1 src 192.168.0.101 metric 204
141.41.241.64/28 dev eth0 proto kernel scope link src 141.41.241.68 metric 202
192.53.102.0/24 via 192.168.0.1 dev eth1 metric 100
192.168.0.0/24 dev eth1 proto kernel scope link src 192.168.0.101 metric 204

traceroute 192.53.102.104

Code:

traceroute to 192.53.102.104 (192.53.102.104), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  0.508 ms  0.556 ms  0.494 ms
 2  141.41.39.2 (141.41.39.2)  1.558 ms  1.502 ms  1.445 ms
 3  141.41.238.249 (141.41.238.249)  0.823 ms  0.768 ms  0.709 ms
 4  141.41.238.245 (141.41.238.245)  0.649 ms  0.756 ms  0.699 ms
 5  141.41.238.235 (141.41.238.235)  1.029 ms  2.266 ms  3.638 ms

traceroute shows the working route table. My interface eth1 is correct.

During traceroute , tracert indicates this:

Code:

UDP (60 bytes) from 192.168.0.101:36971 to 192.53.102.104:33473 on eth1
UDP (60 bytes) from 192.168.0.101:37883 to 192.53.102.104:33474 on eth1
UDP (60 bytes) from 192.168.0.101:38563 to 192.53.102.104:33475 on eth1
UDP (60 bytes) from 192.168.0.101:56203 to 192.53.102.104:33476 on eth1
UDP (60 bytes) from 192.168.0.101:46649 to 192.53.102.104:33477 on eth1

very good. Correct interface.

....but if I start my NTP client, the requests leave the wrong interface... ...again

Code:

UDP (76 bytes) from 141.41.241.68:58799 to 192.53.103.104:123 on eth0
UDP (76 bytes) from 141.41.241.68:58799 to 192.53.103.104:123 on eth0

I give up

Thanks for your help

SBond87 · 02-13-2018, 08:35 AM

=============================================================================================

BREAK

geeeez. Now I found a typo in my IP configuration...

=============================================================================================

I changed

Code:

ip route add 192.53.102.0/24 via 192.168.0.1 dev eth1 metric 100

to

Code:

ip route add 192.53.103.0/24 via 192.168.0.1 dev eth1 metric 100

now my NTP client and server work

thanks for your help jlinkels

best regards,
SBond

jlinkels · 02-14-2018, 09:48 AM

Hey that is great. So eventually it did work with setting 2 default routes and adjusting the metric?

Anyway, the problem with these kind of questions is that when you sit in front of your computer you can try and test 83 different options in an hour. Discussing directions and test results in the forum takes 82 days.

Can you please mark the thread as solved? Thread Tools somewhere at the top of the thread.

Happy NTP-ing

jlinkels