Reply on same network interface (UDP)
 

Hello together,

is it possible to reply all incoming packets/request on the same network interface?

This is my setting:
I have a headless Raspberry Pi (raspbian/debain stretch) with two network interfaces (eth0 and eth1). The first interface (eth0) uses a public IP address, which is static. This interface is intended to provide access to the Pi (time- and web server, SSH) via the Internet. The second interface (eth1) uses the Raspberry Pi for general Internet connection (perform updates, sync own time or whatever) and uses a dynamic IP via DHCP. A general Internet connectivity over eth0 is not possible, so I have to use eth1 on the Pi.

My problem is that Internet (on the Pi) and the Internet access to the Pi are not working correctly.

first configuration (/etc/dhcpcd.conf):
after reboot 'ifconfig' shows the correct IP settings:
result:

• Internet on the Raspberry Pi (eth1): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): works fine

------------------------------------------------------------------------------------------------------

Now I tried to change the metric of the interfaces, hoping that the change of prioritization is successful.

my second configuration (/etc/dhcpcd.conf):
result:

• Internet on the Raspberry Pi (eth1): works fine
• Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): doesn't work

Ok. I think all traffic going out to eth1 at default. With the tool 'iptraf-ng' I was able to see the problem:

We see:
On TCP: some connections going out to the wrong interface (eth1).
On UDP: The request from 80.187.108.126 came over eth0 and the response was sent over eth1.

------------------------------------------------------------------------------------------------------

Next, I defined the routing table to reply incoming packets on same network interface...

result:

• Internet on the Raspberry Pi (eth1): works fine
• Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): doesn't work

and 'iptraf-ng' shows:

We see:
On TCP: now it works correctly
On UDP: same problem :(

What can I do to send UDP responses over the correct interface (eth0)? I have no idea why TCP works fine but UDP fails :( Its very frustrating and I have no more ideas. I hope someone can help.

...or is it possible to bind a process (or processes) to a IP address? Maybe it avoids this behavior.


best regards,
SBond


ps: sorry for my bad english

Why do you have two static routers. A static router is a default gateway. It tells the next hop packets need to be sent to if the IP is not on the same subnet.

That makes sense for the external interface. Everything which is not on 141.41.241.68/28 should be forwarded to 141.41.241.65

But there is no need to assign a router to 192.168.0.0/24. Everything which is on this subnet will be reachable thru eth1. Without routing.

Everything which is not on 192.168.0.0/24 will go out on 141.41.241.65 which is correct.

What I think I see is that traffic coming in on eth0 is sent out on eth1 instead of eth0 because there are two default gateways. I would remove the 192.168.0.1 gw completely and start again from the basic configuration.

I don't know why the difference between UDP and TCP. May because UDP is more stateless that TCP and doesn't know there is a established connection to use for a reply.

jlinkels

oh sorry. It is an copy/paste error in my post. I removed it. I have one static router (141.41.241.65) in my configuration file. This is really weird. I try to solve this problem since 3 days :(

Weird. With the router mixup it seemed easy.

Can you try to initiate a UDP connection from the Pi and see where it is transmitted? So issue this command:
Before you give that command, in a different window monitor the traffic on eth0 or eth1:
once and then repeat:
On one of the interfaces tcpdump should give something like:
On the other interface just nothing.

When tcpdump is running you issue the netcat command. When testing this, you take out the variable that something is replied to.

jlinkels

I'm not able to test this today, but I give you tomorrow the results. Thanks for you help. :)

ok. here are the results:

command:

eth0 monitor:

eth1 monitor:
I think it looks right.


here are other connections:

iptraf-ng:

Well, my Raspberry Pi works as NTP client and NTP server (2 processes). The client connections are fine. My Raspberry Pi is sending requests to the NTP servers over the internet connection (eth1) and receive the response messages on the same interface. This is correct, because I have no regular internet connectivity on eth0. But the problem is my NTP server. If I unplug the cable from eth1, then the server works correctly and sends the response packets over eth0. But in this case my clients doesn't work anymore, because they have no more internet connection.

is it possible to bind a process to a network interface? I thought it is a small problem and easy to solve, but ...geeeez :(
I mean... it must be possible ...somehow :(


best regards,
SBond

-----
I hope my english is good enough for you

Things are not right.
is not supposed to happen. The fake NTP request should go out on eth0, not on eth1.

Please make sure eth0 and eth1 have the IP addresses as you specified before:
I am asking this because you say:
What does the output show of jlinkels

thanks for support :)

here are my current settings:

- Raspberry Pi 3 with Raspbian (Image version: 2017-11-29)
-> headless operation
-> onbord ethernet (eth0):
---> public access (static IP)
---> for Apache web server, SSH, NTP server

-> Ethernet-USB-Dongle (eth1):
---> dynamic IP (DHCP)
---> general Internet connection (RasPi --> TP-Link router --> institute network)
---> for updates and NTP client (time synchronization is important for my Pi)

-> onboard wireless (wlan0):
---> dynamic IP (DHCP)
---> fallback connection, when I locked myself out (RasPi --> TP-Link router --> institute network)
---> same connection like eth1
---> ... only local access ^^



cat /etc/dhcpcd.conf:

the include of the line 'domain_name_servers' does not help. Therefore it is commented out.

ifconfig:


cat /etc/iproute2/rt_tables:


route -n:
192.168.0.1 (eth1) has a lower metric to realize the internet connectivity (because eth0 have no general internet).

ip rule show:

ip route show:

ip route list table public:

OK, one of the problems is definitely in the routing.
The IP stack has 3 default gateways. Note that you ca have multiple NICs but there is only ONE IP stack in the entire OS.

I know you assigned metrics to the various interfaces (and that this has some effect, but I don't know if it is the intended effect)

What is the reason you have multiple connections to the public internet? You have a public IP address, that should void the need for going through your router for yet another internet connection.

I would start with this: disable the wlan, just have eth0 (public IP) and eth1 (internal LAN). Do away with metrics and all that stuff. Easiest is to implement all addresses as static. DHCP comes later.

Your routing table should look like this:Then your Pi should work as NTP client (on eth0) and NTP server (on eth1) as intended. It might not be your eventually wanted situation. But it is better to start from a simple working situation and add exceptions later.

You also say eth0 does not have general internet access. What do you mean by that? Is it connected to the public internet or not? Or is there a router in between restricting certain addresses? In that case it might be easier to route all internet traffic over eth1, and route only the access to the NTP server specifically. I think the 3 default gateways is the source of your problem.

jlinkels

Well, I'm doing research at a technical college in the field of secure time transmission and I need a own (public) NTP server for some tests. For this I got a public IP address form my univerity. The access on this address is restricted and if have only some ports (UDP/TCP) for incoming and outgoing traffic. If I connect my Pi to this ethernet port (with eth0), then I have access from the internet to my Raspberry pi (e.g. SSH). But my Pi has no general "internet connection" in this case. I mean programs cannot connect with the internet and download updates or sync with timer servers or ping a server (e.g. ping www.google.com). And this is bad, because my Raspberry Pi need a continuous time synchronization with other time servers. To do this, I connects my Pi with a local router (TP-Link) that offers a regular internet connection.

I can often help myself, but I'm not a expert for linux or network configurations. But this problem is really annoying, because it looks totaly simple.

this is my table at th beginning:

and we want this:


...um...ok, at first I changed my settings:

/etc/dhcpcd.conf: new config:

After that I rebooted my Pi and turned off the wireless connection:

ip link set wlan0 down

result:
route -n

Do you know how I can delete the second entry?

test rusults:

• Internet on the Raspberry Pi (eth1): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): works fine

Yes, remove the line
from the network interfaces and restart networking.

Don't worry about internet access. Just check if you can have the Pi as NTP server on the LAN network. I assume it is still working correctly as NTP client because Pi know how to get to the public IP of your NTP server.

If that works, the next thing we will do is to remove the defaut gw from 141.n.n.n and make a dedicated route to get to your NTP server over the public internet.

jlinkels

ok :)

changed /etc/dhcpcd.conf:

then rebooted my Pi and turned off the wireless connection (ip link set wlan0 down)

result:

route -n

SSH and NTP server are working (eth0). NTP client (own time sync for Pi) shutting down after some seconds, because no internet access. ...and therefore not able to find other public time servers

should I remove 'static routers=141.41.241.65' now? I'm not sure if I can access my Pi after this. Until next Monday I have no access to the hardware (if I lock myself out).

Don't lock yourself out.

I don't understand why you cannot reach your time server on internet. This interface 141.41.241.68/28 is created to access the NTP server over internet, isn't it?

As I understand you have this public IP address which is heavily restricted but just to reach your NTP server. If the default gw is pointing to 141.41.241.65 why can't you reach your time server? I know the connection is not good for anything else, but the Pi (the NTP client here) does not know better as to direct all traffic including NTP requests to 141.41.241.65.

You also say you don't have internet access, but you reach you Pi from the outside using this IP? You you have somehow access to 141.41.241.68?

Oh wait, while I am writing this I see something else which doesn't look right. I overlooked it first. I thought you were setting the static addressed in /etc/networking/interfaces. But you say you do that in /etc/dhcpd.conf.

Don't for the time being. Put this in your /etc/network/interfaces on the Pi. And check if I use the correct IP addresses which were assigned to you.
And see what happens now.

jlinkels

changed config (disabled entries):

nano /etc/dhcpcd.conf

Now I changed the other config file:
nano /etc/network/interfaces



sudo reboot

ifconfig



route -n


Ok, NTP server and SSH still working, but no internet.
Maybe you are a little confused about the NTP configuration. Well, my Raspberry Pi works with 2 processes as client and server simultaneously. The first process is the NTP client. This mode allow the time sync of my Pi itself. The NTP client contacts other NTP server (3 in my case) over a normal internet connection. But it doesn't work over the eth0 connection. The NTP client need eth1 to reach the needed time servers. The second process is my own NTP server. It opens a socket with the Port 123. This Port is open and the server can receive requests and send response packets on eth0. This works currently.

I can reach my Pi over 'nts1-e.ostfalia.de' over custom ports for SSH and port 123 for NTPv4. Ping does not work (blocked by IT infrastructure)

ok. what are the next steps? ...and thanks for you support and your time :)

Yes, I am confused.

Do I then understand it correctly that you access your Pi from home using nts1-e.ostfalia.de but you are not on one of the addresses between 141.41.241.65 through 141.41.241.78? You have a normal public address at home and you access the Pi from there?

It is clear that your Pi is both server and client. That is normal.

So far, with the last changes you made what I see is totally what I expect. That is the good news.

The only thing now left is how you can access eth0 from the internet, while you still have outgoing internet connections on eth1 to retrieve the time from your 3 time servers. If you can clarify the questions in the first paragraph I'll think about it. It might be over the weekend before I am back.

jlinkels

Yes, I access my Pi from home using nts1-e.ostfalia.de. The domain 'nts1-e.ostfalia.de' is the IP "141.41.241.68" (https://db-ip.com/141.41.241.68). This IP allow the direct access on eth0. At home I have a normal IP address (now 77.23.214.176). So... everyone can reach my Pi.

short overview:

Very clear now.

So you have to set you default gateway to 192.168.0.1 in order for you Pi to access the internet without restrictions. At the same time you have to make sure that a packet which is received on 141.41.241.68 goes out there as well.

First, change the default gateway:

Leave the other lines in /etc/network interfaces as is.

Now you have to add a route which send back the packet on eth0 if they were received at eth0. This article specifies how: https://unix.stackexchange.com/quest...ce-as-incoming.

I remember I did this hundred years ago, and I remember eventually I got it working. But I do not remember the details, so that is why I send you this article and leave the details to you. And I cannot try it because all my Pi-s are still the old model with 1 NIC.

Needless to say you should try this only when you have physical access to the Pi. As long as you do not change things on eth1 (192.168.0.xxx) and you do not set any custom routes on it, it is unlikley you will get locked out. If things really get sour on, take the SD card out, mount it on another computer and edit /etc/iproute2/rt_tables. Maybe it is even a good idea to backup this directory.

Once you get it to work, you can proceed and get DHCP addresses again. But I think really you only have to set DHCP client addresses on eth1 and wlan0. If you have a DHCP server somewhere on the LAN. You don't need dhcpd on the Pi. Unless you want to make the Pi DHCP server for other hosts on the LAN

jlinkels

thanks :)
I try it on monday and post the results ;)

SBond

my steps:

I disabled dhcpd to avoid side effects (if they exist. I don't know):

The file '/etc/dhcpcd.conf' is clear (no content). Next I changed '/etc/network/interfaces':

...and reboot...

result:

• SSH access over local network (eth1): works fine
• Internet on the Raspberry Pi (eth1): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): doesn't work

route -n



ifconfig

ok, looks fine. IPs are right.
Next, I defined the routing table to reply incoming packets on same network interface...
(from: https://unix.stackexchange.com/quest...ce-as-incoming)


result:

• SSH access over local network (eth1): works fine
• Internet on the Raspberry Pi (eth1): doesn't work
• Access to Raspberry Pi over Internet (via eth0) --> SSH (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> Apache web server (TCP): works fine
• Access to Raspberry Pi over Internet (via eth0) --> NTP time server (UDP): doesn't work

route -n


ip route list table public

ip rule show

route show

and 'iptraf-ng' shows:

....so... same problem. TCP works, UDP fails (response going over the wrong interface again)


for SSH access over eth0 (after reboot) I changed /etc/network/interfaces again:

oh man... this is really tricky :(

I wonder why internet access fails over eth1. It should work. Is your internet gateway on the LAN really 192.168.0.1? Maybe the name servers should be mentioned under eth1 instead of eth0. Or try and see how far it comes.

And it looks like your situation is the same as 43 posts ago. NTP replies still do not go out on eth0 if they were received there.

See, UDP is a connectionless protocol, as opposed to TCP. So a UDP packet comes in, is routed to the application which handles it, and the application composes the reply. And the reply is sent to the originating address. I am beginning to think (and fear) that because UDP is a connectionless protocol, it is not tracked where the UDP packet was received. And the application (NTP) simply retrieves the originating address from the UDP message and composes a complete new reply message and delivers that to the IP stack. And the IP stack only knows one thing: it should use the route which is know to be able to reach the internet. Which is through eth1.

And as soon as you set the default route through eth0, you are not able to reach the internet through eth1. Which is needed because eth0 has very limited functionality.

That is where I am stuck now, and maybe you as well.

jlinkels

yes, I'm sure


this is the result:


yes, I've already thought of that and I'm afraid it really is because of it.
A way to find it out is a simple tcp/udp client and server. I can write a small test program in c++, but I hoped it is not necessary.
Such a small problem is causing a great headache. :(


Maybe I find another way or a workaround for that. I ask a person in my university and I have Internet connectivity (incomming/outgoing) on port 80 (HTTP) and 443 (TLS). I only used TCP on port 80 for my web server (http://nts1-e.ostfalia.de/). Port 443 is unused. It looks like the NTP client have a random port at start (unlike the server with the fixed port 123). Maybe I can redirect the NTP traffic to the port 443. ...but a clear solution is better.

..is my problem really that unique? :(


regards,
SBond

It is weird that internet connectivity on eth1 is gone. The static route looks right. You might want to disable eth0 and remove the rule public from the ip rules to see at what point it goes wrong.

For NTP traffic the originating port in the client is random. The destination port in the server is always 123. It cannot change because you have to address a known port.

If you have to make test setups, check nc (or netcat) it might save you work.

Your problem is unique and then it is not. It is unusual to have set the default gw on one interface and have the internet connection (that is for you NTP clients from the internet) on another interface. But the world is full of NTP servers and DNS servers which use UDP. And these servers can be really large and must have multiple NICs for load sharing. I have no idea how that works, but there mist be some mechanism behind. And it must be documented.

Also there could be something in IPtables where a packet receives a mark when it passes through. Even it is UDP, the IP packet will contain not only the source address of the remote client, but also the remote port of that client. A unique combination. Maybe something can be done with that. IPtables is present on the Pi because it is part of the kernel. But it is complicated.

jlinkels

I'll meet someone tomorrow. Maybe I get additional information. I'll let you know if I know more. :)

Hello,

um... at first... I correct my last post. The command traceroute is functional, but my cable on eth1 was disconnectet (only at the last test).
here is the result:

traceroute -n 8.8.8.8

The IP 141.41.238.249 looks very similar to the address 141.41.241.68 (eth0), but as you see the networks are different (this is correct). But same problems as before.
I asked a friend and tried another configuration:

with ip route add 192.53.102.0/24 via 192.168.0.1 dev eth1 metric 100

route -n

OK, we added '192.53.103.0'. My NTP client uses 192.53.103.104 as external time server and the hope was we can route the traffic to eth1.


ip rule show


ip route show


traceroute 192.53.102.104

traceroute shows the working route table. My interface eth1 is correct.

During traceroute , tracert indicates this:

very good. Correct interface.

....but if I start my NTP client, the requests leave the wrong interface... ...again :(

I give up :(

Thanks for your help

=============================================================================================

BREAK

geeeez. Now I found a typo in my IP configuration...

=============================================================================================

I changed

to


now my NTP client and server work :D

thanks for your help jlinkels :)

best regards,
SBond

Hey that is great. So eventually it did work with setting 2 default routes and adjusting the metric?

Anyway, the problem with these kind of questions is that when you sit in front of your computer you can try and test 83 different options in an hour. Discussing directions and test results in the forum takes 82 days.

Can you please mark the thread as solved? Thread Tools somewhere at the top of the thread.

Happy NTP-ing

jlinkels