Piercing NAT with remote rsyslog server
 

I'm tinkering with a remote logging server. Debian 10 and rsyslog. I'm looking for ideas to distinguish devices behind NAT.

I appreciate this is not really an rsyslog issue but a router issue, which is where the NAT happens. The routers are not traditional Linux systems (Mikrotik).

I don't see a way to distinguish devices at the router. Hence I'm trying to find clever ways to distinguish at the logging server. I read a little about relays and property replacers, but don't know if that might help.

We want one log per device. We have more than 1500 devices and everything is behind a NAT. These are embedded devices. While many are Linux based to some degree, all of the devices are proprietary and not traditional Linux systems.

I configured templates to use %FROMHOST-IP% in the log name. I use specific ports to store the logs by device role (AP, backhaul, router, etc.).

So far so good but rather than a log name with a private IP address the log name is the NAT IP address.

Using hostname as a filter won't help because many of the devices do not use standard syslog or bsd file format.

I've been searching the web to no avail, but I am not an rsyslog wizard and might be missing something.

Thanks for any ideas! :)

Can you use originating IP or MAC address? If not, please get real and put a relay behind the NAT. Let this relay log stuff and relay it to you.

In fact all you have to do is get your logging box behind the NAT.

Did you try adding '$PreserveFQDN on' in the rsyslog.conf of the originating servers?

"Get real?"

Not possible.

I did not design this network and the design is not changeable by me.

All devices go through a Mikrotik router, which is where the NAT is configured. Mikrotik logging is rudimentary at best. The devices support forwarding to an external syslog server, but to my understanding, Mikrotik RouterOS does not provide a syslog like service that could act as a relay.

I have not. Would not help because the name servers are not configured to manage private IP addresses.

These are embedded devices on private IPs. Like so many embedded devices, the design is myopic. While the devices have assigned "device names," the device names are not true hostnames. Many of the devices do not use the "device name" in their own logs and do not support standard syslog format.

I'm running out of ideas. I would scrutinize info through the router (perhaps with nmap, wireshark or other tools) to see what info comes out. If you don't get an originating IP or a MAC address you can use, What you seek is not possible, unless someone other than me provides a solution.

Can you get the originating boxes to forward the logging info with their identity? Can you connect yourself into the router, even remotely, on a nic or something, thereby putting yourself behind the NAT? If not, you'll have to present your boss with a list of options along the lines:

"To do this I will need either a) or b) or c)" and let him figure it out. If you're not given a, b, or c then he's not serious about logging.

I stumbled across the terms NAT traversal and NAT hole punching. I have not looked into whether Mikrotik devices can be configured to support some kind of traversal. Even if possible there are too many devices and too many routers to reconfigure. Perhaps I need to configure the logging server differently or configure multiple logging servers as relays where I can translate the NAT address back to private. Or perhaps multiple instances of rsyslog.

NAT or Network Address Translation undoes what you need to log in. If you could get even a raspberry Pi behind the Nat, it could be your relay. But if you can't access the boxes behind the NAT, you can't log them. I would fight shy of some half-assed solution that isn't a good engineering solution to the problem. It will be a rope around your neck until you run away, or crack up. If people are tying your hands, they're not serious about logging. Tell them as much. Tell them what you need to do the job and sit back.

I don't know why everything needs to be NATed. Perhaps there is a good reason. Often this design has confused me.

One silver lining is there is only one router per tower. Because there is only such device on that NAT address, some if-then tests using $fromhost-ip would create a log name based on the preferred private IP address. That would be about 70 if-then tests but doable.

Otherwise everything upstream of the tower routers -- the APs and CPEs -- are NATed. I've updated the company KB that the project is mostly not doable because of NAT. I'm basically in shoulder shrug mode. :)

If you can find a router or switch that doesn't do NAT, then it's doable. If they're all connected by cell phone or wifi, that gets messy. If you're wired, get a switch and it gets easy. If wifi, there are setups where many wifi access points link to a single router (e.g. in hospitals, universities). You can walk around the place from access point to access point and keep your connection. So surely, all those access points are not doing NAT?

That approach means you have to do logging, routing, NAT, & Network security, and the link from the access point to you looks particularly insecure to me… If people are telling you what you can't do, let them go away.