rsyslog splits messages from remote hosts

lpwevers · 05-12-2020, 04:43 AM

Hello,

I've run into a peculiar issue with a rsyslog server. I've configured it to accept messages coming in from remote hosts:

/etc/rsyslog.d/remote.conf

Code:

$template RemoteLogs,"/var/log/hosts/%SOURCE%/%syslogfacility-text%"
if ($fromhost-ip != "127.0.0.1" ) then ?RemoteLogs
& ~

That works fine and I see the messages nicely stored in directories per host. However; some of the hosts logging to this server are VMWare ESX servers and sometimes they do send very large messages. And it looks like rsyslog is splitting these messages.

So, for example, I have a host called vm-hrl-110.lokaal. And when it sends a large message it gets split. The first part is stored in /var/log/hosts/vm-hrl-110.lokaal/local4:

Code:

2020-05-12T08:37:57.032Z vm-hrl-110.lokaal Vpxa: verbose vpxa[28EF5B70] [Originator@6876 sub=Default opID=HB-host-42@357270-4d3fc1b1-7b] [UrlHelper::SetDatastores] /vmfs/volumes/57dbd360-9af8978e-7443-0894ef1eeb0f/ /vmfs/volumes/56d84fd9-1285ca8a-6694-842b2b191712/ /vmfs/volumes/5b1e6dc8-3d78ebf7-ba6b-3440b5e13988/ /vmfs/volumes/57d116f6-b10ff628-23a1-842b2b191712/ /vmfs/volumes/57f39134-9f4d2a1d-8055-842b2b191712/ /vmfs/volumes/571748bc-9595313d-50c4-842b2b191712/ /vmfs/volumes/5dc52902-65d09e13-29ef-3440b5e08480/ /vmfs/volumes/57e3cdec-648f33eb-404a-0894ef1ef307/ /vmfs/volumes/57a3259d-4194da41-1790-842b2b191712/ /vmfs/volumes/5a719897-b05793ac-4f80-0894ef1ef307/ /vmfs/volumes/57d7c4ad-63f8a79a-bd6c-3440b5e151e0/ /vmfs/volumes/57a328b6-3219fe96-4062-842b2b191712/ /vmfs/volumes/57d1171e-f703e27f-92a0-842b2b66a108/ /vmfs/volumes/57ab9de3-a416f480-8878-0894ef1ef307/ /vmfs/volumes/58

And the second part gets stored in /var/log/hosts/fde011-f7f38a63-4ff5-089/local4 and looks like this:

Code:

2020-05-12T10:37:57.057389+02:00 fde011-f7f38a63-4ff5-089 vm-hrl-110.lokaal Vpxa: ef1ef307/ /vmfs/volumes/57f391b4-c978f350-9ce1-0894ef1ef307/ /vmfs/volumes/5d25a51e-50017cb5-34cd-0894ef1ef307/ /vmfs/volumes/57f39374-62d54dd0-2bba-0894ef1ef307/ /vmfs/volumes/57a0acf1-ec1f8c97-9c2c-842b2b191712/ /vmfs/volumes/56ebda92-61e77a52-bfca-3440b5e151e0/ /vmfs/volumes/57d909f5-2266555a-9702-842b2b191712/ /vmfs/volumes/5b191396-3810fb85-c020-0894ef1ef307/ /vmfs/volumes/5809e046-317558a3-3ebd-842b2b66a7db/ /vmfs/volumes/57dbd323-d8c18e08-8622-0894ef1ef307/

Looking at this second message, I see why it's put in the wrong directory, as it has fde011-f7f38a63-4ff5-089 on the second field, where one would expect the hostname; the hostname is on the third field.

Also (and that's why I believe the splitting is done by the rsyslog server), notice the timestamp on both messages. There's a 2 hour difference between them. Now the VMWare server is using UTC for timezone and the rsyslog server uses CET. And now with daylight saving time, that explains the two hour difference. As the time is translated to the local time of the rsyslog server time, I believe it's rsyslog that does the splitting; not the sending VMWare server.

Now I don't want messages to split like this, instead I'd like to see the whole message go into the correct logfile as one line.

Has anyone an idea on how to fix this?

Kind regards,
Louis

smallpond · 05-12-2020, 11:56 AM

Probably too big for a UDP packet. Try switching to TCP.

sgrlscz · 05-12-2020, 12:15 PM

Try setting the $MaxMessageSize. Depending on the rsyslog or OS settings, it may have a default setting that is too small. I think it can be set up to 64k.

According to the rsyslog documentation, 4k seems to be the typical max for UDP connections. With TCP, you don't have the same UDP stack limitations.

You'd want to set the $MaxMessageSize right at the top of the configuration so it is defined before any input is defined.

lpwevers · 05-13-2020, 01:30 AM

I've switched to TCP from UDP, but alas. Also I've set $MaxMessageSize to 64k but still no luck. Messages are still split up.

So I tried writinge a template that would rewrite these messages to use the third field as hostname, and that does seem to work. Looks something like this:

Code:

template(name="VMWare" type="list") {
          property(name="timereported")
          constant(value=" ")
          property(name="msg" position.from="1" position.to="10")
          constant(value=" ")
          property(name="msg" position.from="18")
}

if re_match ($hostname, "[a-z0-9]*-[a-z0-9]*-[a-z0-9]*-[a-z0-9]*") then
{
  ?VMWare
  ~
}

But that's a solution I'm not too happy about. If the message format changes for some reason, I'll have to adapt again.

Ah well, for now it works. Thanks for the help.

Kind regards,
Louis.