Hi Everyone,

We have configured all the clients to forward it's log messages to remote syslog server. It is traversing over TCP port 514 and we do have certificates generated for each client. 90% of servers are sending logs perfect, but very few servers suddenly stops sending to remote syslog server.

We get following error messages while debugging,

unexpected GnuTLS error -9 in nsd_gtls.c:1653: A TLS packet with unexpected length was received. --> This error message in Oracle Enterprise Linux 6
unexpected GnuTLS error -110 in nsd_gtls.c:1840: The TLS connection was non-properly terminated --> This error message in Oracle Enterprise Linux 7

We are unable to identify what is causing this issue, Can anyone help me on this.

You clearly identified what the issue is, by posting the error messages. You also don't provide details about the systems involved, what the syslog server is running, network environment, what syslog server you're running, etc. Both of these errors are covered in the Oracle Linux docs, having to do with a known bug.

OEL6 is very old, and probably unsupported, but do the failing systems have all the patches/updates loaded, specifically those related to SSL? And have you contacted Oracle for support??

I have seen that, switched to UDP. No dropped connection since.

1 members found this post helpful.

Thanks for an update, Sorry for delayed update .

Yes I did checked with oracle support and they confirmed there is no bug with the current package. Yes all failing systems are running with latest patches updated.

The rsyslog setup what we have is,

2 remote syslog servers sitting behind LB, and all clients are connected to LB. Clients are forwarding log messages but after few days it somehow loses the tcp connection and we observed RST flag in tcpdump from remote server.

One thing we noticed in syslog servers I see following error "can not read file '/etc/pki/rsyslog/xxxxxxxxxxxx-pub.opc.oracleoutsourcing.com.cert", but I see it has correct permission as '-r--------. 1 root root 1939 Oct 12 04:28 omcsc'

Server Details:-

Package:-
rsyslog-gnutls-8.24.0-52.el7_8.2.x86_64
rsyslog-8.24.0-52.el7_8.2.x86_64
rsyslog-relp-8.24.0-52.el7_8.2.x86_64

OS - Oracle Linux Server release 7.8

Server config -

# RELP server on tcp/601
$ModLoad imrelp
$InputRELPServerRun 601

# TCP server on tcp/6514 (With TLS)
$ModLoad imtcp

$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca-chain.cert.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/omcscaaltncbdx-pub.opc.oracleoutsourcing.com.cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/omcscaaltncbdx-pub.opc.oracleoutsourcing.com.key.pem

$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerStreamDriverMode 1
$InputTCPMaxSessions 1024
$InputTCPServerRun 514
# $InputTCPServerRun 6514

# UDP server on udp/601
$ModLoad imudp
$UDPServerRun 514

Thanks for an update, Is UDP recommended to avoid these issues?

AGAIN, this is pointing to an SSL issue. Hard to believe Oracle support didn't check more, since that is again a known bug that has been resolved...and again, OEL 6 is ancient at this point. You mention an LB (assuming you mean load balancer), but don't provide any details about it or indicate that you've done any diagnostics on that device.
Bolded a line for emphasis only. You led us to believe you were using OEL6...now you indicate you're using OEL 7.8??? Oracle's reference guide (which they should have pointed you to, since you claim to have called them), has troubleshooting steps that cover that error.
https://support.oracle.com/knowledge...2217580_1.html

And as MadeInGermany indicated, UDP may sidestep your issues.

If the problem occurs, yes.
I haven't analyzed the problem further, but my gut feeling is there is a timing bug in rsyslogd that is triggered if the system is paused for too long. Our affected machines were all VMware guests, and slow automatic migrations pause(halt) them for many seconds.

I want to add that we had the problem without using SSL.

I suspect the sending rsyslogd, because restarting it fixed the problem.
Restarting rsyslogd on the central log server did not fix it.

Hi,

Sorry for the confusion. The earlier updates which indicates OEL6 is one of the client which pushes logs to remote server. The latest update which I shared is syslog server which is running with OEL7.

Yes, we did analysis at LB side by capturing tcpdump but not much evidence we got.

Have been asked to check at syslog server.

Thanks for the doc, will go through that and see if any changes can help us.