Clients suddenly stops forwarding log messages to remote rsyslog server
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Clients suddenly stops forwarding log messages to remote rsyslog server
Hi Everyone,
We have configured all the clients to forward it's log messages to remote syslog server. It is traversing over TCP port 514 and we do have certificates generated for each client. 90% of servers are sending logs perfect, but very few servers suddenly stops sending to remote syslog server.
We get following error messages while debugging,
unexpected GnuTLS error -9 in nsd_gtls.c:1653: A TLS packet with unexpected length was received. --> This error message in Oracle Enterprise Linux 6
unexpected GnuTLS error -110 in nsd_gtls.c:1840: The TLS connection was non-properly terminated --> This error message in Oracle Enterprise Linux 7
We are unable to identify what is causing this issue, Can anyone help me on this.
Hi Everyone,
We have configured all the clients to forward it's log messages to remote syslog server. It is traversing over TCP port 514 and we do have certificates generated for each client. 90% of servers are sending logs perfect, but very few servers suddenly stops sending to remote syslog server.
We get following error messages while debugging,
unexpected GnuTLS error -9 in nsd_gtls.c:1653: A TLS packet with unexpected length was received. --> This error message in Oracle Enterprise Linux 6
unexpected GnuTLS error -110 in nsd_gtls.c:1840: The TLS connection was non-properly terminated --> This error message in Oracle Enterprise Linux 7
We are unable to identify what is causing this issue, Can anyone help me on this.
You clearly identified what the issue is, by posting the error messages. You also don't provide details about the systems involved, what the syslog server is running, network environment, what syslog server you're running, etc. Both of these errors are covered in the Oracle Linux docs, having to do with a known bug.
OEL6 is very old, and probably unsupported, but do the failing systems have all the patches/updates loaded, specifically those related to SSL? And have you contacted Oracle for support??
You clearly identified what the issue is, by posting the error messages. You also don't provide details about the systems involved, what the syslog server is running, network environment, what syslog server you're running, etc. Both of these errors are covered in the Oracle Linux docs, having to do with a known bug.
OEL6 is very old, and probably unsupported, but do the failing systems have all the patches/updates loaded, specifically those related to SSL? And have you contacted Oracle for support??
Thanks for an update, Sorry for delayed update .
Yes I did checked with oracle support and they confirmed there is no bug with the current package. Yes all failing systems are running with latest patches updated.
The rsyslog setup what we have is,
2 remote syslog servers sitting behind LB, and all clients are connected to LB. Clients are forwarding log messages but after few days it somehow loses the tcp connection and we observed RST flag in tcpdump from remote server.
One thing we noticed in syslog servers I see following error "can not read file '/etc/pki/rsyslog/xxxxxxxxxxxx-pub.opc.oracleoutsourcing.com.cert", but I see it has correct permission as '-r--------. 1 root root 1939 Oct 12 04:28 omcsc'
Thanks for an update, Sorry for delayed update . Yes I did checked with oracle support and they confirmed there is no bug with the current package. Yes all failing systems are running with latest patches updated.
The rsyslog setup what we have is, 2 remote syslog servers sitting behind LB, and all clients are connected to LB. Clients are forwarding log messages but after few days it somehow loses the tcp connection and we observed RST flag in tcpdump from remote server. One thing we noticed in syslog servers I see following error
Code:
can not read file '/etc/pki/rsyslog/xxxxxxxxxxxx-pub.opc.oracleoutsourcing.com.cert
but I see it has correct permission as '-r--------. 1 root root 1939 Oct 12 04:28 omcsc'
AGAIN, this is pointing to an SSL issue. Hard to believe Oracle support didn't check more, since that is again a known bug that has been resolved...and again, OEL 6 is ancient at this point. You mention an LB (assuming you mean load balancer), but don't provide any details about it or indicate that you've done any diagnostics on that device.
Quote:
Server Details:-
Code:
Package:-
rsyslog-gnutls-8.24.0-52.el7_8.2.x86_64
rsyslog-8.24.0-52.el7_8.2.x86_64
rsyslog-relp-8.24.0-52.el7_8.2.x86_64
OS - Oracle Linux Server release 7.8
Server config -
# RELP server on tcp/601
$ModLoad imrelp
$InputRELPServerRun 601
# TCP server on tcp/6514 (With TLS)
$ModLoad imtcp
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca-chain.cert.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/omcscaaltncbdx-pub.opc.oracleoutsourcing.com.cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/omcscaaltncbdx-pub.opc.oracleoutsourcing.com.key.pem
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerStreamDriverMode 1
$InputTCPMaxSessions 1024
$InputTCPServerRun 514
# $InputTCPServerRun 6514
# UDP server on udp/601
$ModLoad imudp
$UDPServerRun 514
Bolded a line for emphasis only. You led us to believe you were using OEL6...now you indicate you're using OEL 7.8??? Oracle's reference guide (which they should have pointed you to, since you claim to have called them), has troubleshooting steps that cover that error. https://support.oracle.com/knowledge...2217580_1.html
And as MadeInGermany indicated, UDP may sidestep your issues.
Thanks for an update, Is UDP recommended to avoid these issues?
If the problem occurs, yes.
I haven't analyzed the problem further, but my gut feeling is there is a timing bug in rsyslogd that is triggered if the system is paused for too long. Our affected machines were all VMware guests, and slow automatic migrations pause(halt) them for many seconds.
I want to add that we had the problem without using SSL.
I suspect the sending rsyslogd, because restarting it fixed the problem.
Restarting rsyslogd on the central log server did not fix it.
Last edited by MadeInGermany; 10-21-2020 at 08:21 AM.
Reason: text added: without SSL, suspect client
AGAIN, this is pointing to an SSL issue. Hard to believe Oracle support didn't check more, since that is again a known bug that has been resolved...and again, OEL 6 is ancient at this point. You mention an LB (assuming you mean load balancer), but don't provide any details about it or indicate that you've done any diagnostics on that device.
Bolded a line for emphasis only. You led us to believe you were using OEL6...now you indicate you're using OEL 7.8??? Oracle's reference guide (which they should have pointed you to, since you claim to have called them), has troubleshooting steps that cover that error. https://support.oracle.com/knowledge...2217580_1.html
And as MadeInGermany indicated, UDP may sidestep your issues.
Hi,
Sorry for the confusion. The earlier updates which indicates OEL6 is one of the client which pushes logs to remote server. The latest update which I shared is syslog server which is running with OEL7.
Yes, we did analysis at LB side by capturing tcpdump but not much evidence we got.
Have been asked to check at syslog server.
Thanks for the doc, will go through that and see if any changes can help us.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.