LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Red Hat (https://www.linuxquestions.org/questions/red-hat-31/)
-   -   sshd hangs - rsyslog related? (https://www.linuxquestions.org/questions/red-hat-31/sshd-hangs-rsyslog-related-4175550947/)

davezap 08-17-2015 04:24 AM

sshd hangs - rsyslog related?
 
Hi again.. I have a worse problem. (CentOS release 6.7)

Several hours after restart (some times over night) I am not able to login remotely via ssh. I can login from the terminal OK.

After restarting a whole bunch of services I finally found the only reliable way to get it running again was.

Code:

# service rsyslog restart
When I run that from the console I immediately get a whole bunch of output relating to aide and modules? I suppose these apps had been waiting for rsyslog.

This morning it happened again while I was logged in remotely. I notice the ssh session became very sluggish, but top did not show any heavy CPU load. At this point I could not open a second SSH session it would just hang forever and not prompt for user name.

time started around 0930
Code:

[root@ANT2 log]# date
Mon Aug 17 09:49:13 BST 2015

/var/log/secure
Code:

Aug 16 19:12:28 ANT2 sshd[14778]: debug3: Wrote 64 bytes for a total of 14621
Aug 16 19:12:28 ANT2 sshd[14778]: debug3: Wrote 80 bytes for a total of 14701

Note most logs are from yesterday.

/var/log/messages : Seems to only be logging that I restart rsyslog and nothing else? Aside from the mouse that keep getting remounted all the time?

Code:

Aug 16 19:11:21 ANT2 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Aug 16 19:11:21 ANT2 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="14811" x-info="http://www.rsyslog.com"] start
Aug 16 19:12:14 ANT2 kernel: usb 3-4: USB disconnect, device number 55
Aug 16 19:12:15 ANT2 kernel: usb 3-4: new low speed USB device number 56 using xhci_hcd
Aug 16 19:12:16 ANT2 kernel: usb 3-4: New USB device found, idVendor=17ef, idProduct=6019
Aug 16 19:12:16 ANT2 kernel: usb 3-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Aug 16 19:12:16 ANT2 kernel: usb 3-4: Product: Lenovo USB Optical Mouse
Aug 16 19:12:16 ANT2 kernel: usb 3-4: Manufacturer: PixArt
Aug 16 19:12:16 ANT2 kernel: usb 3-4: configuration #1 chosen from 1 choice
Aug 16 19:12:16 ANT2 kernel: usb 3-4: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
Aug 16 19:12:16 ANT2 kernel: input: PixArt Lenovo USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/input/input1296
Aug 16 19:12:16 ANT2 kernel: generic-usb 0003:17EF:6019.050F: input,hidraw2: USB HID v1.11 Mouse [PixArt Lenovo USB Optical Mouse] on usb-0000:00$
Aug 16 19:12:28 ANT2 kernel: Kernel logging (proc) stopped.
Aug 16 19:12:28 ANT2 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="14811" x-info="http://www.rsyslog.com"] exiting on signal 15.
Aug 16 19:12:28 ANT2 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Aug 16 19:12:28 ANT2 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="14864" x-info="http://www.rsyslog.com"] start
Aug 17 07:47:20 ANT2 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Aug 17 07:47:20 ANT2 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="20937" x-info="http://www.rsyslog.com"] start
Aug 17 09:47:19 ANT2 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Aug 17 09:47:19 ANT2 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="6359" x-info="http://www.rsyslog.com"] start
Aug 17 09:55:22 ANT2 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Aug 17 09:55:22 ANT2 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1842" x-info="http://www.rsyslog.com"] start

boot.log : shows an error
Code:

Starting auditd: ^[[60G[^[[0;32m  OK  ^[[0;39m]

key option needs a watch or syscall given prior to it
There was an error in line 194 of /etc/audit/audit.rules
Starting portreserve: (not starting, no services registered)
Starting system logger: ^[[60G[^[[0;32m  OK  ^[[0;39m]

line 194 of audit.rules is
Code:

-a always,exit -F arch=b32 -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access


I have tried reinstalling rsyslog and turning off UseDNS in sshd_config but has had no effect.

A few similar reports online mentioned this may be an unrelated service but I lost the link :(

Thanks.

davezap 08-18-2015 03:57 AM

Default forwarding rule may have been the issue.

/etc/rsyslog.conf
Code:

*.* @@192.168.1.56
This host does not exist on my network. So just commented this, restarted rsyslog and everything seems fine so far.

Logging to /var/log/secure and /var/log/messages looks normal again.

Code:

#*.* @@192.168.1.56
This post gave some clues, as well as reading the documentation helped.
http://b.kl3in.com/2011/10/ubuntu-se...ps-responding/


Will mark resolved once it's been running for 24 hours.


All times are GMT -5. The time now is 02:49 AM.