kern.log is now always empty

Lefevre · 12-10-2019, 01:06 PM

Hello,
I just do an upgrade of my debian server from Jessie to Stretch.
But I totaly forgotten to do it in a init S status.
It seems that there was no big issue during the update but I can see now that my kern.log is now always empty. If I do "logger -p kern.info "kern.info: test" for example, it is not written into kern.log.
I'm not familiar with syslog-ng and did'nt find any document that explain how it works and how to debug.
Is there anybody that could help me on this?

# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.11 (stretch)
Release: 9.11
Codename: stretch

# ps -ef | grep syslog
root 24768 1 0 19:21 ? 00:00:00 /usr/sbin/syslog-ng -F --no-caps

ondoho · 12-10-2019, 02:21 PM

What information are you looking for?
I'd search the journal; for this boot:

Code:

journalctl -b

or dmesg:

Code:

dmesg

Lefevre · 12-10-2019, 04:19 PM

Thanks for your reply ondoho.
I just would like to understand how syslog-ng works as there is no more logs written.
I suppose I broken something somewhere so I compared config files before/after but found nothing that could explain my issue.
I understood that syslog-ng is in charge to write kern.log, mail.log and others using some filter rules from system messages (syslog file ?)
I uninstalled syslog-ng (apt-get remove) and re-installed it (apt-get install) but kern.log is still empty.

Thanks anyway for journalctl command I don't know well. It shows some errors that I have to fix before all. Just to find the concerned conf file :-)
This is a first step for debug !

journalctl -b | grep syslog give:

Quote:

déc. 10 19:21:19 vps171520 systemd[1]: syslog-ng.service: Failed to set invocation ID on control group /system.slice/syslog-ng.service, ignoring: Operation not permitted
déc. 10 19:21:19 vps171520 syslog-ng[24768]: [2019-12-10T19:21:19.660636] WARNING: Default value changed for the prefix() option of systemd-journal source in syslog-ng 3.8; old_value='', new_value='.journald.'
déc. 10 19:21:39 vps171520 systemd[1]: [/lib/systemd/system/syslog-ng.service:1] Assignment outside of section. Ignoring.
déc. 10 19:21:39 vps171520 systemd[1]: syslog-ng.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
déc. 10 19:21:47 vps171520 systemd[1]: [/lib/systemd/system/syslog-ng.service:1] Missing '='.
déc. 10 19:24:43 vps171520 systemd[1]: [/lib/systemd/system/syslog-ng.service:1] Assignment outside of section. Ignoring.
déc. 10 19:24:43 vps171520 systemd[1]: syslog-ng.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
déc. 10 19:24:43 vps171520 systemd[1]: [/lib/systemd/system/syslog-ng.service:1] Assignment outside of section. Ignoring.
déc. 10 19:24:43 vps171520 systemd[1]: syslog-ng.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
déc. 10 19:24:43 vps171520 systemd[1]: [/lib/systemd/system/syslog-ng.service:1] Assignment outside of section. Ignoring.
déc. 10 19:24:43 vps171520 systemd[1]: syslog-ng.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
déc. 10 19:24:43 vps171520 systemd[1]: [/lib/systemd/system/syslog-ng.service:1] Assignment outside of section. Ignoring.
déc. 10 19:24:43 vps171520 systemd[1]: syslog-ng.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

ondoho · 12-11-2019, 01:36 AM

Quote:

Originally Posted by Lefevre

Just to find the concerned conf file :-)

The output is telling you:
/lib/systemd/system/syslog-ng.service
(BTW, please use CODE tags for code, not QUOTE tags)

My question still stands: What information do you require from kern.log etc.? Is this info not found in the systemd journal?

PS:
I still use stretch on one machine. /var/log/kern.log does exist (and apparently ufw is the only application still using it). Syslog-ng is not installed, but rsyslog is.

Lefevre · 12-11-2019, 02:46 AM

I mainly use it to get what comes from iptables. For example:

Code:

Dec  7 22:50:56 vps171520 kernel: [30191954.877696] VPSFW SSH DROP: IN=venet0 OUT= MAC= SRC=51.91.136.174 DST=176.31.171.214 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=46069 PROTO=TCP SPT=55312 DPT=22 WINDOW=1024 RES=0x00 SYN URGP=0

Thanks for the file name. I was looking for the syslog config files as I never did any change into libs directories.

There, something is strange and very probably bad. I have:

Code:

# ls -ali /etc/systemd/system/multi-user.target.wants/syslog-ng.service
lrwxrwxrwx 1 root root 37 févr.  4  2016 /etc/systemd/system/multi-user.target.wants/syslog-ng.service -> /lib/systemd/system/syslog-ng.service

# ls -ali /lib/systemd/system/syslog-ng.service
281515 -rw-r--r-- 1 root root 149 déc.  10 08:45 /lib/systemd/system/syslog-ng.service

# view /lib/systemd/system/syslog-ng.service
lrwxrwxrwx 1 root root 37 févr.  4  2016 /etc/systemd/system/multi-user.target.wants/syslog-ng.service -> /lib/systemd/system/syslog-ng.service

The file contains a line showing a link like ls do!
The 10th of december is the date where I updated my debian so something goes wrong while the update process was running.

I tried to do a comparison with my system @home. It doesn't use syslog-ng but mine was installed for a long time and followed regular updates.
I suppose ng was not used when I installed my debian.
The server where I get my trouble was first installed with Debian7 by my service provider (OVH). I updated it to debian 8 without issue.
I think syslog-ng was already there.

I wonder if I shouldn't do a total remove of syslog-ng (with config files) and reinstall it.
What do you fell?

NB: Ok for code tag instead of quote. This tag is not displayed to me, so I'm going to use it "by hand" ;-)

Lefevre · 12-11-2019, 03:37 AM

I re-installed syslog-ng.core

Code:

# apt --reinstall install syslog-ng.core
...
Préparation du dépaquetage de .../syslog-ng-core_3.8.1-10_amd64.deb ...
Dépaquetage de syslog-ng-core (3.8.1-10) sur (3.8.1-10) ...
Paramétrage de syslog-ng-core (3.8.1-10) ...
Traitement des actions différées («*triggers*») pour systemd (232-25+deb9u12)*...
Traitement des actions différées («*triggers*») pour man-db (2.7.6.1-2)*...

so now I have:

Code:

# cat /lib/systemd/system/syslog-ng.service
[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)

[Service]
Type=notify
ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS
ExecReload=/bin/kill -HUP $MAINPID
EnvironmentFile=-/etc/default/syslog-ng
EnvironmentFile=-/etc/sysconfig/syslog-ng
StandardOutput=journal
StandardError=journal
Restart=on-failure

[Install]
WantedBy=multi-user.target

Lefevre · 12-12-2019, 05:24 PM

Ok, I was not able to find were is the problem as I don't really know how all these logs are managed.
As it was important and urgent for me to fix that issue, I totaly remove syslog-ng and installed rsyslog (which is installed by default on my own server @ home with the same debian release) and now it works.
So, I probably never know what was broken ... and I don't like this very much.
Many thanks anyway for your kind assistance.

ondoho · 12-13-2019, 12:55 AM

Quote:

Originally Posted by Lefevre

I probably never know what was broken ... and I don't like this very much.

You yourself found what is broken:

Quote:

Originally Posted by Lefevre

Code:

# view /lib/systemd/system/syslog-ng.service
lrwxrwxrwx 1 root root 37 févr.  4  2016 /etc/systemd/system/multi-user.target.wants/syslog-ng.service -> /lib/systemd/system/syslog-ng.service

The file contains a line showing a link like ls do!

It could be that you did this (inadvertently), or that it indeed happened during an update.
Considering how long this system has been running, and how many upgrades it has seen, I would say it's machine-specific, and not something to report to the maintainers.

Lefevre · 12-13-2019, 01:22 AM

Well, you helped me showing the command journalctl -b.
I don't use it often as I'm used too use vi and logs (too old school) and I now have to explore it more deeply.

In fact, I think the things are not so simple.
When I changed that broken file with a good one (extract from official Debian package) syslog-ng was still not running.
I checked other dependant syslog-ng packages but even after reload configs and even a server reboot, kern.log was desperately empty.

In my undestanding kern.log is written by syslog-ng (or rsyslog) after reading /proc/kmsg and filter to dispatch.
So I suppose that the process (?) to read /proc/kmsg was broken for any reason. Install rsyslog fixed it.
I don't want to test it now as it is a live server and all seems to me ok now, but I'm pretty sure, if I would reinstall syslog-ng, it would work.

Thanks again for your time ondoho.