Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Distribution: Ubuntu Mate 18.04 (production), Arch rolling (tinkering)
Posts: 102
Rep:
Snort catches exit signal unexpectedly
Hello,
I have a problem with my Snort configuration. I want to keep Snort running 24/7 on my CentOS machine but it sometimes quits (caught exit signal) unexpectedly.
Digging into the logs it seems that it quits daily at around 3:30 AM. Digging further shows that Snort's termination correlates with the time Cron starts its daily routine on my machine which includes a Logrotate but noting else relating to Snort.
Is it possibly that Snort exits because of a Logrotate? That would be kinda silly in productive environments, wouldn't it?
(..) Snort's termination correlates with the time Cron starts its daily routine on my machine which includes a Logrotate but noting else relating to Snort.
Is it possibly that Snort exits because of a Logrotate?
Have you created your own logrotate script or did it come with the distribution package? If the latter, did you modify it?
Have you checked if the stop / start or restart command in the logrotate file for snort is correct?
What are the files contents?
And what is the logged line Snort stops with?
Distribution: Ubuntu Mate 18.04 (production), Arch rolling (tinkering)
Posts: 102
Original Poster
Rep:
Quote:
Originally Posted by unSpawn
Have you created your own logrotate script or did it come with the distribution package? If the latter, did you modify it?
No, I didn't make or modify the Logrotate scripts. It was a fresh installation of CentOS 6.3 from the Live-CD. So either Logrotate was set up by the CentOS installer via Cron or by the Snort installation routine. I installed Snort according to this (http://rsabalburo.blogspot.de/2011/0...centos-60.html) tutorial, so basically via
Code:
rpm -Uvh snort-xxx.rpm
and then set it up according to the above mentioned tutorial.
Quote:
Originally Posted by unSpawn
Have you checked if the stop / start or restart command in the logrotate file for snort is correct?
What are the files contents?
The message from the "cron" log reads as following:
Code:
Nov 8 03:01:01 localhost anacron[450]: Anacron started on 2012-11-08
Nov 8 03:01:01 localhost anacron[450]: Will run job `cron.daily' in 24 min.
Nov 8 03:01:01 localhost anacron[450]: Jobs will be executed sequentially
Nov 8 03:01:01 localhost run-parts(/etc/cron.hourly)[452]: finished 0anacron
Nov 8 03:10:01 localhost CROND[470]: (root) CMD (/usr/lib/sa/sa1 1 1)
Nov 8 03:20:01 localhost CROND[491]: (root) CMD (/usr/lib/sa/sa1 1 1)
Nov 8 03:25:01 localhost anacron[450]: Job `cron.daily' started
Nov 8 03:25:01 localhost run-parts(/etc/cron.daily)[500]: starting cups
Nov 8 03:25:01 localhost run-parts(/etc/cron.daily)[508]: finished cups
Nov 8 03:25:01 localhost run-parts(/etc/cron.daily)[500]: starting logrotate
Nov 8 03:25:46 localhost run-parts(/etc/cron.daily)[543]: finished logrotate
Nov 8 03:25:46 localhost run-parts(/etc/cron.daily)[500]: starting makewhatis.cron
Nov 8 03:25:50 localhost run-parts(/etc/cron.daily)[672]: finished makewhatis.cron
Nov 8 03:25:50 localhost run-parts(/etc/cron.daily)[500]: starting mlocate.cron
Nov 8 03:25:56 localhost run-parts(/etc/cron.daily)[683]: finished mlocate.cron
Nov 8 03:25:56 localhost run-parts(/etc/cron.daily)[500]: starting prelink
Nov 8 03:25:56 localhost run-parts(/etc/cron.daily)[695]: finished prelink
Nov 8 03:25:56 localhost run-parts(/etc/cron.daily)[500]: starting readahead.cron
Nov 8 03:25:56 localhost run-parts(/etc/cron.daily)[706]: finished readahead.cron
Nov 8 03:25:56 localhost run-parts(/etc/cron.daily)[500]: starting tmpwatch
Nov 8 03:25:57 localhost run-parts(/etc/cron.daily)[744]: finished tmpwatch
Nov 8 03:25:57 localhost anacron[450]: Job `cron.daily' terminated
Nov 8 03:25:57 localhost anacron[450]: Normal exit (1 job run)
Quote:
Originally Posted by unSpawn
And what is the logged line Snort stops with?
Here is the excerpt from the "messages" log with the moment Snort exited:
Code:
Nov 7 23:45:56 localhost kernel: device br0 entered promiscuous mode
Nov 8 00:28:11 localhost kernel: device br0 left promiscuous mode
Nov 8 00:30:55 localhost kernel: device br0 entered promiscuous mode
Nov 8 03:25:02 localhost snort[29460]: *** Caught Term-Signal
Nov 8 03:25:03 localhost kernel: device br0 left promiscuous mode
Nov 8 03:25:03 localhost snort[29460]: ===============================================================================
Nov 8 03:25:03 localhost snort[29460]: Run time for packet processing was 85041.797021 seconds
Nov 8 03:25:03 localhost snort[29460]: Snort processed 27344581 packets.
Nov 8 03:25:03 localhost snort[29460]: Snort ran for 0 days 23 hours 37 minutes 21 seconds
Nov 8 03:25:03 localhost snort[29460]: Pkts/hr: 1188894
Nov 8 03:25:03 localhost snort[29460]: Pkts/min: 19297
Nov 8 03:25:03 localhost snort[29460]: Pkts/sec: 321
Nov 8 03:25:04 localhost snort[29460]: ===============================================================================
Nov 8 03:25:04 localhost snort[29460]: Packet I/O Totals:
Nov 8 03:25:04 localhost snort[29460]: Received: 27384306
Nov 8 03:25:04 localhost snort[29460]: Analyzed: 27344581 ( 99.855%)
Nov 8 03:25:04 localhost snort[29460]: Dropped: 39441 ( 0.144%)
Nov 8 03:25:04 localhost snort[29460]: Filtered: 0 ( 0.000%)
Nov 8 03:25:04 localhost snort[29460]: Outstanding: 39725 ( 0.145%)
Nov 8 03:25:04 localhost snort[29460]: Injected: 0
Nov 8 03:25:04 localhost snort[29460]: ===============================================================================
Nov 8 03:25:04 localhost snort[29460]: Breakdown by protocol (includes rebuilt packets):
... more lines indicating the results of the Snort run that was exited...
So the exit message is basically just:
Code:
Nov 8 03:25:02 localhost snort[29460]: *** Caught Term-Signal
Strangely enough, right after the messages from the exited Snort instance, there is a new Snort initialization in the "messages" log:
Code:
...messages from exited Snort instance...
Nov 8 03:25:04 localhost snort[29460]: Responses: 0
Nov 8 03:25:04 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 29460 due to rate-limiting
Nov 8 03:25:07 localhost snort[536]: Running in IDS mode
Nov 8 03:25:07 localhost snort[536]:
Nov 8 03:25:07 localhost snort[536]: --== Initializing Snort ==--
Nov 8 03:25:07 localhost snort[536]: Initializing Output Plugins!
Nov 8 03:25:07 localhost snort[536]: Initializing Preprocessors!
Nov 8 03:25:07 localhost snort[536]: Initializing Plug-ins!
Nov 8 03:25:07 localhost snort[536]: Parsing Rules file "/etc/snort/snort.conf"
...initialization messages from new Snort instance, up to:
Nov 8 03:25:46 localhost snort[538]: Decoding Ethernet
Nov 8 03:25:46 localhost snort[538]: Checking PID path...
Nov 8 03:25:46 localhost snort[538]: PID path stat checked out ok, PID path set to /var/run/
Nov 8 03:25:46 localhost snort[538]: Writing PID "538" to file "/var/run//snort_eth0.pid"
Nov 8 03:25:46 localhost snort[538]: Set gid to 501
Nov 8 03:25:46 localhost snort[538]: Set uid to 501
Nov 8 03:25:46 localhost snort[538]:
Nov 8 03:25:46 localhost snort[538]: --== Initialization Complete ==--
Nov 8 03:25:46 localhost snort[538]: Commencing packet processing (pid=538)
The next message in "messages" log is this:
Code:
Nov 8 08:27:26 localhost kernel: device br0 entered promiscuous mode
-> 08:27:26 is the moment when I returned to the computer in the morning, saw this in the terminal window:
Code:
--== Initialization Complete ==--
,,_ -*> Snort! <*-
o" )~ Version 2.9.3.1 IPv6 GRE (Build 40)
'''' By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
Copyright (C) 1998-2012 Sourcefire, Inc., et al.
Using libpcap version 1.2.1
Using PCRE version: 7.8 2008-09-05
Using ZLIB version: 1.2.3
Rules Engine: SF_SNORT_DETECTION_ENGINE Version 1.16 <Build 18>
Preprocessor Object: SF_SSH (IPV6) Version 1.1 <Build 3>
Preprocessor Object: SF_IMAP (IPV6) Version 1.0 <Build 1>
Preprocessor Object: SF_SMTP (IPV6) Version 1.1 <Build 9>
Preprocessor Object: SF_GTP (IPV6) Version 1.1 <Build 1>
Preprocessor Object: SF_POP (IPV6) Version 1.0 <Build 1>
Preprocessor Object: SF_SIP (IPV6) Version 1.1 <Build 1>
Preprocessor Object: SF_REPUTATION (IPV6) Version 1.1 <Build 1>
Preprocessor Object: SF_SDF (IPV6) Version 1.1 <Build 1>
Preprocessor Object: SF_MODBUS (IPV6) Version 1.1 <Build 1>
Preprocessor Object: SF_SSLPP (IPV6) Version 1.1 <Build 4>
Preprocessor Object: SF_DCERPC2 (IPV6) Version 1.0 <Build 3>
Preprocessor Object: SF_FTPTELNET (IPV6) Version 1.2 <Build 13>
Preprocessor Object: SF_DNS (IPV6) Version 1.1 <Build 4>
Preprocessor Object: SF_DNP3 (IPV6) Version 1.1 <Build 1>
Commencing packet processing (pid=1449)
*** Caught Term-Signal
[rokyo@centos$]~ _
and restarted Snort via:
Code:
snort -c /etc/snort/snort.conf -A full -b -i br0
So, I assume (now after reading the log files) that when I started this new Snort instance at 8:27 AM there was already an instance running (the one started at 3:25 AM according to the "messages" log), which was just not shown in the terminal because it wasn't started by me as a user but automatically. Could that be correct?
Distribution: Ubuntu Mate 18.04 (production), Arch rolling (tinkering)
Posts: 102
Original Poster
Rep:
Oh, I forgot:
The contents of /etc/cron.daily/logrotate:
Code:
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
and of /etc/logrotate.conf:
Code:
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0600 root utmp
rotate 1
}
# system-specific logs may be also be configured here.
Distribution: Ubuntu Mate 18.04 (production), Arch rolling (tinkering)
Posts: 102
Original Poster
Rep:
Do I see it correctly that at the moment of log-rotation the Snort log-files are compressed, new ones are created and the Snort-Deamon is restarted using the newly created log-files?
Because that would explain why I don't see any alerts from after 3:25 AM in the terminal window I have running with the command
Code:
tail -f /var/log/snort/alerts
, yet there are logged alerts from after 3:25 AM when I looked at the "alerts" file with
Code:
cat /var/log/snort/alerts
at 8:27 AM...? I assume, the "tail" command gets cancelled (or "runs into nothing") by the creation of a new "alerts" file by logrotate at 3:25 AM?
Also, this would mean that even though the Snort instance I see in my terminal window at 8:27 AM that exited with
Code:
*** Caught Term-Signal
was killed at 3:25 AM, there was already a new instance running which was started by the snort-logrotate script at 3:25 AM but just wasn't shown in the terminal because it was started by a deamon, right?
So if, from what you posted, the restart was triggered by logrotate then Snortd should have restarted between 03:25:01 and 03:25:46.
Let's combine some log lines:
Strangely enough, right after the messages from the exited Snort instance, there is a new Snort initialization in the "messages" log
That's because /etc/logrotate.d/snort takes care of stopping and then starting Snort as shown above.
Quote:
Originally Posted by rokyo
The next message in "messages" log is this:
Code:
Nov 8 08:27:26 localhost kernel: device br0 entered promiscuous mode
-> 08:27:26 is the moment when I returned to the computer in the morning, saw this in the terminal window:
(..)
and restarted Snort via:
Code:
snort -c /etc/snort/snort.conf -A full -b -i br0
You should not do that unless you know Snort died.
If you want to assess its state use the basic tools the system provides like
Code:
pgrep -l snort
or the more specific
Code:
pgrep -lf "snort -c /etc/snort/snort.conf"
or the tools the distribution provides like
Code:
/sbin/service snortd status
or
Code:
/etc/init.d/snortd status
Quote:
Originally Posted by rokyo
when I started this new Snort instance at 8:27 AM there was already an instance running (..), which was just not shown in the terminal because it wasn't started by me as a user but automatically. Could that be correct?
Unless the process died somewhere along the way (some vulnerability, network device disappearance, no disk space for logging, whatever else) you may expect Snort to run by default as a daemon like most services and w/o any output to stdout. If you would want to monitor the process continuously, have it restart automagically and alert on (selected) trouble then see for instance Monit.
Distribution: Ubuntu Mate 18.04 (production), Arch rolling (tinkering)
Posts: 102
Original Poster
Rep:
Thank you for the quick answer!
So, basically I don't have any problem here, because Snort is running & restarting as a deamon 24/7 and I don't even need to start it manually anymore. I just need to find a different way than "tail -f" to show the alerts on screen because that would quit showing alerts after logrotation.
So, basically I don't have any problem here, because Snort is running & restarting as a deamon 24/7
You don't have one.
Quote:
Originally Posted by rokyo
and I don't even need to start it manually anymore.
Generally speaking you should never have to.
Quote:
Originally Posted by rokyo
I just need to find a different way than "tail -f" to show the alerts on screen
Adhering to the UNIX philosophy of having a tool excelling at performing a task, Snorts main function is traffic analysis. Anything else, like having to process and output human-readable reporting, just makes it less efficient, slows it down. You should prefer to log in binary format (unified logging), have Barnyard2 process those logs and have it (or via analysis by other tools) report via email. Else see one of the current, maintained (web-based) front-ends.
Distribution: Ubuntu Mate 18.04 (production), Arch rolling (tinkering)
Posts: 102
Original Poster
Rep:
Perfect! Now the only thing that is missing is full logging. When Snort is started as a Daemon at startup, it automatically seems to use the "fast" logging option. How do I make it use the "full" logging option which on the CLI would be the "-a full" option?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.