LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 11-08-2012, 12:45 AM   #1
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Rep: Reputation: Disabled
Snort catches exit signal unexpectedly


Hello,

I have a problem with my Snort configuration. I want to keep Snort running 24/7 on my CentOS machine but it sometimes quits (caught exit signal) unexpectedly.

Digging into the logs it seems that it quits daily at around 3:30 AM. Digging further shows that Snort's termination correlates with the time Cron starts its daily routine on my machine which includes a Logrotate but noting else relating to Snort.

Is it possibly that Snort exits because of a Logrotate? That would be kinda silly in productive environments, wouldn't it?
 
Old 11-08-2012, 05:16 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by rokyo View Post
(..) Snort's termination correlates with the time Cron starts its daily routine on my machine which includes a Logrotate but noting else relating to Snort.
Is it possibly that Snort exits because of a Logrotate?
Have you created your own logrotate script or did it come with the distribution package? If the latter, did you modify it?
Have you checked if the stop / start or restart command in the logrotate file for snort is correct?
What are the files contents?
And what is the logged line Snort stops with?
 
Old 11-08-2012, 06:27 AM   #3
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Have you created your own logrotate script or did it come with the distribution package? If the latter, did you modify it?
No, I didn't make or modify the Logrotate scripts. It was a fresh installation of CentOS 6.3 from the Live-CD. So either Logrotate was set up by the CentOS installer via Cron or by the Snort installation routine. I installed Snort according to this (http://rsabalburo.blogspot.de/2011/0...centos-60.html) tutorial, so basically via
Code:
rpm -Uvh snort-xxx.rpm
and then set it up according to the above mentioned tutorial.

Quote:
Originally Posted by unSpawn View Post
Have you checked if the stop / start or restart command in the logrotate file for snort is correct?
What are the files contents?
The message from the "cron" log reads as following:
Code:
Nov  8 03:01:01 localhost anacron[450]: Anacron started on 2012-11-08
Nov  8 03:01:01 localhost anacron[450]: Will run job `cron.daily' in 24 min.
Nov  8 03:01:01 localhost anacron[450]: Jobs will be executed sequentially
Nov  8 03:01:01 localhost run-parts(/etc/cron.hourly)[452]: finished 0anacron
Nov  8 03:10:01 localhost CROND[470]: (root) CMD (/usr/lib/sa/sa1 1 1)
Nov  8 03:20:01 localhost CROND[491]: (root) CMD (/usr/lib/sa/sa1 1 1)
Nov  8 03:25:01 localhost anacron[450]: Job `cron.daily' started
Nov  8 03:25:01 localhost run-parts(/etc/cron.daily)[500]: starting cups
Nov  8 03:25:01 localhost run-parts(/etc/cron.daily)[508]: finished cups
Nov  8 03:25:01 localhost run-parts(/etc/cron.daily)[500]: starting logrotate
Nov  8 03:25:46 localhost run-parts(/etc/cron.daily)[543]: finished logrotate
Nov  8 03:25:46 localhost run-parts(/etc/cron.daily)[500]: starting makewhatis.cron
Nov  8 03:25:50 localhost run-parts(/etc/cron.daily)[672]: finished makewhatis.cron
Nov  8 03:25:50 localhost run-parts(/etc/cron.daily)[500]: starting mlocate.cron
Nov  8 03:25:56 localhost run-parts(/etc/cron.daily)[683]: finished mlocate.cron
Nov  8 03:25:56 localhost run-parts(/etc/cron.daily)[500]: starting prelink
Nov  8 03:25:56 localhost run-parts(/etc/cron.daily)[695]: finished prelink
Nov  8 03:25:56 localhost run-parts(/etc/cron.daily)[500]: starting readahead.cron
Nov  8 03:25:56 localhost run-parts(/etc/cron.daily)[706]: finished readahead.cron
Nov  8 03:25:56 localhost run-parts(/etc/cron.daily)[500]: starting tmpwatch
Nov  8 03:25:57 localhost run-parts(/etc/cron.daily)[744]: finished tmpwatch
Nov  8 03:25:57 localhost anacron[450]: Job `cron.daily' terminated
Nov  8 03:25:57 localhost anacron[450]: Normal exit (1 job run)


Quote:
Originally Posted by unSpawn View Post
And what is the logged line Snort stops with?
Here is the excerpt from the "messages" log with the moment Snort exited:

Code:
Nov  7 23:45:56 localhost kernel: device br0 entered promiscuous mode
Nov  8 00:28:11 localhost kernel: device br0 left promiscuous mode
Nov  8 00:30:55 localhost kernel: device br0 entered promiscuous mode
Nov  8 03:25:02 localhost snort[29460]: *** Caught Term-Signal
Nov  8 03:25:03 localhost kernel: device br0 left promiscuous mode
Nov  8 03:25:03 localhost snort[29460]: ===============================================================================
Nov  8 03:25:03 localhost snort[29460]: Run time for packet processing was 85041.797021 seconds
Nov  8 03:25:03 localhost snort[29460]: Snort processed 27344581 packets.
Nov  8 03:25:03 localhost snort[29460]: Snort ran for 0 days 23 hours 37 minutes 21 seconds
Nov  8 03:25:03 localhost snort[29460]:     Pkts/hr:      1188894
Nov  8 03:25:03 localhost snort[29460]:    Pkts/min:        19297
Nov  8 03:25:03 localhost snort[29460]:    Pkts/sec:          321
Nov  8 03:25:04 localhost snort[29460]: ===============================================================================
Nov  8 03:25:04 localhost snort[29460]: Packet I/O Totals:
Nov  8 03:25:04 localhost snort[29460]:    Received:     27384306
Nov  8 03:25:04 localhost snort[29460]:    Analyzed:     27344581 ( 99.855%)
Nov  8 03:25:04 localhost snort[29460]:     Dropped:        39441 (  0.144%)
Nov  8 03:25:04 localhost snort[29460]:    Filtered:            0 (  0.000%)
Nov  8 03:25:04 localhost snort[29460]: Outstanding:        39725 (  0.145%)
Nov  8 03:25:04 localhost snort[29460]:    Injected:            0
Nov  8 03:25:04 localhost snort[29460]: ===============================================================================
Nov  8 03:25:04 localhost snort[29460]: Breakdown by protocol (includes rebuilt packets):
... more lines indicating the results of the Snort run that was exited...
So the exit message is basically just:

Code:
Nov  8 03:25:02 localhost snort[29460]: *** Caught Term-Signal
Strangely enough, right after the messages from the exited Snort instance, there is a new Snort initialization in the "messages" log:

Code:
...messages from exited Snort instance...
Nov  8 03:25:04 localhost snort[29460]:   Responses: 0
Nov  8 03:25:04 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 29460 due to rate-limiting
Nov  8 03:25:07 localhost snort[536]: Running in IDS mode
Nov  8 03:25:07 localhost snort[536]: 
Nov  8 03:25:07 localhost snort[536]:         --== Initializing Snort ==--
Nov  8 03:25:07 localhost snort[536]: Initializing Output Plugins!
Nov  8 03:25:07 localhost snort[536]: Initializing Preprocessors!
Nov  8 03:25:07 localhost snort[536]: Initializing Plug-ins!
Nov  8 03:25:07 localhost snort[536]: Parsing Rules file "/etc/snort/snort.conf"

...initialization messages from new Snort instance, up to:

Nov  8 03:25:46 localhost snort[538]: Decoding Ethernet
Nov  8 03:25:46 localhost snort[538]: Checking PID path...
Nov  8 03:25:46 localhost snort[538]: PID path stat checked out ok, PID path set to /var/run/
Nov  8 03:25:46 localhost snort[538]: Writing PID "538" to file "/var/run//snort_eth0.pid"
Nov  8 03:25:46 localhost snort[538]: Set gid to 501
Nov  8 03:25:46 localhost snort[538]: Set uid to 501
Nov  8 03:25:46 localhost snort[538]: 
Nov  8 03:25:46 localhost snort[538]:         --== Initialization Complete ==--
Nov  8 03:25:46 localhost snort[538]: Commencing packet processing (pid=538)
The next message in "messages" log is this:

Code:
Nov  8 08:27:26 localhost kernel: device br0 entered promiscuous mode
-> 08:27:26 is the moment when I returned to the computer in the morning, saw this in the terminal window:
Code:
        --== Initialization Complete ==--

   ,,_     -*> Snort! <*-
  o"  )~   Version 2.9.3.1 IPv6 GRE (Build 40) 
   ''''    By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
           Copyright (C) 1998-2012 Sourcefire, Inc., et al.
           Using libpcap version 1.2.1
           Using PCRE version: 7.8 2008-09-05
           Using ZLIB version: 1.2.3

           Rules Engine: SF_SNORT_DETECTION_ENGINE  Version 1.16  <Build 18>
           Preprocessor Object: SF_SSH (IPV6)  Version 1.1  <Build 3>
           Preprocessor Object: SF_IMAP (IPV6)  Version 1.0  <Build 1>
           Preprocessor Object: SF_SMTP (IPV6)  Version 1.1  <Build 9>
           Preprocessor Object: SF_GTP (IPV6)  Version 1.1  <Build 1>
           Preprocessor Object: SF_POP (IPV6)  Version 1.0  <Build 1>
           Preprocessor Object: SF_SIP (IPV6)  Version 1.1  <Build 1>
           Preprocessor Object: SF_REPUTATION (IPV6)  Version 1.1  <Build 1>
           Preprocessor Object: SF_SDF (IPV6)  Version 1.1  <Build 1>
           Preprocessor Object: SF_MODBUS (IPV6)  Version 1.1  <Build 1>
           Preprocessor Object: SF_SSLPP (IPV6)  Version 1.1  <Build 4>
           Preprocessor Object: SF_DCERPC2 (IPV6)  Version 1.0  <Build 3>
           Preprocessor Object: SF_FTPTELNET (IPV6)  Version 1.2  <Build 13>
           Preprocessor Object: SF_DNS (IPV6)  Version 1.1  <Build 4>
           Preprocessor Object: SF_DNP3 (IPV6)  Version 1.1  <Build 1>
Commencing packet processing (pid=1449)
 *** Caught Term-Signal
[rokyo@centos$]~ _
and restarted Snort via:

Code:
snort -c /etc/snort/snort.conf -A full -b -i br0
So, I assume (now after reading the log files) that when I started this new Snort instance at 8:27 AM there was already an instance running (the one started at 3:25 AM according to the "messages" log), which was just not shown in the terminal because it wasn't started by me as a user but automatically. Could that be correct?
 
Old 11-08-2012, 06:37 AM   #4
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Original Poster
Rep: Reputation: Disabled
Oh, I forgot:

The contents of /etc/cron.daily/logrotate:
Code:
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
and of /etc/logrotate.conf:
Code:
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
	minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.
and finally /etc/logrotate.d/snort:

Code:
# /etc/logrotate.d/snort
# $Id$

/var/log/snort/alert /var/log/snort/*log /var/log/snort/*/alert /var/log/snort/*/*log  {
    daily
    rotate 7
    missingok
    compress
    sharedscripts
    postrotate
	/etc/init.d/snortd restart 1>/dev/null || true
    endscript
}
 
Old 11-08-2012, 07:09 AM   #5
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Original Poster
Rep: Reputation: Disabled
Do I see it correctly that at the moment of log-rotation the Snort log-files are compressed, new ones are created and the Snort-Deamon is restarted using the newly created log-files?

Because that would explain why I don't see any alerts from after 3:25 AM in the terminal window I have running with the command
Code:
tail -f /var/log/snort/alerts
, yet there are logged alerts from after 3:25 AM when I looked at the "alerts" file with
Code:
cat /var/log/snort/alerts
at 8:27 AM...? I assume, the "tail" command gets cancelled (or "runs into nothing") by the creation of a new "alerts" file by logrotate at 3:25 AM?

Also, this would mean that even though the Snort instance I see in my terminal window at 8:27 AM that exited with
Code:
 *** Caught Term-Signal
was killed at 3:25 AM, there was already a new instance running which was started by the snort-logrotate script at 3:25 AM but just wasn't shown in the terminal because it was started by a deamon, right?
 
Old 11-08-2012, 07:12 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
First of all thanks for the comprehensive reply.

So if, from what you posted, the restart was triggered by logrotate then Snortd should have restarted between 03:25:01 and 03:25:46.
Let's combine some log lines:

Code:
03:25:01 localhost run-parts(/etc/cron.daily)[500]: starting logrotate
03:25:02 localhost snort[29460]: *** Caught Term-Signal
03:25:03 localhost kernel: device br0 left promiscuous mode
03:25:46 localhost snort[538]: Commencing packet processing (pid=538)
03:25:46 localhost run-parts(/etc/cron.daily)[543]: finished logrotate
Fits like a glove.


Quote:
Originally Posted by rokyo View Post
Strangely enough, right after the messages from the exited Snort instance, there is a new Snort initialization in the "messages" log
That's because /etc/logrotate.d/snort takes care of stopping and then starting Snort as shown above.


Quote:
Originally Posted by rokyo View Post
The next message in "messages" log is this:
Code:
Nov  8 08:27:26 localhost kernel: device br0 entered promiscuous mode
-> 08:27:26 is the moment when I returned to the computer in the morning, saw this in the terminal window:
(..)
and restarted Snort via:
Code:
snort -c /etc/snort/snort.conf -A full -b -i br0
You should not do that unless you know Snort died.
If you want to assess its state use the basic tools the system provides like
Code:
pgrep -l snort
or the more specific
Code:
pgrep -lf "snort -c /etc/snort/snort.conf"
or the tools the distribution provides like
Code:
/sbin/service snortd status
or
Code:
/etc/init.d/snortd status

Quote:
Originally Posted by rokyo View Post
when I started this new Snort instance at 8:27 AM there was already an instance running (..), which was just not shown in the terminal because it wasn't started by me as a user but automatically. Could that be correct?
Unless the process died somewhere along the way (some vulnerability, network device disappearance, no disk space for logging, whatever else) you may expect Snort to run by default as a daemon like most services and w/o any output to stdout. If you would want to monitor the process continuously, have it restart automagically and alert on (selected) trouble then see for instance Monit.
 
1 members found this post helpful.
Old 11-08-2012, 07:21 AM   #7
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Original Poster
Rep: Reputation: Disabled
Thank you for the quick answer!

So, basically I don't have any problem here, because Snort is running & restarting as a deamon 24/7 and I don't even need to start it manually anymore. I just need to find a different way than "tail -f" to show the alerts on screen because that would quit showing alerts after logrotation.

I will check out Monit right now!
 
Old 11-08-2012, 07:53 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by rokyo View Post
So, basically I don't have any problem here, because Snort is running & restarting as a deamon 24/7
You don't have one.


Quote:
Originally Posted by rokyo View Post
and I don't even need to start it manually anymore.
Generally speaking you should never have to.


Quote:
Originally Posted by rokyo View Post
I just need to find a different way than "tail -f" to show the alerts on screen
Adhering to the UNIX philosophy of having a tool excelling at performing a task, Snorts main function is traffic analysis. Anything else, like having to process and output human-readable reporting, just makes it less efficient, slows it down. You should prefer to log in binary format (unified logging), have Barnyard2 process those logs and have it (or via analysis by other tools) report via email. Else see one of the current, maintained (web-based) front-ends.
 
1 members found this post helpful.
Old 11-08-2012, 10:56 AM   #9
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Original Poster
Rep: Reputation: Disabled
Perfect! Now the only thing that is missing is full logging. When Snort is started as a Daemon at startup, it automatically seems to use the "fast" logging option. How do I make it use the "full" logging option which on the CLI would be the "-a full" option?
 
Old 11-08-2012, 11:55 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by rokyo View Post
When Snort is started as a Daemon at startup, it automatically seems to use the "fast" logging option.
Packagers tend to choose reasonable default startup options.


Quote:
Originally Posted by rokyo View Post
How do I make it use the "full" logging option which on the CLI would be the "-a full" option?
'man snort' for available options, /etc/rc.d/init.d/snortd for hardcoded options and /etc/sysconfig/snort* for customizable ones.
 
1 members found this post helpful.
Old 11-11-2012, 11:23 AM   #11
rokyo
Member
 
Registered: Oct 2012
Posts: 70

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
'man snort' for available options, /etc/rc.d/init.d/snortd for hardcoded options and /etc/sysconfig/snort* for customizable ones.
Perfect! I found it and the config file is very well documented, so changing what I wanted worked well. Now everything is running as I wanted it. ^^
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
exit signal Bus error (7) rvbalraj Linux - Server 1 07-18-2008 12:19 AM
signal SIGINT handler SOMETIMES doesn't exit Peterius Programming 3 05-25-2008 11:55 PM
exit signal Segmentation fault Swakoo Linux - Newbie 3 12-22-2005 09:52 PM
Unusual signal 11 exit, Enemy Territory - help please? tmadhavan Linux - Software 2 03-02-2004 12:53 PM
kppp daemon died unexpectedly exit status 4 steelrose Linux - Networking 1 04-03-2002 04:00 AM


All times are GMT -5. The time now is 10:35 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration