LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 11-03-2012, 03:39 PM   #1
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Rep: Reputation: Disabled
rsyslog filtering syslog messages


Hello there,

I have problem with rsyslog, for filtering syslog messages with regex and then write them on mysql database.
rsyslog.conf file format is following:
:msg, regex, "ASA-0" mmysql:127.0.0.1,Syslog,rsyslog,password
So i am trying to write all syslog messages containing ASA-0 to mysql database, but unfortunately it isnít working, all messages are written in database. Rsyslog version is 4.6.4.
 
Old 11-04-2012, 07:39 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Shouldn't you be using something like :hostname,contains,"ASA-0":ommysql:database-address,database-name,database-userid,database-password?
 
1 members found this post helpful.
Old 11-06-2012, 02:28 AM   #3
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hello unSpawn,


I have already trued your mentioned configuration but it isn't working.

I have observed very strange operation of rsyslog, even when i have removed all statements of:
*.* mmysql:database-address,database-name,database-userid,database-password
log messages are always written to database. If I unload module only the it stops writing syslog to database.
 
Old 11-06-2012, 04:01 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
You probably need to drop ASA-0 after you log to the database, something like
Code:
:hostname,contains,"ASA-0" ~
and apply other rules afterwards, or else filter something like
Code:
if $hostname contains 'ASA-0' then :ommysql:database-address,database-name,database-userid,database-password
if $hostname ! contains 'ASA-0' then /var/log/syslog
Also please note posting your complete rsyslog.conf may provide more insight but saying "it isn't working" on its own isn't the best way to start troubleshooting problems.
 
1 members found this post helpful.
Old 11-06-2012, 04:35 AM   #5
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hire is configuration file. Hire I wont to discard any logs that contains ASA-7, and anything other to write into database.

Quote:
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html


#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

$ModLoad ommysql


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


###############
#### RULES ####
###############


:msg, regex, "ASA-7" ~

*.* mmysql:127.0.0.1,Syslog,rsyslog,password

#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log

#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err

#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice

#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#


#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
Hire is example of received syslog message from cisco ASA:

Quote:
%ASA-7-302014: Teardown TCP connection 54553486 for Outside:192.168.166.4/8080 to inside:HOST/1850 duration 0:00:13 bytes 3320 TCP FINs
 
Old 11-06-2012, 06:03 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
You posted a rsyslog configuration file which:
- doesn't show the rule you posted in your OP,
- doesn't show any trace of what I wrote about in my replies,
- discards any "ASA-7" message before it hits the database filter and
- logs any facility / priority pair (*.*) to database.


Here's what you should do / show:
- check your ASA "logging" for its "facility" setting: it may default to "20", the equivalent of the "local4" facility, so if nothing else uses that you could use it as filter in Rsyslog.
- then apply any ASA filter,
- then drop ASA items after the filter rule,
- then proceed processing other logs.

Then test and post back the result (as in 'grep -v ^# /etc/rsyslog.conf|grep .;').
 
1 members found this post helpful.
Old 11-06-2012, 11:00 AM   #7
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hello unSpawn,

Can you please explain or show documentation which describes order of how rsyslog works with property based filters.
For this rule configuration:

:msg, regex, "ASA-7" ~
*.* mmysql:127.0.0.1,Syslog,rsyslog,password

I was hoping that at first time rsyslog will discards all debugging syslog messages, and all others are written in database ?
How does it process with property based filters, does it process subsequently?


Quote:
Originally Posted by unSpawn View Post
You posted a rsyslog configuration file which:
- doesn't show the rule you posted in your OP,
- doesn't show any trace of what I wrote about in my replies,
- discards any "ASA-7" message before it hits the database filter and
- logs any facility / priority pair (*.*) to database.


Here's what you should do / show:
- check your ASA "logging" for its "facility" setting: it may default to "20", the equivalent of the "local4" facility, so if nothing else uses that you could use it as filter in Rsyslog.
- then apply any ASA filter,
- then drop ASA items after the filter rule,
- then proceed processing other logs.

Then test and post back the result (as in 'grep -v ^# /etc/rsyslog.conf|grep .;').
 
Old 11-06-2012, 11:17 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Basics: http://www.rsyslog.com/doc/rsyslog_conf.html
Writing and discarding: http://www.rsyslog.com/writing-speci...scarding-them/
Examples: http://wiki.rsyslog.com/index.php/Configuration_Samples
Filtering: http://www.rsyslog.com/doc/rsyslog_conf_filter.html (property-based filters halfway down the page)
 
1 members found this post helpful.
Old 11-06-2012, 11:35 AM   #9
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
After reading before mentioned documents, I could't understand why this filter rules don't work ?

:msg, regex, "ASA-7" ~
*.* mmysql:127.0.0.1,Syslog,rsyslog,password

I expect that this two rules won't write into database logs that contains ASA-7 string in message body. But they don't work, messages that contain ASA-7 still are written into database. Can you please explain why it don't work ?

Quote:
Originally Posted by unSpawn View Post
 
Old 11-06-2012, 03:33 PM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Your initial question was:
Quote:
Originally Posted by sanjose View Post
i am trying to write all syslog messages containing ASA-0 to mysql database
I suggested you use a filter like:
Code:
:hostname,contains,"ASA-0" :ommysql:127.0.0.1,Syslog,rsyslog,password
and drop it direct after:
Code:
:hostname,contains,"ASA-0" ~
Of course you're free to use your regex form:
Code:
:msg, regex, "ASA-0":ommysql:127.0.0.1,Syslog,rsyslog,password
:msg, regex, "ASA-0" ~
*Before you do anything else I strongly suggest you practice with standard syslog first to get a grasp of syslog basic concepts.
 
1 members found this post helpful.
Old 11-07-2012, 01:04 PM   #11
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hello,

After implementing following configuration.

Code:
:msg, regex, "ASA-0":ommysql:127.0.0.1,Syslog,rsyslog,password
:msg, regex, "ASA-0" ~
Everything is still written into database. It seems that regex filtering isn't working. Is there any way to make it work ?

Quote:
Originally Posted by unSpawn View Post
Your initial question was:


I suggested you use a filter like:
Code:
:hostname,contains,"ASA-0" :ommysql:127.0.0.1,Syslog,rsyslog,password
and drop it direct after:
Code:
:hostname,contains,"ASA-0" ~
Of course you're free to use your regex form:
Code:
:msg, regex, "ASA-0":ommysql:127.0.0.1,Syslog,rsyslog,password
:msg, regex, "ASA-0" ~
*Before you do anything else I strongly suggest you practice with standard syslog first to get a grasp of syslog basic concepts.
 
Old 11-07-2012, 01:08 PM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Dunno. Haven't seen your modified rsyslog.conf.
 
1 members found this post helpful.
Old 11-07-2012, 01:14 PM   #13
sanjose
LQ Newbie
 
Registered: Nov 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hire is modified config file.

Code:
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
$ModLoad imudp
$UDPServerRun 514
$ModLoad ommysql
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$IncludeConfig /etc/rsyslog.d/*.conf
:msg, regex, "ASA-0":ommysql:127.0.0.1,Syslog,rsyslog,password
:msg, regex, "ASA-0" ~
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log
mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err
news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages
daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole

Quote:
Originally Posted by unSpawn View Post
Dunno. Haven't seen your modified rsyslog.conf.
 
Old 11-07-2012, 03:08 PM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,264
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Some possibilities:
- Instead of ":msg,regex" test ":hostname,contains" or "syslogtag,contains" (after all I'm not sure it even is a hostname tag),
- Instead of ":msg, regex, "ASA-0" ~" try "& ~",
- Check your Cisco machine for its log facility and use that (post #6) instead of a regex,
- See if a convoluted if-something-else filter would do the job,
- Check the vendor for improvements in the ommysql output module and if necessary upgrade. They're at version major 7 BTW.
- Debug the hell out of Reyslogd (http://www.rsyslog.com/doc/debug.html),
or else ask in the rsyslog mailing list.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Send syslog log to rsyslog server? is possible? JohnV2 Slackware 9 10-13-2011 01:37 PM
[SOLVED] I can send syslog log to rsyslog server? how? JohnV2 Linux - Server 6 10-13-2011 08:55 AM
[SOLVED] syslog remote logging with rsyslog server Chenchu Linux - Newbie 3 09-17-2011 01:34 PM
support of third party tools logs in syslog/rsyslog Raheel Hassan Linux - Security 2 08-23-2010 10:30 AM
*realtime* syslog monitoring/alerting with Rsyslog? TotalDefiance Linux - Software 3 11-01-2005 10:23 AM


All times are GMT -5. The time now is 06:34 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration