LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-17-2014, 10:00 AM   #1
cdaly26@gmail.com
LQ Newbie
 
Registered: Dec 2014
Posts: 2

Rep: Reputation: Disabled
Firewall Log parser/reporting


I'm looking for a utility, piece of software, perl script idea or other method to be able to parse a firewall log (or any large log file) and generate reports - either text-based or HTML - based on specific criteria:
Example: From the firewall log, I want to generate a report that shows all TCP/UDP traffic destined for a particular host.

Any ideas?

Note: Most of the tools I've seen thus far are parcelled with a syslog server function - all I need is something I can take an archived log file and run reports against it.
 
Old 12-17-2014, 12:08 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 17,934

Rep: Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692
Quote:
Originally Posted by cdaly26@gmail.com View Post
I'm looking for a utility, piece of software, perl script idea or other method to be able to parse a firewall log (or any large log file) and generate reports - either text-based or HTML - based on specific criteria:
Example: From the firewall log, I want to generate a report that shows all TCP/UDP traffic destined for a particular host. Any ideas?

Note: Most of the tools I've seen thus far are parcelled with a syslog server function - all I need is something I can take an archived log file and run reports against it.
Your question is a bit light on the details. You don't say what firewall log you want to parse, how you want to interact with it (you say you want things for a 'particular host'..how, exactly, are you going to SPECIFY it?), what kind of report you are looking for, in what format, and how much data you want to save. If you're concerned about your firewall logs for a week, that is far different than having to report on things going back a year.

You can easily use a tool like splunk for such things, or write your own. How difficult that is, depends on what you're after. Personally, I'd write a perl program to shove the firewall logs into a database (MariaDB or MySQL), then run reports from there. You can then specify what you're after on the command line, or easily write a PHP script to do it from a web browser.
 
Old 12-17-2014, 01:24 PM   #3
cdaly26@gmail.com
LQ Newbie
 
Registered: Dec 2014
Posts: 2

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by TB0ne View Post
Your question is a bit light on the details. You don't say what firewall log you want to parse, how you want to interact with it (you say you want things for a 'particular host'..how, exactly, are you going to SPECIFY it?), what kind of report you are looking for, in what format, and how much data you want to save. If you're concerned about your firewall logs for a week, that is far different than having to report on things going back a year.

You can easily use a tool like splunk for such things, or write your own. How difficult that is, depends on what you're after. Personally, I'd write a perl program to shove the firewall logs into a database (MariaDB or MySQL), then run reports from there. You can then specify what you're after on the command line, or easily write a PHP script to do it from a web browser.
Thanks TB0ne!

I did look at Splunk briefly - but there seemed to be a lot of extra functionality there that (at least at this point) I didn't need. The host would be specified by IP address in this case - as far as report format and data retention - the logfiles I'm working with are generated and archived daily. Rather than slog thru it doing a manual search for the IP address, I was hoping I could generate a report that would dump it to a file or shove it on an HTML page. For example, the IP address might be 10.10.10.100, and it may receive connections throughout the day on TCP ports 8080, and udp port 8081. It would be cool to be able to run the script/report against the firewall log for that day, specifying the host's IP address and querying on the TCP/UDP service field so the output would come back something like:
Host Services
10.10.10.100 8080-TCP, 8081-UDP



As for the file format, they are files from a syslog server that receives log messages from the firewall. Format as below:
Dec 11 11:45:50 10.88.194.114 Dec: 11 2014 11:45:57 <Firewall Name> : %ASA-6-106100: access-list OUTSIDE permitted tcp outside/1.1.1.1(56888) -> inside/10.10.10.100(8080) hit-cnt 1 first hit [0xeb2f9280, 0x9def7ee9]

Dec 11 11:45:50 10.88.194.114 Dec: 11 2014 11:45:57 <Firewall Name> : %ASA-6-106100: access-list OUTSIDE permitted udp outside/1.1.1.1(56888) -> inside/10.10.10.100(8081) hit-cnt 1 first hit [0xeb2f9280, 0x9def7ee9]
inside/10.10.10.100(8080) - this indicates the destination IP address - 10.10.10.100, and the destination port - 8081, and the service "udp"


Is the perl-script to database still the best option?
 
Old 12-17-2014, 01:31 PM   #4
lsalab
LQ Newbie
 
Registered: Jan 2009
Posts: 16

Rep: Reputation: 1
By the looks of it, you want to parse the firewall logs from a cisco ASA device.

I would recomend you to take a look at 'Sawmill'. I believe it's exactly what you need. You specify a path for your logs and it reads all the logs, parses them and generate reports via a web interface. Also, you can send the reports via e-mail.

Check it out: www.sawmill.net
 
Old 12-17-2014, 02:53 PM   #5
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 17,934

Rep: Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692
Quote:
Originally Posted by cdaly26@gmail.com View Post
Thanks TB0ne!
I did look at Splunk briefly - but there seemed to be a lot of extra functionality there that (at least at this point) I didn't need.
Splunk does have a good bit of functionality...the upside is, you can direct OTHER systems to it as well, and get a lot more use out of your investment.
Quote:
The host would be specified by IP address in this case - as far as report format and data retention - the logfiles I'm working with are generated and archived daily. Rather than slog thru it doing a manual search for the IP address, I was hoping I could generate a report that would dump it to a file or shove it on an HTML page. For example, the IP address might be 10.10.10.100, and it may receive connections throughout the day on TCP ports 8080, and udp port 8081. It would be cool to be able to run the script/report against the firewall log for that day, specifying the host's IP address and querying on the TCP/UDP service field so the output would come back something like:
Host Services
10.10.10.100 8080-TCP, 8081-UDP

As for the file format, they are files from a syslog server that receives log messages from the firewall. Format as below:
Dec 11 11:45:50 10.88.194.114 Dec: 11 2014 11:45:57 <Firewall Name> : %ASA-6-106100: access-list OUTSIDE permitted tcp outside/1.1.1.1(56888) -> inside/10.10.10.100(8080) hit-cnt 1 first hit [0xeb2f9280, 0x9def7ee9]

Dec 11 11:45:50 10.88.194.114 Dec: 11 2014 11:45:57 <Firewall Name> : %ASA-6-106100: access-list OUTSIDE permitted udp outside/1.1.1.1(56888) -> inside/10.10.10.100(8081) hit-cnt 1 first hit [0xeb2f9280, 0x9def7ee9]
inside/10.10.10.100(8080) - this indicates the destination IP address - 10.10.10.100, and the destination port - 8081, and the service "udp"

Is the perl-script to database still the best option?
It is *AN* option..."best" is open to interpretation. lsalab gave another option as well. Splunk and/or sawmill are good choices, and will do what you're after. But, for me, I'd use something I already wrote, and just make my own, but again, that is personal preference. Shoveling events into a database will give you lots of reporting options, but it WILL require coding and work, as well as upkeep....nothing in life is free.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
W3C log parser acedreds Linux - Software 0 10-16-2008 04:10 PM
Apache log parser tool manojbarot1 Linux - Software 2 01-03-2008 01:10 AM
Log File Parser Program kaplan71 Linux - General 1 05-11-2005 09:55 PM
qpopper log parser linuxbox25 Programming 0 03-08-2004 05:47 PM
firewall log parser tarballedtux Linux - Software 0 08-04-2003 10:04 PM


All times are GMT -5. The time now is 10:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration