LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 12-18-2017, 01:25 PM   #1
ranmanh
LQ Newbie
 
Registered: Apr 2010
Posts: 10

Rep: Reputation: 0
Lightbulb OpenSource Log Analyzers ? Suggestions?


Hi

I got a number of Linux Servers with syslog-ng on sending all the logs to an rsyslog server where all the log data is being stored.

Now I am looking into some sort of scalable OpenSource solution which can capture all that data for analysis.

For Alerting purposes I have thought on Sensu as it's scalable and I got some previous experience with it, but I am still not sure about the middle man.

I have found graylog, but not sure whether that would be a good solution or not.
Note: I do not want to touch the rsyslog server but just maybe to setup an agent which will be sending the logs to the log-analyzer server

Base on your experience, any recommendations I can follow on my research?

Many thanks
 
Old 12-18-2017, 08:09 PM   #2
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mageia, and whatever VMs I happen to be playing with
Posts: 13,574
Blog Entries: 20

Rep: Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570
Nagios comes to mind. Log analysis is one of the services it offers.

A web search for open source log analyzers turns up a number of articles.
 
1 members found this post helpful.
Old 12-19-2017, 10:40 AM   #3
Habitual
LQ 5k Club
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: High Sierra
Posts: 8,969
Blog Entries: 36

Rep: Reputation: Disabled
ELK.
 
1 members found this post helpful.
Old 01-04-2018, 02:16 AM   #4
ranmanh
LQ Newbie
 
Registered: Apr 2010
Posts: 10

Original Poster
Rep: Reputation: 0
Hi Guys,

Very much appreciated for your suggestions.

I had look into everything and spent some "quality time" around those + others.

In the end I have decided to go fo Graylog2.
Main Reasons:

* I need to filter + have some alerting . Graylog2 got both.
* As habitual suggested: ELK. Graylog2 got the main ELK components + the alerting and I do not have to deal with licensing in terms of security + other features. Nevertheless ElasticSearch is running underneath, so same principle.
* After a bit of a fight, I got GROK splitting the syslog files into meaningful chunks so I can filter and alter up the the string. (It wasn't too hard there...)


I am with frankbell on Nagios. That's the most common and probably pretty down the ground one. Even using Graylog2, I am thinking on how to also get Nagios to complete all those missing parts of information I still need.

Many thanks to everyone and I wanted to get back to you with the outcome of my research and all your suggestions.
 
Old 01-04-2018, 10:22 AM   #5
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,078
Blog Entries: 4

Rep: Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165Reputation: 3165
In my experience, tools like Nagios excel at being operational monitors.

If you want to do more intensive analysis of log-file data, I suggest that the best way to approach the task is to do it as a true statistics project. SO carried this interesting and detailed forum-post on Logfile analysis in 'R'. Many more such articles await your search.

In many cases, I have been most successful by attaching application-specific instrumentation to a process, usually arranging for it to write to a pipe that is quickly pumped by another process into a set of static files. For example, a workflow-management system might write "event" records at key points in the flow. First-stage analysis tools then assimilate these records into "wide" records that capture all of the salient data about each work-unit. Subsequent analysis is based on random samples taken from this dataset, and it is geared toward "testing some specific hypothesis, or objective." (For instance: "all class-B jobs should complete in less than four seconds, 95% of the time, and with a standard deviation of no more than 2." Pass/Fail: did this occur?)

Although "R" has modest data-capacity relative to some other tools, the fact that it is a true programming language gives it powerful flexibility for such investigations.

Last edited by sundialsvcs; 01-04-2018 at 10:27 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
are there any great snort log analyzers baronobeefdip Linux - Security 1 02-23-2013 08:29 PM
LXer: Top 8 Log Analyzers LXer Syndicated Linux News 0 12-23-2012 12:40 PM
GUI Log analyzers 999michal Linux - Server 1 04-30-2009 08:34 AM
Any good free log analyzers? aquaboot Ubuntu 4 08-21-2005 08:34 AM
log analyzers varun_saa Mandriva 2 01-24-2005 04:32 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 07:43 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration