LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 12-20-2018, 09:35 AM   #1
1s440
Member
 
Registered: Mar 2018
Posts: 266

Rep: Reputation: Disabled
ELK, Kibana, logstash


Hello all,

I have come across with Elasticsearch and have gone through the websites but nothing goes in to my mind. I understood its for logging analysis, but is it only useful for cloud infrastructure like AWS, google? please share me your ideas.
 
Old 12-20-2018, 04:39 PM   #2
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
ELK stands for ElasticSearch Logstash Kibana. It should be called LEK, because the dataflow starts with Logstash, but I guess ELK sounds better.

Logstash is a tool that standardizes log messages. Standardized log messages go into Elasticsearch, which puts them into a database that is optimized for very fast read access. Kibana is a graphical presentation tool that allows you to correlate log information in many ways.

Useful for any infrastructure where you have several log files from several servers that you want to analyze and correlate.

Any specific question?

Last edited by berndbausch; 12-20-2018 at 04:41 PM.
 
Old 12-21-2018, 01:57 AM   #3
1s440
Member
 
Registered: Mar 2018
Posts: 266

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by berndbausch View Post
ELK stands for ElasticSearch Logstash Kibana. It should be called LEK, because the dataflow starts with Logstash, but I guess ELK sounds better.

Logstash is a tool that standardizes log messages. Standardized log messages go into Elasticsearch, which puts them into a database that is optimized for very fast read access. Kibana is a graphical presentation tool that allows you to correlate log information in many ways.

Useful for any infrastructure where you have several log files from several servers that you want to analyze and correlate.

Any specific question?
Thank you so much for the answer. Yes, In our infrastructure we have many apache logs to monitor so would ELK be the best to monitor all logs with in the infrastructure?
 
Old 12-21-2018, 03:26 AM   #4
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
I can’t serve as a consultant, even more so because I only have a one-line description of a complex problem, but I would check how much work and computing resources are required to set up and maintain an ELK installation and compare that to the scale of your problem.

Since you only need to manage Apache logs, perhaps there are more focused, easier to use and leaner solutions.
 
Old 12-24-2018, 06:38 AM   #5
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6 / 7
Posts: 3,474

Rep: Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553
If you're looking for a commercial solution take a look at Splunk, we use it to ingest around 50Gb of log files a day.
 
Old 01-23-2019, 03:11 AM   #6
1s440
Member
 
Registered: Mar 2018
Posts: 266

Original Poster
Rep: Reputation: Disabled
Hello all,

I have installed ELK on my system (Debian version: 8), after the installation when i try to connect to my website https://example.com it asks for username and password and I am unable to authenticate it. Can anyone help me with this.
Code:
root@mytest# cat /etc/nginx/htpasswd.kibana
admin:$apr1$TBz5AlRo$WkDPQA6XWwFtiZd5FldKH1
admin:$apr1$mb.lKAxP$PeIsVdVAj7T3Sv61LRVJu0
when i checked the error log
Code:
2019/01/23 10:02:44 [error] 5583#5583: *11 user "admin" was not found in "/etc/nginx/htpasswd.kibana", client: 94.209.200.140,, server: _, request: "GET / HTTP/2.0", host: "example.com"

Last edited by 1s440; 01-24-2019 at 02:58 AM.
 
Old 01-23-2019, 10:04 AM   #7
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6 / 7
Posts: 3,474

Rep: Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553
Which part of the error message "testuser" was not found in "/etc/nginx/htpasswd.kibana" doesn't make sense?
 
Old 02-14-2019, 08:48 AM   #8
1s440
Member
 
Registered: Mar 2018
Posts: 266

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by TenTenths View Post
If you're looking for a commercial solution take a look at Splunk, we use it to ingest around 50Gb of log files a day.
I have installed Splunk on my server, So I can monitor all the clients on a single server?

Last edited by 1s440; 02-14-2019 at 09:35 AM.
 
Old 02-14-2019, 09:51 AM   #9
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,617

Rep: Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963
Quote:
Originally Posted by 1s440 View Post
I have installed Splunk on my server, So I can monitor all the clients on a single server?
Have you CONFIGURED all your servers to send the logs to one? Have you read the Splunk documentation?? And you do realize that Splunk requires configuration, just like Kibana does too?
 
Old 02-15-2019, 02:49 AM   #10
1s440
Member
 
Registered: Mar 2018
Posts: 266

Original Poster
Rep: Reputation: Disabled
I have created an instance to monitor apache access log as per the documentation, unforunately i am unable to see the logs when i start seraching for it. Any idea?
 
Old 02-15-2019, 06:27 AM   #11
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,617

Rep: Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963
Quote:
Originally Posted by 1s440 View Post
I have created an instance to monitor apache access log as per the documentation, unforunately i am unable to see the logs when i start seraching for it. Any idea?
Besides "You didn't configure it properly"...no, none at all. And an 'instance' of what? This thread was started about Kibana, and after you seemed to bypass a very simple error message, (that TenTenths explained), you said you the installed Splunk.

Again: you have to configure things before they'll work. If you're not seeing the logs you want (but see others), that narrows down the list of potential problems considerably.
 
Old 02-15-2019, 07:21 AM   #12
1s440
Member
 
Registered: Mar 2018
Posts: 266

Original Poster
Rep: Reputation: Disabled
I configured the app Splunk Add-on for Apache Web Server unfortunately when i try to launch app it redirects to the page where the application not found. when i download and upload files it says successfully done but still seems the application doesnot exist.
 
Old 02-15-2019, 07:27 AM   #13
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,617

Rep: Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963Reputation: 7963
Quote:
Originally Posted by 1s440 View Post
I configured the app Splunk Add-on for Apache Web Server unfortunately when i try to launch app it redirects to the page where the application not found. when i download and upload files it says successfully done but still seems the application doesnot exist.
Right; again, you have **NOT INSTALLED OR CONFIGURED IT CORRECTLY**

It's giving you errors and telling you it's not running...why would you then think that you'd see log file entries as if it WAS running? Again, you are providing NO DETAILS about what you've done/tried how you installed, to what version/distro of Linux. All things you've been asked several times before. Since you are now asking about Splunk, you need to open a new thread where you provide all of the details you SHOULD be providing. Do this after you read and follow the Splunk installation instructions (which you haven't yet).
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Grafana and Kibana DennyY Linux - Newbie 6 08-04-2018 03:46 AM
Elastic search stats to Kibana bmxakias Linux - Server 1 05-25-2018 11:40 AM
LXer: Monitoring Your Picluster with Elasticsearch and Kibana LXer Syndicated Linux News 0 01-18-2018 05:13 PM
LXer: Elasticsearch and Kibana : installation and basic usage on Ubuntu 16.04 LXer Syndicated Linux News 0 03-03-2017 03:03 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 01:53 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration