logstash-1.4.0-1 on CentOS 5.10
I installed logstash 1.4.0 from elasticsearch HQ
and I can start it, but within 1 minute it dies and status shows "Logstash Daemon dead but pid file exists" Nothing is written to the /etc/logstash/logstash.log or any other log file in /var/log about the "event" /opt/logstash/bin/logstash -vv barfs with Code:
+---------------------------------------------------------+ I'm wondering if I need to install rubygems...? The strange part is, I had ElasticSearch+kibana+rsyslog data in Kibana, but then I tried to be 'clever' and change the rsyslog>logstash config and now, zilch data. I had success earlier using just plain ol' rsyslog as an input but now, zilch. Do I even need logstash for regular rsyslogd files > ES? References for the initial install: http://blog.basefarm.com/blog/how-to...rface-on-rhel/ and http://sharadchhetri.com/2014/03/01/...el-6-centos-6/ Anyone able to help me get rsyslog data into ES/Kibana? Thanks for your time. |
Quote:
The only requirement is JAVA Let's try the basic testing as it is mentioned in the above link before going to use your own logstash configuration |
Tests with
Code:
bin/logstash -e 'input { stdin { } } output { elasticsearch { host => localhost } }' Code:
/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/logstash.conf --configtest Code:
/usr/src/logstash-1.4.0/bin/logstash -f /usr/src/logstash-1.4.0/apache.conf web & I even did the shakespeare.json import but that never showed up in the Kibana interface either. Starting the 'service' is where it chokes after about 1 minute and dies with mentioned output. Thanks. |
can share your logstash configuration file here?
|
from the terrible notes I took on it, I believe I tried this one last in etc/logstash/conf.d/logstash.conf:
Code:
input { I have since disabled rsyslog on my 3 hosts since my /var/log/messages file on the rsyslog-server filled up hda1 at 11G Oh the horror. I think entries in /etc/rsyslog.conf can suppress the 'sending' to /var/log/messages with such as Code:
$ModLoad imfile Thank you for your time. I really appreciate the help. |
If you're downloading and hoping to use Kibana3, like me, you can't use RPMs for elasticsearch and logstash.
All repos I found in howtos on the net for logstash > rsyslog > elasticsearch > kibana DON'T WORK, at least not on CentOS5 Since these are included with https://download.elasticsearch.org/k...-latest.tar.gz or https://download.elasticsearch.org/k...a-3.0.1.tar.gz You can install a logstash*.rpm, you just can't use Code:
service logstash start The only command that works is: Code:
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf http://blog.basefarm.com/blog/how-to...rface-on-rhel/ http://www.denniskanbier.nl/blog/log...-and-centos-6/ Enjoy the Goodness! |
Quote:
Could you give the output of "ls -l /etc/logstash/conf.d/logstash.conf"? And as which user do you start logstash when using the command line? If you're not doing so already you could try starting it by hand with the "-v" option as the logstash user. The -vv option also gives me the same error. |
Quote:
Code:
-rw-r--r-- 1 root root 364 May 14 07:57 /etc/logstash/conf.d/logstash.conf Quote:
A result of the rpm install: Code:
logstash:x:104:164:logstash:/opt/logstash:/sbin/nologin Code:
sudo -u logstash bash Code:
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -v Code:
# ls -dl /root/data/elasticsearch Code:
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf Thanks. |
I'm just trying to help you troubleshoot, if you're done with it I'm fine with that too just let me know! But I like to discover what goes wrong :D
I think the error code we get is valid when we run logstash as the logstash user: Code:
Exception in thread ">output" org.elasticsearch.ElasticsearchIllegalStateException: Failed to obtain node lock, is the following location writable?: [/root/data/elasticsearch] Now before jumping to conclusions I'd also like to know if you installed a separate elasticsearch server or if you'd like to use the embedded one in Logstash. If I look at the logstash configuration you use, you're trying to use a separate elasticsearch server. If so, could we get the configuration on that? Also, I think you can get some more output from logstash by modifying it's init script /etc/init.d/logstash. I simply added a --debug option in the DAEMON_OPTS line: Code:
DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug" One last thing, this is how I change to users without a shell. A bit less messy ;) Code:
[root@dev var]# su - logstash -s /bin/bash |
Quote:
Quote:
It should log to /var/log/logstash now. Quote:
Thanks for the feedback. Edit: the rpm-provided /etc/init.d/logstash doesn't have a DAEMON_OPTS It does have Code:
args="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}" Edit: Didn't work. only produced 2 additional files in /var/log/logstash logstash.err - 0 bytes logstash.stderr - barking about "-- -" not allowed. There is/was no "-- -" just --debug" added |
No problem!
I'm actually surprised Logstash keeps running if you run it by hand with the logstash configuration you posted. I always thought you needed to specify the "embedded" option in your output section if you want to log to the embedded elasticsearch. I'm very curious if there is process listening on the elasticsearch port (default 9200 and 9300 I think) if you're running logstash in screen: Code:
netstat -tulpn | grep LISTEN Code:
output { Quote:
|
Code:
netstat -plaunt | grep 92 | grep java Code:
cat /etc/logstash/conf.d/logstash.conf I've tried all of these: http://download.elasticsearch.org/lo...tos.noarch.rpm https://download.elasticsearch.org/l...c09.noarch.rpm https://download.elasticsearch.org/l...7eb.noarch.rpm |
Quote:
Did you try to run logstash as a service when you added the --debug option? Did it log anything? |
Quote:
I'd be really interested in is the /etc/init.d/logstash provided by logstash-1.4.0-1_c82dc09.noarch works on your version of CentOS or Redhat/SuSE/other rpm-based OS as logstash-1.4.1-1_bd507eb's version of it does not have a "DAEMON_OPTS=" line in that rpm's /etc/init.d/logstash. Perhaps it will "just work" when I reboot my host later today, after a zabbix upgrade since Code:
logstash 0:off 1:off 2:on 3:on 4:on 5:on 6:off Anyway, logstash 1.4.1 with Kibana3.x and rsyslogd 7.6.3 are all working in conjunction over here. So that idiosyncrasy is the only thing left. Have a Great Day! |
Aha!
Fixed service logstash (start|status) using Code:
mkdir /tmp/test Code:
service logstash status My current indexes are gone using Code:
curl http://localhost:9200/_aliases?pretty=1 I restarted using Code:
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf I suppose I could nuke logstash-1.4.1-1_bd507eb and install logstash-1.4.0-1_c82dc09.noarch.rpm To be continued... Have a Great Day. |
All times are GMT -5. The time now is 11:00 PM. |