Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
and I can start it, but within 1 minute it dies and status shows "Logstash Daemon dead but pid file exists"
Nothing is written to the /etc/logstash/logstash.log or any other log file in /var/log about the "event"
/opt/logstash/bin/logstash -vv barfs with
Code:
+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug. |
| You can find help with this problem in a few places: |
| |
| * chat: #logstash IRC channel on freenode irc. |
| IRC via the web: http://goo.gl/TI4Ro |
| * email: logstash-users@googlegroups.com |
| * bug system: https://logstash.jira.com/ |
| |
+---------------------------------------------------------+
The error reported is:
undefined method `+' for nil:NilClass
/opt/logstash/lib/logstash/agent.rb:100:in `execute'
/opt/logstash/lib/logstash/runner.rb:190:in `run'
org/jruby/RubyProc.java:271:in `call'
/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/task.rb:12:in `initialize'
Guys on #logstash@Freenode said the word 'bug'...and one person said OpenJDK-7 is not the 'fix'
I'm wondering if I need to install rubygems...?
The strange part is, I had ElasticSearch+kibana+rsyslog data in Kibana, but then I tried to be 'clever' and change the rsyslog>logstash config and now, zilch data.
I had success earlier using just plain ol' rsyslog as an input but now, zilch.
Do I even need logstash for regular rsyslogd files > ES?
I have since disabled rsyslog on my 3 hosts since my /var/log/messages file on the rsyslog-server filled up hda1 at 11G
Oh the horror.
I think entries in /etc/rsyslog.conf can suppress the 'sending' to /var/log/messages with such as
Code:
$ModLoad imfile
$InputFileName /var/log/rsyslog_custom/web/httpd.log
$InputFileTag c9web-httpd-log:
$InputFileStateFile state-c9web-httpd-log
$InputRunFileMonitor
$InputFilePollInterval 1
if $programname == "c9web-httpd-log" then @fqdn:5544
if $programname == "c9web-httpd-log" then stop
I'm still very much interested in a solution, and I'm will to turn one back "on" for testing a solution.
Thank you for your time. I really appreciate the help.
If you're downloading and hoping to use Kibana3, like me, you can't use RPMs for elasticsearch and logstash.
All repos I found in howtos on the net for logstash > rsyslog > elasticsearch > kibana DON'T WORK, at least not on CentOS5
If you're downloading and hoping to use Kibana3, like me, you can't use RPMs for elasticsearch and logstash.
All repos I found in howtos on the net for logstash > rsyslog > elasticsearch > kibana DON'T WORK, at least not on CentOS5
Since these are included with or
You can install a logstash*.rpm, you just can't use
Code:
service logstash start
to get it going. At least, I couldn't. It'd start, then die about 1 minute later as mentioned in 1st post.
I'm wondering what the permissions are on the config file /etc/logstash/conf.d/logstash.conf.
Could you give the output of "ls -l /etc/logstash/conf.d/logstash.conf"?
And as which user do you start logstash when using the command line? If you're not doing so already you could try starting it by hand with the "-v" option as the logstash user. The -vv option also gives me the same error.
I'm wondering what the permissions are on the config file /etc/logstash/conf.d/logstash.conf.
Could you give the output of "ls -l /etc/logstash/conf.d/logstash.conf"?
Code:
-rw-r--r-- 1 root root 364 May 14 07:57 /etc/logstash/conf.d/logstash.conf
Quote:
And as which user do you start logstash when using the command line? If you're not doing so already you could try starting it by hand with the "-v" option as the logstash user. The -vv option also gives me the same error.
sudo -u logstash bash
bash: /root/.bashrc: Permission denied
bash-3.2$ id
uid=104(logstash) gid=164(logstash) groups=164(logstash)
bash-3.2$ pwd
/root
$ ls -dl /opt/logstash
drwxr-xr-x 9 logstash logstash 4096 May 13 13:36 /opt/logstash
It gets really ugly using the -v switch as the logstash user:
Code:
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -v
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:warn}
Registering file input {:path=>["/var/log/remotes/web/*.log"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_38f4dc076e48e774d7872c63b279d079", :path=>["/var/log/remotes/web/*.log"], :level=>:info}
Registering file input {:path=>["/var/log/remotes/cirrhus9a/*.log"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_f3f1a09b7195f62d15e9bfe6d07044c7", :path=>["/var/log/remotes/cirrhus9a/*.log"], :level=>:info}
Registering file input {:path=>["/var/log/remotes/cirrhus9b/*.log"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_42f4e99991f750cf42bd5d2e154ef9de", :path=>["/var/log/remotes/cirrhus9b/*.log"], :level=>:info}
Pipeline started {:level=>:info}
Starting embedded Elasticsearch local node. {:level=>:info}
log4j, [2014-05-15T06:23:28.585] INFO: org.elasticsearch.node: [Raa of the Caves] version[1.1.1], pid[26441], build[f1585f0/2014-04-16T14:27:12Z]
log4j, [2014-05-15T06:23:28.586] INFO: org.elasticsearch.node: [Raa of the Caves] initializing ...
log4j, [2014-05-15T06:23:28.604] INFO: org.elasticsearch.plugins: [Raa of the Caves] loaded [], sites []
Exception in thread ">output" org.elasticsearch.ElasticsearchIllegalStateException: Failed to obtain node lock, is the following location writable?: [/root/data/elasticsearch]
at org.elasticsearch.env.NodeEnvironment.<init>(org/elasticsearch/env/NodeEnvironment.java:114)
at org.elasticsearch.node.internal.InternalNode.<init>(org/elasticsearch/node/internal/InternalNode.java:150)
at org.elasticsearch.node.NodeBuilder.build(org/elasticsearch/node/NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(org/elasticsearch/node/NodeBuilder.java:166)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:622)
at RUBY.start_local_elasticsearch(/opt/logstash/lib/logstash/outputs/elasticsearch.rb:299)
at RUBY.register(/opt/logstash/lib/logstash/outputs/elasticsearch.rb:247)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
at RUBY.outputworker(/opt/logstash/lib/logstash/pipeline.rb:220)
at RUBY.start_outputs(/opt/logstash/lib/logstash/pipeline.rb:152)
at java.lang.Thread.run(java/lang/Thread.java:701)
Caused by: java.io.IOException: failed to obtain lock on /root/data/elasticsearch/nodes/49
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:99)
at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:150)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304)
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:52)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
at org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:182)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.ast.IfNode.interpret(IfNode.java:116)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:112)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:164)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:286)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:85)
at org.jruby.RubySymbol$1.yieldInner(RubySymbol.java:445)
at org.jruby.RubySymbol$1.yield(RubySymbol.java:465)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
at org.jruby.RubyArray.each(RubyArray.java:1613)
at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
at org.jruby.ast.CallNoArgBlockPassNode.interpret(CallNoArgBlockPassNode.java:53)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:182)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
at org.jruby.runtime.Block.call(Block.java:101)
at org.jruby.RubyProc.call(RubyProc.java:290)
at org.jruby.RubyProc.call(RubyProc.java:228)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.IOException: Cannot create directory: /root/data/elasticsearch/nodes/49
at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:171)
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:81)
... 61 more
Interrupt received. Shutting down the pipeline. {:level=>:warn}
Sending shutdown signal to input thread {:thread=>#<Thread:0x6b211be8 run>, :level=>:info}
Plugin is finished {:plugin=><LogStash::Inputs::Stdin type=>"stdin-type">, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x38d4d428 sleep>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x7e7be20b sleep>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x85168c6 sleep>, :level=>:info}
probably since I started it as root in the first place.
Code:
# ls -dl /root/data/elasticsearch
drwxr-xr-x 3 root root 4096 May 14 06:41 /root/data/elasticsearch
I'm just trying to help you troubleshoot, if you're done with it I'm fine with that too just let me know! But I like to discover what goes wrong
I think the error code we get is valid when we run logstash as the logstash user:
Code:
Exception in thread ">output" org.elasticsearch.ElasticsearchIllegalStateException: Failed to obtain node lock, is the following location writable?: [/root/data/elasticsearch]
The user logstash shouldn't be able to access anything in /root, so that's good!
Now before jumping to conclusions I'd also like to know if you installed a separate elasticsearch server or if you'd like to use the embedded one in Logstash. If I look at the logstash configuration you use, you're trying to use a separate elasticsearch server. If so, could we get the configuration on that?
Also, I think you can get some more output from logstash by modifying it's init script /etc/init.d/logstash. I simply added a --debug option in the DAEMON_OPTS line:
Now before jumping to conclusions I'd also like to know if you installed a separate elasticsearch server or if you'd like to use the embedded one in Logstash. If I look at the logstash configuration you use, you're trying to use a separate elasticsearch server. If so, could we get the configuration on that?
since EL is provided by the logstash.rpm, I used no other. I had previously installed one but removed it when I found the logstash.rpm has one included.
Quote:
Originally Posted by dkanbier
Also, I think you can get some more output from logstash by modifying it's init script /etc/init.d/logstash. I simply added a --debug option in the DAEMON_OPTS line:
Edit: Didn't work. only produced 2 additional files in /var/log/logstash
logstash.err - 0 bytes
logstash.stderr - barking about "-- -" not allowed. There is/was no "-- -" just --debug" added
I'm actually surprised Logstash keeps running if you run it by hand with the logstash configuration you posted. I always thought you needed to specify the "embedded" option in your output section if you want to log to the embedded elasticsearch.
I'm very curious if there is process listening on the elasticsearch port (default 9200 and 9300 I think) if you're running logstash in screen:
Code:
netstat -tulpn | grep LISTEN
If there is nothing listening, you could try this for the output section:
I'd be really interested in is the /etc/init.d/logstash provided by logstash-1.4.0-1_c82dc09.noarch works on your version of CentOS or Redhat/SuSE/other rpm-based OS as logstash-1.4.1-1_bd507eb's version of it does not have a "DAEMON_OPTS=" line in that rpm's /etc/init.d/logstash.
Perhaps it will "just work" when I reboot my host later today, after a zabbix upgrade since
Code:
logstash 0:off 1:off 2:on 3:on 4:on 5:on 6:off
?
Anyway, logstash 1.4.1 with Kibana3.x and rsyslogd 7.6.3 are all working in conjunction over here. So that idiosyncrasy is the only thing left.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.