LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 05-21-2014, 11:46 AM   #31
dkanbier
LQ Newbie
 
Registered: May 2014
Distribution: Fedora
Posts: 13

Rep: Reputation: Disabled

Quote:
Originally Posted by Habitual View Post
I hear you.

Thanks for all you've done.
No problem at all. If you have new entries in logstash.log at least logstash is working like it should. If not, checkout the debug output and see if you your inputs are registered:

Code:
{:timestamp=>"2014-05-20T20:22:49.200000+0200", :message=>"Registering file input", :path=>["/var/log/*.log", "/var/log/messages", "/var/log/syslog"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
And see if it discovers the files you attempt to monitor

Code:
{:timestamp=>"2014-05-20T20:22:49.386000+0200", :message=>"_discover_file_glob: /var/log/*.log: glob is: [\"/var/log/yum.log\", \"/var/log/anaconda.log\", \"/var/log/anaconda.storage.log\", \"/var/log/anaconda.program.log\", \"/var/log/dracut.log\", \"/var/log/boot.log\", \"/var/log/anaconda.ifcfg.log\", \"/var/log/anaconda.yum.log\"]", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}
{:timestamp=>"2014-05-20T20:22:49.393000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/yum.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.401000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/anaconda.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.411000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/anaconda.storage.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.418000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/anaconda.program.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.436000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/dracut.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.443000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/boot.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.448000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/anaconda.ifcfg.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.457000+0200", :message=>"_discover_file: /var/log/*.log: new: /var/log/anaconda.yum.log (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.467000+0200", :message=>"_discover_file_glob: /var/log/messages: glob is: [\"/var/log/messages\"]", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}
{:timestamp=>"2014-05-20T20:22:49.473000+0200", :message=>"_discover_file: /var/log/messages: new: /var/log/messages (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
{:timestamp=>"2014-05-20T20:22:49.486000+0200", :message=>"_discover_file_glob: /var/log/syslog: glob is: []", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}
If you're not sure I won't mind taking a look at the complete logfile. Cheers!
 
Old 05-21-2014, 12:00 PM   #32
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,569
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
Oh they're there:
Code:
grep "Registering file input" /var/log/logstash/logstash.log
{:timestamp=>"2014-05-21T08:23:14.999000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/web/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:23:15.003000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9a/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:23:15.006000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9b/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:28:11.573000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/web/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:28:11.580000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9a/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:28:11.584000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9b/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
Now that perms and the conf file has been updated to use
Code:
sincedb_path => "/opt/logstash/sincedb-access"
I was thinking of nuking via yum erase and cleaning leftovers in /opt/logstash and /var/lib/logstash/ and re-installing...
This seems to have worked once or twice.

Last edited by Habitual; 05-21-2014 at 12:01 PM.
 
Old 05-21-2014, 12:12 PM   #33
dkanbier
LQ Newbie
 
Registered: May 2014
Distribution: Fedora
Posts: 13

Rep: Reputation: Disabled
Quote:
Originally Posted by Habitual View Post
Oh they're there:
Code:
grep "Registering file input" /var/log/logstash/logstash.log
{:timestamp=>"2014-05-21T08:23:14.999000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/web/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:23:15.003000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9a/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:23:15.006000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9b/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:28:11.573000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/web/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:28:11.580000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9a/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
{:timestamp=>"2014-05-21T08:28:11.584000-0700", :message=>"Registering file input", :path=>["/var/log/remotes/cirrhus9b/*.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
Now that perms and the conf file has been updated to use
Code:
sincedb_path => "/opt/logstash/sincedb-access"
I was thinking of nuking via yum erase and cleaning leftovers in /opt/logstash and re-installing...
This seems to have worked once or twice.

Yes that might be a good idea, but I guess there is a change you'll end up in the exact same state.

One more thing, I guess you probably thought of this but are the actual logfiles in /var/log/remotes owned by root? Your previous posts only show the directory permissions, not of the actual files. Just a thought, if they are owned by root the logstash user can list but not read them

If it's possible check out that if you trigger a new log message it's picked up by logstash. In my case I also monitor /var/log/messages. When I trigger a log message:

Code:
logger TESTING
It's immediately picked up by logstash and visible in logstash.log (don't mind the "grokked" messages):

Code:
{:timestamp=>"2014-05-21T18:07:13.528000+0200", :message=>"Event now: ", :event=>#<LogStash::Event:0x23155c9e @accessors=#<LogStash::Util::Accessors:0x4f739b10 @store={"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, @lut={"type"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "type"], "host"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "host"], "path"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "path"], "message"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "message"], "timestamp"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "timestamp"], "logsource"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "logsource"], "program"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "program"], "tags"=>[{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, "tags"]}>, @data={"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, @cancelled=false>, :level=>:debug, :file=>"logstash/filters/grok.rb", :line=>"299"}
{:timestamp=>"2014-05-21T18:07:13.523000+0200", :message=>"writing sincedb (delta since last write = 2462)", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"177"}
{:timestamp=>"2014-05-21T18:07:13.533000+0200", :message=>"output received", :event=>{"message"=>["May 21 18:07:12 dev root: TESTING", "TESTING"], "@version"=>"1", "@timestamp"=>"2014-05-21T16:07:13.519Z", "type"=>"syslog", "host"=>"dev.kanbier.lan", "path"=>"/var/log/messages", "timestamp"=>"May 21 18:07:12", "logsource"=>"dev", "program"=>"root", "tags"=>["syslog", "grokked"]}, :level=>:debug, :file=>"(eval)", :line=>"43"}
So if you see this and it's not visible within elasticsearch (or Kibana talking to elasticsearch), the problem isn't logstash per se.
 
Old 05-21-2014, 01:30 PM   #34
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,569
Blog Entries: 1

Original Poster
Rep: Reputation: Disabled
clean logstash recipe

Well, we'll know tomorrow when indexes roll over, but for now we have re-initialized logstash with this recipe:
Code:
service logstash stop
rm -fr /var/lib/logstash/ /opt/logstash
rpm -Uvh /usr/src/logstash-1.4.1-1_bd507eb.noarch.rpm
touch /opt/logstash/sincedb-access
chown logstash:logstash /opt/logstash/sincedb-access
chmod -R 770 /opt/logstash/sincedb-access
chown -R logstash:logstash /var/log/remotes/
chmod -R 770 /var/log/remotes/
cp /root/logstash /etc/init.d/
/opt/logstash/bin/logstash -f  /etc/logstash/conf.d/logstash.conf --configtest
vi /opt/logstash/vendor/kibana/config.js # and adjust "http://fqdn:9200",
service logstash start
The only irregular "what the hell?" I see is:
Code:
stat -c%a /var/log/remotes/*
744
744
744
when I set them via c-line for 770

the init script I cp'd from /root is here... and I believe that's stock except for
Code:
args="agent -f ${LS_CONF_DIR}/logstash.conf
and line 153
where it was
Code:
stop && start
and is now
Code:
stop || start
logstash.conf is here...

You've been a tremendous help on this issue and I am grateful for it.

Have a Great Day!

Last edited by Habitual; 05-21-2014 at 01:34 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Centralized log setup awesant elasticsearch logstash and kibana3 LXer Syndicated Linux News 0 01-26-2014 07:12 PM
CentOS centralised logging, syslogd, rsyslog, syslog-ng, logstash sender? batfastad Linux - Server 4 11-29-2012 04:56 AM
Centos5 Ramdisk help cf500 Linux - General 8 02-22-2011 02:59 AM
centOS5.2 ekac Linux - Newbie 4 06-05-2009 08:53 AM
XEN on centos5 hackintosh Linux - Server 2 10-20-2007 12:11 AM


All times are GMT -5. The time now is 04:38 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration