LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-12-2014, 11:46 AM   #1
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
rsylog 5.8.6 sends duplicates to syslog-server


on the source host, I run grep for a bad guy's IP and I get this:
Code:
grep 85.110.235.101 /var/log/syslog /var/log/apache2/access.log
/var/log/syslog:Sep 11 16:39:23 web access: 85.110.235.101 - - [11/Sep/2014:16:39:21 -0700] "POST /xmlrpc.php HTTP/1.1" 403 2095 "http://www.domain.com/xmlrpc.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
/var/log/apache2/access.log:85.110.235.101 - - [11/Sep/2014:16:39:21 -0700] "POST /xmlrpc.php HTTP/1.1" 403 2095 "http://www.domain.com/xmlrpc.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
1 hit in each file.

the same grep on the syslog-server shows 3 identical entries:
Code:
grep 85.110.235.101 /kibana/web/access.log
Sep 11 16:39:23 web access: 85.110.235.101 - - [11/Sep/2014:16:39:21 -0700] "POST /xmlrpc.php HTTP/1.1" 403 2095 "http://www.domain.com/xmlrpc.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Sep 11 16:39:23 web access: 85.110.235.101 - - [11/Sep/2014:16:39:21 -0700] "POST /xmlrpc.php HTTP/1.1" 403 2095 "http://www.domain.com/xmlrpc.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Sep 11 16:39:23 web access: 85.110.235.101 - - [11/Sep/2014:16:39:21 -0700] "POST /xmlrpc.php HTTP/1.1" 403 2095 "http://www.domain.com/xmlrpc.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
and nothing in the /kibana/web/syslog file for that IP?

The apache.conf for the site has this access.log directive:
Code:
CustomLog /var/log/apache2/access.log combined
I'm sending access.log to the syslog-server using a "watchfile"(.conf) under
/etc/rsyslog.d/watchfile.conf
Code:
$ModLoad imfile
# apache error.log
$InputFileName /var/log/apache2/error.log
$InputFileTag error:
$InputFileStateFile state_file_error_apache
$InputFileFacility local7
$InputRunFileMonitor
$InputFilePollInterval 1

# apache access.log
$InputFileName /var/log/apache2/access.log
$InputFileTag access:
$InputFileStateFile state_file_access_apache
$InputFileFacility local7
$InputRunFileMonitor
$InputFilePollInterval 1

if $programname == 'access' then @@logserver.domain.com
& stop
if $programname == 'errors' then @@logserver.domain.com
& stop
Input for this hosts' file in /etc/logstash/conf.d/logstash.conf on the syslog-server is
Code:
input 
{

file 
    {
    type => "syslog"
    path => [ "/kibana/web/*.log" ]
    start_position => beginning
    sincedb_path => "/opt/logstash/sincedb-access"
    }
...
}
output 
   {
   stdout { codec => rubydebug }
   elasticsearch 
    { 
    embedded => true 
    } 
   }

This setup is producing duplicate and triplicate 'hits' in my Kibana interface.
I'm not sure which element of this configuration is causing this issue and I'm stumped.

Thanks.

Forgot as these may be important:
from the source hosts' /etc/rsyslog.conf
Code:
#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

###########################
#### GLOBAL DIRECTIVES ####
###########################

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$RepeatedMsgReduction on

# Set the default permissions for all log files.
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup adm

# Where to place spool files
$WorkDirectory /var/spool/rsyslog

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

### Sending to logserver.domain.com ###
*.* @@logserver.domain.com:514
I had UDP ("@") specified but I just changed it to
Code:
*.* @@logserver.domain.com:514
and it didn't seem to matter

Last edited by Habitual; 09-12-2014 at 12:48 PM.
 
Old 09-12-2014, 02:18 PM   #2
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
Bah.
I installed rsyslog5 on a CentOS 5.10 host and sent it to @@logserver.domain.com:514
and I get a /kibana/vds64_centos55/rsyslogd.log

with... <wait for it...>
Triplicates

So, at least I know it's the syslog-server configuration?

Last edited by Habitual; 09-12-2014 at 03:13 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Server sends emails ever 30 minutes NotionCommotion Linux - Newbie 3 04-26-2014 09:06 PM
Chemistry problem: Identify duplicates and non-duplicates within TWO sdf files robertselwyne Programming 5 12-09-2011 06:20 AM
My email server sends mails but does not receive hubergeek Linux - Networking 2 07-14-2010 08:27 AM
sendmail server just sends mail from localhost reneve59 Linux - Server 3 09-21-2007 09:32 PM
LXer: Centralized Syslog Server Using syslog-NG LXer Syndicated Linux News 0 04-28-2006 06:21 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration