syslog-ng3.2 central log server

JonDoe · 09-20-2011, 06:41 AM

Greetings,

I'm having 'problems' with my setup of my central syslog server using syslog-ng.
In short: I work for a company that has about 400 Unix servers(RedHat, SuSe and Solaris10). I'm asked to create a log file for each of these operating systems.
All clients are currently using the default syslogd, syslog-ng is only running on the central syslog.

What I have in mind is:
-------------------------
for every server in a certain "list"-file (eg: /opt/syslog-ng/all-redhat-servers.list) syslog-ng should move all events from all servers inside that list to logfile X. All logs coming from other hosts(Suse) go to logfile Y, and the same for Solaris. I can handle additional filtering afterwards(facility,...)

Thus instead of typing:
filter f_redhat {
host("hostA") or
host("hostB") or
host("hostC") or
...
};

I'd like to have something like: (Bash-y written)
RedHat=/path/to/list
filter f_redhat {
host("$RedHat")
};

log {
source(s_udp_514); filter(f_redhat); destination(d_redhat)
};

Bearing in mind that more people will be using it aswell... and we are speaking of a lot of hosts, it would seem a lot more manageable to add all my hosts into a file. It also makes it easier when we add/remove servers from our park.

I was wondering if it is even possible to do it? I can't seem to find a way to do it, besides adding every host manually.
Thx in advance!

Kind regards,

acid_kewpie · 09-20-2011, 07:36 AM

why would you define the hosts at all? They have their hostname in the syslog message, and syslog recognises it. just use the $HOST_FULL variable, or similar, and jobs done.

But then I read the question again... per operating system?? that's just dumb. I would genuinely suggest yo ulook to change that requirement, it's stupid. If utterly forced into it, I would possibly look to present directories per operating system with a handful of symlinks and leave syslog alone. still doesn't give a single file, but again, that's daft.

btw, if you want a really sexy log server, you might want to look at splunk which can probably get you lots of plaudits for a shiny search engine experience for probably no cost.

acid_kewpie · 09-20-2011, 07:41 AM

oh, how about this... listen on different ports, 5140, 5141 and 5142, one per OS. that'll do it with no farting about on the server. Still a daft requirement though.

JonDoe · 09-20-2011, 08:05 AM

Quote:

Originally Posted by acid_kewpie

why would you define the hosts at all? They have their hostname in the syslog message, and syslog recognises it. just use the $HOST_FULL variable, or similar, and jobs done.

But then I read the question again... per operating system?? that's just dumb. I would genuinely suggest yo ulook to change that requirement, it's stupid. If utterly forced into it, I would possibly look to present directories per operating system with a handful of symlinks and leave syslog alone. still doesn't give a single file, but again, that's daft.

btw, if you want a really sexy log server, you might want to look at splunk which can probably get you lots of plaudits for a shiny search engine experience for probably no cost.

splunk is indeed a very nice option for a central log server. However, I'm being asked to check multiple options and syslog-ng is one of them. Making splunk not a valid option right now.

On the other hand, I don't really get you point. I plan on making several folders - based on the OS - with again folders into them - based on an application. That last folder will contain the actual log. But, in order to do so I need to get some sort of filter that can filter out the hosts coming via udp.
Basically, what I aim to get is that servers, in a certain environment, running app1 put their events in the appropriate logfile.
so if server1-50 run app1, and 10 of them are development host, then their events should should go to .RedHat/app1/app1_dev.log

I hope that clarifies what my goal is.

tree:
.
|-- RedHat
| |-- app1
| | |-- app1_dev.log
| | `-- app1_uat.log
| |-- app2
| | `-- app2.log
| |-- app3
| | |-- app3_dev.log
| | `-- app3_uat.log
| `-- redhat.log
|-- Solaris10
| |-- app4
| | `-- app4_dev.log
| `-- solaris10.log
|-- Solaris8
| `-- solaris8.log
|-- SuSe
| `-- suse.log
`-- messages ===> dumpfile, catch'em all rule puts everything in here

acid_kewpie · 09-20-2011, 08:08 AM

I would really think that OS should not (be allowed to) matter. You can possibly log using different syslog facilities to distinguish the application traffic, but I don't see the need to an arbitrary division at OS level.

These sorts of solutions can often depend a lot on the rest of the management environment. Running things through puppet, for example, can make you devise a system that would be a nightmare without it, but easy with it.

JonDoe · 09-20-2011, 08:45 AM

Quote:

Originally Posted by acid_kewpie

I would really think that OS should not (be allowed to) matter. You can possibly log using different syslog facilities to distinguish the application traffic, but I don't see the need to an arbitrary division at OS level.

These sorts of solutions can often depend a lot on the rest of the management environment. Running things through puppet, for example, can make you devise a system that would be a nightmare without it, but easy with it.

true, an OS-layer is not exactly required. but it makes it more structured.
How would you distinguish application traffic based on facilities? A server running an application can have several facility-errors(0-15).
Since I know what server runs what application i can easily do the manual filtering using the above syntax host(hostA). But instead I just want to replace the numerous definitions with 1 rule.
Is host("$server" = /path/to/list) a possibilty?

kind regards,

acid_kewpie · 09-20-2011, 09:28 AM

Not in syslog-ng it's not. you would identify by facility by setting the facility within the app, or change the facility when the syslog client layer sends it. Again though, I would really suggest dropping the OS identification. There are so many arbitrary ways to categorize data, and it's seldom actually useful.

JonDoe · 09-21-2011, 03:51 AM

Quote:

Originally Posted by acid_kewpie

Not in syslog-ng it's not. you would identify by facility by setting the facility within the app, or change the facility when the syslog client layer sends it. Again though, I would really suggest dropping the OS identification. There are so many arbitrary ways to categorize data, and it's seldom actually useful.

I agree on the OS-identification, however: I'm 'gently' forced into it

Same goes for applications.
The actual problem is that there are architects that create a scheme(and do nothing besides that) and that I'm stuck with figuring it out...
Either way,how can you categorize data then?(bearing in mind that I need to be alter to filter out the hosts)

I was thinking something like this then:
we have a certain naming convention in which I can use host(host*)-alike syntax. Based, on that I can already do some Linux & solaris filtering, giving some sort of OS-split-up.
However, I need to be able to redirect events coming from server X to ../App X/whatever.log. But I don't see me changing every application's facility. What I can do is put the bulk of our apps into a log, eventually adding with additional filters,eg facility 0 or levels that lead to other logs. But for our major apps I'll need to think around something else, perhaps that in a few cases I can fallback on a wildcard.
However, it's not very manageable...
That is why I'd love to work with a variable that is pointed to a file. If I install/remove a host, all you have to do is add/remove it from that file. If not, I have to manually add them to syslog-ng.conf, and I'm not really into that.

Kr,

Reuti · 09-21-2011, 04:56 AM

You can split all logs you get into different files in syslog-ng by:

Code:

filter f_dsm_error  { facility(daemon) and level(err) and program(dsm); };
filter f_dsm_sched  { facility(daemon) and level(info) and program(dsm); };
filter f_common     { not program(dsm); };

#
# Log all remote messages in a file whoes name include the origin
#

destination remote_logfiles { file("/var/log/$HOST.messages"); };
log { source(remote); filter(f_common); destination(remote_logfiles); };

destination remote_dsmsched { file("/var/log/dsm/$HOST.dsmsched.log" perm(0644)); };
log { source(remote); filter (f_dsm_sched); destination(remote_dsmsched); };
destination remote_dsmerror { file("/var/log/dsm/$HOST.dsmerror.log" perm(0644)); };
log { source(remote); filter (f_dsm_error); destination(remote_dsmerror); };

I use this to store the messages for each host in a separate file and filter all logs belonging to the backup from IBM’s TSM in a dedicated subdirectory.