LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-15-2010, 06:42 AM   #1
ziadh
Member
 
Registered: Aug 2010
Location: France, Paris
Distribution: Solaris10
Posts: 88

Rep: Reputation: 0
syslog


Hello,
one simple question: syslog is used to store simple log files or we can manage them too?

well, the thing is, that i need to run a software (like syslog) to collect my logs and put them in order and organize them so it makes them "understandable"

i have been told that syslog can do the job and that it doesnt need a complex configuration to work
 
Old 09-15-2010, 06:57 AM   #2
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
What do you mean by "organized and understandable"?

Sysloggers create the logs, and log stuff to them, but how understandable these logs are depends much on what application(s) are writing to them, and how understandably the logs are created/written. The logger doesn't make the logs more or less understandable. And, syslog is pretty easy to configure, yes.

For keeping your /var/log folder organized, there's the `logrotate` tool, which can compress or delete or mail the logs.

So, please clarify exactly what it is you are looking to do? Thanks!
 
Old 09-15-2010, 07:13 AM   #3
ziadh
Member
 
Registered: Aug 2010
Location: France, Paris
Distribution: Solaris10
Posts: 88

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by GrapefruiTgirl View Post
What do you mean by "organized and understandable"?
i mean to get logs as tables or diagram or whatever shape... rather than reading thousand of pages and find the exact day and exact hour and exact IP address..
i have a lot of infos in my logs, so when searching for an event its really hard!
i want to be able to organize them (per day, per traffic, per network address, per time of connection...) in a way so it becomes easy to search...

about mailing the logs, do you know any tool that can mail me specific logs? i mean erroneous messages only ( i dont want logs to be automaticaly sent to my inbox, just some critical ones)

thx a lot!!
 
Old 09-15-2010, 07:38 AM   #4
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
I don't know by name of a specific tool that will organize a bunch of logs into some sort of table or diagram for easy reading, though there might be one..

But, it should not be a gigantic task to write a script and use some relatively simple grep, awk, sed, sort, perl, shell, or whatever you're comfortable with, or even use the commands on the commandline, to search for items in the logs or to maybe parse, format, and output the logs into a neater layout in a new file. All that could be done with a cronjob too (see below). You could then either easily search this new file for information, OR open it in a text editor which provides easy-to-use search functions.

As for mailing yourself some specific logs, or notifications of certain critical events, I again suggest you read about logrotate; but also, consider maybe using CRON to execute a cronjob regularly to look at the logs for specific events, dump them to a new file, and mail you that file. If there are no important events to be mailed, the cronjob doesn't mail you.

There's a tool for monitoring firewall logs for security issues, called `fwlogwatch` which may interest you also.

If you really want some piece of software that will try to put a bunch of logs together into a table or something, I'll let someone else try to answer that with a suggestion; but meanwhile, if parsing logs for certain information is a particular problem for you, then perhaps giving us a real-life example of the sort of data you're looking at, and what you'd like to search for within that data, would allow someone to help you devise a relatively easy programmatical way of searching or parsing the data. Also, for example, what might this occasional "erroneous messages" be? How often might it occurr? Again, a regular cronjob would be ideal for checking for this erroneous message and sending an email.

Cheers!
 
Old 09-15-2010, 07:57 AM   #5
ziadh
Member
 
Registered: Aug 2010
Location: France, Paris
Distribution: Solaris10
Posts: 88

Original Poster
Rep: Reputation: 0
First thank you for the reply, and i would like you to know that I'm a real newbie in Linux world, so i did not much understand about CRON, and i do not do any programming so creating a script and "simple grep, awk, sed, sort, Perl, shell" is not that simple for me, actually what I'm looking for is something easy to configure and who can do much of the job for me....

here's a sample of the logs i got:
Packet-Type = Access-Request
Fri Sep 10 15:05:09 2010
NAS-IP-Address = 172.26.x.x
NAS-Port = 1
NAS-Port-Type = Virtual
User-Name = "switch"
Calling-Station-Id = "172.26.x.x"
User-Password = "cisco"
Client-IP-Address = 172.26.10.3
Huntgroup-Name = "cisco_switch"

Fri Sep 10 15:05:23 2010
NAS-IP-Address = 172.26.x.x
NAS-Port = 1
NAS-Port-Type = Virtual
User-Name = "switch"
Calling-Station-Id = "172.26.x.x"
Acct-Status-Type = Start
Acct-Authentic = RADIUS
Service-Type = NAS-Prompt-User
Acct-Session-Id = "00000002"
Acct-Delay-Time = 0
Client-IP-Address = 172.26.x.x
Acct-Unique-Session-Id = "eb327551db69260c"
Timestamp = 1284123923

Fri Sep 3 09:52:29 2010
NAS-IP-Address = 172.26.x.x
NAS-Port = 1
NAS-Port-Type = Virtual
User-Name = "switch"
Calling-Station-Id = "172.26.x.x"
Acct-Status-Type = Stop
Acct-Authentic = RADIUS
Service-Type = NAS-Prompt-User
Acct-Session-Id = "00000001"
Acct-Terminate-Cause = User-Request
Acct-Session-Time = 149
Acct-Delay-Time = 0
Client-IP-Address = 172.26.x.x
Acct-Unique-Session-Id = "d8c68f3bc9386a71"
Timestamp = 1283500349

and these are just " exemple of thousands!! each with different IP, username, etc...
 
Old 09-15-2010, 08:24 AM   #6
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
OK.. So there are 1000's of entries like this - looks like a lot to us humans, but not really a lot of work for those tools I mentioned; please give some examples of what you might have to locate at any given time, in a logfile like that.

I understand you're new to Linux - and that's cool - we'll do our best to help, and try to keep it on the simple side, but at some point, especially if you're doing server-administration or maintenance sort of things (which this appears to be related to) and dealing with large logfiles like this, you will really want (if not need) to become somewhat familiar with the abilities of your machine's shell environment (your shell is the command-line; Bash or some relative of Bash, most likely), and have at least a small working knowledge of how to use grep, awk, etc., as they are indispensable for one-off searches and parsing files like this. All these tools, including the shell, have a 'man page' or 'User Manual' page.

CRON is a task scheduler, which runs as a daemon (a background process) and comes with nearly every Linux OS. It uses small files called 'crontabs' which contain lists of tasks, and it regularly checks these task lists to see if anything is scheduled, and if so, it performs the tasks. Cron usually is started up at system boot, during when your Linux OS is booting up and getting ready to use.

As mentioned above, man pages: Very important to get familiar with, and accustomed to summoning. You can access the man page (help documentation) for virtually any linux command or tool, just by typing, for example:
Code:
shell$ man grep
The above produces the Manual Page for the `grep` command.

So, anyhow.. If you still want to await some software for organizing this data, that's OK - but if you'd maybe like to see how some folks might search for various data in that logfile, please tell us what you'd want to search for, and how you'd like it outputted.

Cheers!

Last edited by GrapefruiTgirl; 09-15-2010 at 08:25 AM. Reason: formatting adjustment
 
Old 09-15-2010, 08:48 AM   #7
ziadh
Member
 
Registered: Aug 2010
Location: France, Paris
Distribution: Solaris10
Posts: 88

Original Poster
Rep: Reputation: 0
Again, thank you for the time you're giving me
in these logs, lets say for example i would like to know, in a specific month ( or any period of time) how many times the user "User-Name = "switch" logged into this device "Client-IP-Address = 172.26.1x.x", and so i can build a table to compare Users session time (for example), just at the end of the month i can have some clear statistics out of my logs..
the cool thing is that there's no much info in them, they r pretty easy to read and to understand one by one, but the problem is the presence of many users and many devices.
btw, im using Linux SUSE, and these logs are from a RADIUS server (AAA services server)
 
Old 09-15-2010, 11:45 PM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,328

Rep: Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364
I know there's an interface to MySQL for RADIUS for Auth ie you can store Authentication (user/passwd) in MySQL for RADIUS to check with. There may also be a way to tell RADIUS to store directly into MySQL, but I've never looked for it.
In any case, as you want to be able to do random viewing/grouping of info by various keys, I'd definitely suggest storing the log content in a DB eg MySQL.
Then you can use various tools (MRTG, CACTI etc) to graph/picture it.
If you have to write your own prog to insert the data from the logs into the DB, I'd recommend Perl. ( You may find your RADIUS server is writtten in Perl).

http://perldoc.perl.org/
http://www.perlmonks.org/?node=Tutorials
 
Old 09-17-2010, 03:08 AM   #9
ziadh
Member
 
Registered: Aug 2010
Location: France, Paris
Distribution: Solaris10
Posts: 88

Original Poster
Rep: Reputation: 0
can any1 show me a basic config of syslog.conf, where i need to import logs from a location /home/fernando/downloads/one, and put them into syslog folder etc/syslog/one

and also import from another locatio, /home/fernando/downloads/two, and put them into etc/syslog/two

thank you all
 
Old 09-17-2010, 03:16 AM   #10
prayag_pjs
Senior Member
 
Registered: Feb 2008
Location: Pune - India
Distribution: Fedora,RedHat,CentOS,Gentoo
Posts: 1,143
Blog Entries: 4

Rep: Reputation: 147Reputation: 147
Hi,you can refer the examples here
 
Old 09-23-2010, 11:23 AM   #11
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
AWK script to find data in radius server logs.

Hi ziadh,

Sorry for having 'disappeared' from this thread for a few days - I had been without internet service while I changed providers. While I was offline, I came up with an AWK program which is attached below, and which I'll explain soon.

Meanwhile.. First, the above poster gave a link to what appears to be a good little tutorial on syslog.conf configurations. Note though that syslog itself does not do any 'importing' or moving of logs from one location to another; you'd have to come up with a solution to do that. Maybe `logrotate` could be used, but that too is not precisely the right tool for moving/importing logs from one location to another, except for its ability to create, delete, compress, and mail the logs. You might want to look into a cron job or some other method of moving/importing logs.

And.. Chrism's suggestion about a database, is probably the very best suggestion if you wish to be able to search a bunch of log data arbitrarily based on some particular keys or values. Similar to Chris, I have not either ever looked into interfacing RADIUS with MySql, nor dealt with RADIUS logs, so you'll need to read some more to see if this is an option, and how to go about it.

So now, about this AWK program... While I had no internet, I spent a bit of time when I might otherwise be using the internet, occupying myself by practicing some AWK programming, and since this thread was the last thing I read before my internet went away, I decided to write an AWK program to scan your log file (which contains the data format you showed in post #5) and produce the data you mentioned in post #7 - a count of how many times a certain user logged in from a certain IP address, within a given month. This is exactly, and only, what my program does. Its use is specific to the exact data format and search criteria you gave as examples, but you could also learn by example from the program, and modify it or write your own, to do other similar tasks or to be able to search for different combinations of key/value pairs from log files.
Please be aware that the program is somewhat larger than it really needs to be to do the basic job; but, a program is rarely ever "totally completely done", and so when I had time, I kept looking at it, adding stuff, fancying it up, for the heck of it, and to play with AWK some more - so it's kinda bloated (and has LOTS of comments) - but it's very fast; it can scan multiple files containing tens of thousands of log records like you showed above, in less than a few seconds, and produce a tally of results.

As with all my scripts, this comes with no warranty! It was a learning experience for me, I hope it helps you or someone else learning how to use a bit of AWK to parse some files. I would have put this in the /Programming forum, but since it specifically is intended to do what you showed in post #7, here it is:
Code:
#!/usr/bin/awk -f

# radius.awk by Sasha, Sept. 2010 - Use, modify, or distribute as you see fit.
# Script comes with no warranty, and almost certainly could be improved upon.
# Written & tested with awk version:
# GNU Awk 3.1.8 Copyright (C) 1989, 1991-2010 Free Software Foundation.

# Reason for writing this script was more as a learning exercise for me
# than anything else, to experiment with some awk syntax and functionality.

# Purpose of script is to parse radius server log file(s) and count logins
# of a given User-Name from a given Client-IP-Address during a given Month.
# Script can fairly easily be adapted to find/count pretty much any info in
# any sort of log file(s).
# Script needs minimum of 4 arguments on the command line:
#  ./radius.awk filename [filename ...] --user=USER --ip=IP --month=MONTH
# So as shown, give names of one or more files to search, and one each of
# USER, IP, and MONTH.
# Arguments may be given in any order. Month can be abbreviated, and is not
# case-sensitive, so: Sep, sept, or September are equally valid.
# Execute script with -h option to see usage instructions. Below is a sample
# command line, which would check a file named radius.log to see if John
# logged in from 172.26.1.2 during September:
#  ./radius.awk radius.log --user=John --ip=172.26.1.2 --month=September

BEGIN {
# Probably clunky argument sanity checking;
    param_error = 0; count_args = ARGC-1
    for (x = 1; x <= count_args; x++) {
        if (ARGV[x] == "-h") { param_error = 1; exit }
    }
    if (count_args < 4) {
        printf ("%s","\nError: Not enough arguments.\n") > "/dev/stderr"
        param_error = 2; exit
    }
# check arguments for a username, valid IPv4 address, and valid month;
    for (x = count_args; x >= 1; x--) {
        if (ARGV[x] ~ /^--month=|-m=|--m=|-M=|--M=/) {
            split(ARGV[x],month_array,"="); search_mon = month_array[2]; search_month = tolower(search_mon)
            if (search_month !~ /^jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec/) {
                printf ("%s","\nError: Unknown month: " search_mon "\n") > "/dev/stderr"
                param_error = 2; exit
            } else {
                delete ARGV[x]
            }
        }
        if (ARGV[x] ~ /^--ip=|-i=|--i=|-I=|--I=/) {
            split(ARGV[x],ip_array,"="); search_ipaddr = ip_array[2]
            if (search_ipaddr !~ /^[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+$/) {
                printf ("%s","\nError: Invalid IPv4 address: " search_ipaddr "\n") > "/dev/stderr"
                param_error = 2; exit
            } else {
                delete ARGV[x]
            }
        }
        if (ARGV[x] ~ /^--user=|-u=|--u=|-U=|--U=/) {
            split(ARGV[x],user_array,"="); search_username = user_array[2]
            delete ARGV[x]
        }
    }
    if (search_month == "" || search_ipaddr == "" || search_username == "") {
        printf ("%s","\nError: Not enough arguments.\n") > "/dev/stderr"
        param_error = 2; exit
    }
# check for duplicates of given file names, and check that the files exist;
    for (x = 1; x <= count_args; x++) {
        if (ARGV[x] != "" && system("test -f " ARGV[x] " && exit 0 || exit 1") == 1) {
            printf ("%s","\nError: " ARGV[x] ": File not found.\n") > "/dev/stderr"
            param_error = 2; exit
        }
        all_filenames = all_filenames ARGV[x] "\n"
    }
    if ((system("test $(printf \"" all_filenames "\" | sort | uniq -d) && exit 1 || exit 0") == 1)) {
        printf ("%s","\nError: Duplicate input filenames.\n") > "/dev/stderr"
        param_error = 2; exit
    }
# initialize some other variables; see NOTEs following:
    RS               = "\n\n"
    FS               = "\n"
    count_logins     = 0
    show_records     = "NO"
    show_login_dates = "NO"

#-NOTE: show_records = "NO" if you do NOT want each matching record printed;
# if "YES", matching records are printed as they are found during program
# execution, before the final tally is printed.
#-NOTE: show_login_dates = "NO" if you do NOT want to see a list of the dates
# when matching logins happened; if "YES", the list of dates will be printed
# after all matching records have been printed (if you opted for that above)
# but before the final tally is printed.
# ...Could maybe add some -v or -vv options to turn on the above two items?
}

{
# MAIN PROGRAM
# get month of log entry record. If we have correct month, go on to examine
# username and IP of record. We assume the date is always 1st field in record
# and month is always the 2nd element of array created by splitting that field;
    split ($1,date," "); found_month = tolower(date[2])
    if (search_month ~ found_month) {
        found_ipaddr = ""; found_username = ""
        for (x = 2; x <= NF; x++) {
            if ($x ~ "^User-Name") {
                split ($x,user," "); found_username = user[3]
            } else {
                if ($x ~ "^Client-IP-Address") {
                    split ($x,foundipaddr," "); found_ipaddr = foundipaddr[3]
                }
            }
        }
# if username & IP match search criteria, increment tally of matches;
        if (found_ipaddr == search_ipaddr &&
        substr(found_username,2,(length(found_username)-2)) == search_username) {
            if (show_records == "YES") {
                printf ("\nMatching login record from '" FILENAME "':\n" $0 "\n")
            }
            if (show_login_dates == "YES") { login_date[count_logins] = FILENAME": "$1 }
            count_logins++
        }
    }
# End of MAIN PROGRAM
}

END {
# Either all files/records have been processed, or we arrived here because of
# an exit statement.. If there were no errors with the args passed to the
# program, and no -h option given, then display names of files we searched
# and output the tally of successful matches, if any;
    if (param_error == 0) {
        if (count_logins == 1) { logins = "login" } else { logins = "logins" }
        if (NR == 1) { total_records = "record" } else { total_records = "records" }
        if (count_args == 4) { total_files = "file";  input_filenames = "   Input file: "
            } else {           total_files = "files"; input_filenames = "  Input files: "
        }
        if (show_login_dates == "YES" && count_logins != 0) {
            printf ("%s","\nDates when matching logins happened:\n")
            for (x = 0 ; x <= count_logins; x++) { printf ("%s",login_date[x]"\n") }
        } else { printf "\n"
        }
        printf ("%s","Results total: "count_logins" "logins" by '"search_username"' from "search_ipaddr" in month of "search_mon".\n")
        printf ("%s",input_filenames)
        for (x = 1; x <= count_args; x++) {
            if (ARGV[x] != "") { printf ("%s",ARGV[x]) }; if (ARGV[x+1] != "") { printf ("%s",", ") }
        }
        printf ("%s","\n")
        printf ("%s","    Processed: "NR" "total_records" in "count_args-3" "total_files".\n")
    }
    if (param_error == 1) {
# user gave -h so show instructions;
        printf ("%s","\nThis script searches the given FILE(s) to see if the given USER has logged in\n")
        printf ("%s","from the given IP address within the given MONTH, and counts how many times.\n\n")
        printf ("%s","Usage: execute this script in your shell, with 4 or more arguments like so:\n\n")
        printf ("%s","shell#: ./this_script filename [filename ...] --user=USER --ip=IP --month=MONTH\n\n")
        printf ("%s","MONTH isn't case-sensitive, and can be abbreviated: Sep == sept == September.\n")
        printf ("%s","USER and filename are case-sensitive.\n")
        printf ("%s","Opts may be long or short form: --user= or -u= or -U= are all OK.\n\n")
        printf ("%s","To see this help info again, use -h argument.\n")
    }
}
# End of file
You notice there is a section highlighted in bold text - the two variables I bolded can be "YES" or "NO", and as the comments following them explain, they are for making the program output more data, such as the dates when logins occurred and/or the entire record from each login.
Here's a sample run, demonstrating a scan of two sample log files I created, looking for logins by "switch" from "172.26.1.2" in "September":
Code:
sasha@reactor: ./radius.awk radius6.log radius5.log -U=switch -I=172.26.1.2 --m=Sept

Results total: 7056 logins by 'switch' from 172.26.1.2 in month of Sept.
  Input files: radius6.log, radius5.log
    Processed: 12096 records in 2 files.
sasha@reactor:
So, that's that - if you want to try it, copy the script and save it as a text file, and give it execute permissions; then run it from the command-line as shown in my example above. You can run it with a -h option to see its 'help' output.

Again, I hope it helps someone in some way, even if it may be too narrow-scoped to help you in a general way with day to day requirements. If you have any questions I'll be happy to do my best to answer them.

Good luck!
 
Old 09-23-2010, 09:07 PM   #12
willcastle
Member
 
Registered: Sep 2010
Location: Philippines
Distribution: Centos
Posts: 63

Rep: Reputation: 0
Hi,

How can CRON mail the logs to my Email?

Thanks in advance.
 
Old 09-24-2010, 12:50 AM   #13
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,328

Rep: Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364Reputation: 2364
Strictly speaking that should probably have been a new thread, but anyway, here's one way
Code:
cat file | mailx -s "subject here" user@mail.addr.com
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
syslog-ng -> syslog-ng logging, how to troubleshoot sir-lancealot Linux - Server 1 01-24-2009 06:07 AM
LXer: OpenLDAP Quick Tips: Using syslog or syslog-ng with slapd for OpenLDAP logging LXer Syndicated Linux News 0 11-14-2008 08:41 PM
I need help getting syslog to log remotely, this is just the regular syslog. abefroman Linux - Software 2 06-05-2008 11:36 AM
syslog client to log to syslog-ng and itself noir911 Linux - Server 1 02-08-2008 09:51 AM
LXer: Centralized Syslog Server Using syslog-NG LXer Syndicated Linux News 0 04-28-2006 06:21 PM


All times are GMT -5. The time now is 03:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration