LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 01-16-2013, 11:29 AM   #1
mlewis
Member
 
Registered: Mar 2006
Posts: 172

Rep: Reputation: 16
Extracting stats from a text email


I monitor my networks and jot down things I need based on reports such as I've posted below which I receive via email.

However, I very badly need to convert these outages into meaningful times/dates/patterns/stats which I can show the provider so that they can get things fixed.

I have scoured the net but simply don't quite understand enough about scripting to do something this complex.

Below is a snippet of what the emails looks like. What I need to do is to read the start times when failures occur and when the connection was reestablished.

So for example, I need to read only the lines which have a Track IP start and stop date/time. So based on the snippet below, the first one would read something like;

Dn 2013-01-15 18:06:53
Up 2013-01-15 18:06:55

Then, I need to be able to turn the above into a statistic showing the amount of down time and when so that I can force the provider to fix their services.

I sure would appreciate some help on this and again, I'm not a programmer which is why I'm asking for help.

Thanks.


[00001] 2013-01-15 18:10:12 [Root]system-alert-00008: IP spoofing! From 192.168.0.10 to 224.0.0.1, proto 2 (zone Untrust, int ethernet0/0). Occurred 1 times.
[00002] 2013-01-15 18:10:06 [Root]system-alert-00008: IP spoofing! From 192.168.0.1:1024 to 239.255.255.250:1900, proto UDP (zone Untrust, int ethernet0/0). Occurred 8 times.
[00003] 2013-01-15 18:09:06 [Root]system-alert-00008: IP spoofing! From 192.168.0.1:1024 to 239.255.255.250:1900, proto UDP (zone Untrust, int ethernet0/0). Occurred 8 times.
[00004] 2013-01-15 18:08:06 [Root]system-alert-00008: IP spoofing! From 192.168.0.1:1024 to 239.255.255.250:1900, proto UDP (zone Untrust, int ethernet0/0). Occurred 8 times.
[00005] 2013-01-15 18:08:02 [Root]system-alert-00008: IP spoofing! From 192.168.0.10 to 224.0.0.1, proto 2 (zone Untrust, int ethernet0/0). Occurred 1 times.
[00006] 2013-01-15 18:07:06 [Root]system-alert-00008: IP spoofing! From 192.168.0.1:1024 to 239.255.255.250:1900, proto UDP (zone Untrust, int ethernet0/0). Occurred 8 times.
[00007] 2013-01-15 18:06:55 [Root]system-critical-00062: Track IP IP address 24.116.2.50 succeeded.
[00008] 2013-01-15 18:06:53 [Root]system-critical-00062: Track IP failure reached threshold.
[00009] 2013-01-15 18:06:52 [Root]system-critical-00062: Track IP IP address 24.116.2.50 failed.
[00010] 2013-01-15 18:06:43 [Root]system-critical-00062: Track IP IP address 24.116.2.50 succeeded.
[00011] 2013-01-15 18:06:42 [Root]system-critical-00062: Track IP failure reached threshold.
[00012] 2013-01-15 18:06:41 [Root]system-critical-00062: Track IP IP address 24.116.2.50 failed.
[00013] 2013-01-15 18:06:40 [Root]system-critical-00062: Track IP IP address 4.2.2.2 succeeded.
[00014] 2013-01-15 18:06:40 [Root]system-critical-00062: Track IP failure reached threshold.
[00015] 2013-01-15 18:06:39 [Root]system-critical-00062: Track IP IP address 4.2.2.2 failed.
[00016] 2013-01-15 18:06:33 [Root]system-critical-00062: Track IP IP address 4.2.2.2 succeeded.
[00017] 2013-01-15 18:06:31 [Root]system-critical-00062: Track IP IP address 24.116.2.50 succeeded.
 
Old 01-17-2013, 06:24 AM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,261

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
1. I note that the rec nums are increasing, but the timestamps are decreasing ... ???

2. if you wanted to take a purely textual approach, I'd use Perl for this. Data/Text munging (fold/spindle/mutilate/re-format etc) is something Perl is very good at.
http://perldoc.perl.org/
http://www.tizag.com/perlT/index.php
http://www.perlmonks.org/?node=Tutorials

3. if you can't guarantee that you won't get rec pairs split over 2 emails (eg down in one email, up in the next), it might be better to save all the data to a DB (MYSQL) and use SQL to pull out pairs and stats.
http://dev.mysql.com/doc/refman/5.1/en/

4. A graphical approach might be to install Cacti and then send screen shots of the graphs or even give them a read-only access and just email a link.
http://www.cacti.net/downloads/docs/html/

5. Nagios is an alerting system that often goes with Cacti and you should be able to pull out that sort of data from the logs or the DB (if you choose to use that with Nagios)
http://www.nagios.org/documentation/
 
Old 01-17-2013, 08:03 AM   #3
mlewis
Member
 
Registered: Mar 2006
Posts: 172

Original Poster
Rep: Reputation: 16
The reports are coming from my juniper firewalls which do allow for remote syslog. I had thought of feeding syslog to a proper logging server but that didn't help me with past problems where I need to show a curve.

In the example I posted above, I was showing how the output might look after it's been taken from the text log.

The logs look like;
[00007] 2013-01-15 18:06:55 [Root]system-critical-00062: Track IP IP address 24.116.2.50 succeeded.
[00008] 2013-01-15 18:06:53 [Root]system-critical-00062: Track IP failure reached threshold.

This means the connection went down at 18:06:53 and recovered at 18:06:55.

I think I just need to find a decent Linux based syslog of which I've tried several over the years so know they aren't hard to find. I just need to find something however that will give me the stats of the past logs which I've kept.

Thanks.
 
Old 01-17-2013, 08:20 PM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,261

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
I'd still like to know why the timestamps are in reverse?

As for syslog tools; they are for logging.
If you want to produce text stats or graphical representation, you're going to have to process the log recs (emails).

Speaking of Graphs, Cacti is based on snmpd, which even firewalls do (usually) have, so you'd get real-time graphs.
 
Old 01-18-2013, 07:06 AM   #5
mlewis
Member
 
Registered: Mar 2006
Posts: 172

Original Poster
Rep: Reputation: 16
>I'd still like to know why the timestamps are in reverse?

I explained it above.

>As for syslog tools; they are for logging.

They can also be used for statistics, alerts, and much more than simply logging.

>If you want to produce text stats or graphical representation, you're going to have to
>process the log recs (emails).

What I asked is how can I do that with the logs I already have. Moving forward, I need to get some proper syslogs working again. That fell behind because the network is new so not everything is built just yet.

>Speaking of Graphs, Cacti is based on snmpd, which even firewalls do (usually) have, so you'd get real-time graphs

Yes, most do but this question is about the logs I already gave .

Thanks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Extracting certain lines from a text and outputting to new text files? paradeboy Linux - Newbie 4 03-14-2012 12:02 AM
[SOLVED] Extracting text from a file. TheNewGuy2936 Linux - Newbie 13 04-26-2011 10:16 AM
extracting particular lines from a text file skuz_ball Programming 18 10-28-2008 12:31 PM
extracting a chunk of text from a large text file lothario Linux - Software 3 02-28-2007 08:16 AM
python email module extracting headers artur Programming 3 02-06-2004 01:59 PM


All times are GMT -5. The time now is 12:50 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration