LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 05-17-2007, 06:28 PM   #1
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Rep: Reputation: 0
Bash script for server log (namely var/log/messages)


Hi all,

I need a bash script to traverse a filesystem from a server, and look for suspicious activity. This is mainly suspicious activity in the messages log, such as port probing from the same ip address, repeated failed login attempts etc.

Ive managed to traverse the filesystem but could do with a hand on looking for suspicious activity as I am not used to checking server logs.

Any help with the script or how to identify indicators of suspicious activity appreciated.

Thanks,

Bob.
 
Old 05-17-2007, 06:52 PM   #2
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
that would depend on what you actually want to find and also understanding of the log format.
for example, if your applications log to file using "failed","Failed","Error" etc, then you might want to use grep/awk/sed on these words..
 
Old 05-17-2007, 07:27 PM   #3
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Thanks for the quick response.

I had a search through for failed logins, but there doesnt appear to be any. I've been told there has been some port probing, but i wouldnt know the first place to look for that. An example of the log, after server startup, is:

May 2 13:20:53 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=59828 PROTO=TCP SPT=40414 DPT=80 WINDOW=1024 RES=0x00 ACK URGP=0
May 2 13:20:53 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3956 DF PROTO=TCP SPT=80 DPT=40414 WINDOW=0 RES=0x00 RST URGP=0
May 2 13:20:53 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3956 DF PROTO=TCP SPT=80 DPT=40414 WINDOW=0 RES=0x00 RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=38 ID=27772 PROTO=TCP SPT=40390 DPT=984 WINDOW=3072 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=38 ID=27772 PROTO=TCP SPT=40390 DPT=984 WINDOW=3072 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3957 DF PROTO=TCP SPT=984 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3957 DF PROTO=TCP SPT=984 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=19924 PROTO=TCP SPT=40390 DPT=489 WINDOW=1024 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=19924 PROTO=TCP SPT=40390 DPT=489 WINDOW=1024 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3958 DF PROTO=TCP SPT=489 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3958 DF PROTO=TCP SPT=489 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0

Again, any help either with the script or things to look out for is greatly appreciated.

Thanks,
Bob.
 
Old 05-17-2007, 07:50 PM   #4
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86; Gentoo PPC; Gentoo Sparc64; FreeBSD; OS X; Solaris
Posts: 3,731
Blog Entries: 4

Rep: Reputation: 66
Any reason you don't want to use one of the many scripts that do this already? Here's a couple:

Swatch: http://swatch.sourceforge.net/
Logwatch: http://www2.logwatch.org:81/

If you are really paranoid you may want to look into a full-blown IDS such as Snort or Nessus.
 
Old 05-17-2007, 08:06 PM   #5
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Hi,

The script is part of a project im doing, so i have to write the majority myself, although i can adapt scripts already out there. I imagine my script doesn't need to be as polished/complete as those either. Thanks for the suggestion though.

Bob.
 
Old 05-17-2007, 08:11 PM   #6
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86; Gentoo PPC; Gentoo Sparc64; FreeBSD; OS X; Solaris
Posts: 3,731
Blog Entries: 4

Rep: Reputation: 66
OK, well the two I posted are written in Perl, but you may be able to find things to look for by reading their source.
 
Old 05-17-2007, 10:00 PM   #7
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Many thanks bulliver. I'll take a look at those. In the meantime, if anyone has any other suggestions they'll be greatly appreciated.
 
Old 05-17-2007, 10:42 PM   #8
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
looking at your log , SRC and DST values (even SPT and DPT) may be useful. because you will have a list of internal LAN IPs and IP segments ( i assume you have), you can write script to check SRC values, if they are not in your internal IP list, then you can filter them out. You can also count the number of times the same time value comes out. eg count how many occurences of 13:20:53. ( or even 13:20 within the minute )
etc..the list goes on..
 
Old 05-17-2007, 11:03 PM   #9
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Thanks ghostdog, that sounds like something i should do, any tips on how to code that?

I was also thinking maybe checking if the same src ip address is probing different ports on the system. If anyone could help with the code for what ghostdog suggested or the probing issue id be grateful.

EDIT: Just so its clear, i need my script to traverse the server filesystem and once it finds the messages log file to read through it for suspicious activity (as mentioned above, things like port probing etc).
Any other files it could chek on for suspicious activity is a bonus, as are other suspicious things to look out for in the messages log.

Thanks,
Bob.

Last edited by tenaciousbob; 05-18-2007 at 01:50 AM.
 
Old 05-18-2007, 02:20 AM   #10
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
Quote:
Originally Posted by tenaciousbob
Thanks ghostdog, that sounds like something i should do, any tips on how to code that?
what's your scripting experience like?
this is quite a task, but i just give you some suggestion. you can use sed/grep/awk/perl/python to do file processing...i give a short example in awk
Code:
awk '
 {
   for (i=1;i<=NF;i++){
      if ($i ~ /SRC/) {
          sub(/^SRC=/,"",$i)
	  srcip[$i]++
      }
   }
 }

END{
 for (i in srcip ){
    print srcip[i] , i
 }
}' logfile
output:
Code:
# ./test.sh
5 127.0.0.1
6 10.0.2.15
the script above just count the number of source ips and displaying them out at the end. so by viewing the output if you get a lot of transactions from a single ip, you can start investigating..
Just an example, but it shows you how you can proceed on. similarly, this can be done in other language tools .
 
Old 05-20-2007, 05:48 PM   #11
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Thanks for that ghostdog, i tried it but i keep getting a syntax error near line 5 (once you have #!/bin/bash at the top). I have some coding experience, but im stuck on this.

Thanks,
Bob.
 
Old 05-20-2007, 06:48 PM   #12
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
what OS are you on, and your awk version? i use GNU awk
 
Old 05-23-2007, 06:08 PM   #13
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Thanks for that ghostdog - the server im working on has nawk and the script is working perfectly now.

Is there any way of adding to the script so that it would count the ip's as it does now, but also count the destination ports too. So for example if ip 10.0.2.15 probed destination port 984 10 times, it would display '10.0.2.15 probed port 984 10 times'

I tried fiddling about adding another array to count ips which worked, but it obviously didnt make the connection with the ip's too.

Thanks for the help so far,
Bob.
 
Old 05-23-2007, 08:31 PM   #14
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
can you show the code you have now?
 
Old 05-24-2007, 08:39 AM   #15
tenaciousbob
LQ Newbie
 
Registered: May 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Ok, ive got it to count ip's with destination ports. The scipt:

nawk '
{
for (j=1;j<=NF;j++)
{
for (i=1;i<=NF;i++)
{
if ($i ~ /^SRC/)
{
if ($j ~ /^DPT/)
{ srcip[$i, $j]++ }
}
}
}
}

END { print "Freq"" ""Source IP"
for (i in srcip ){
print srcip[i]" "srcip[j] , i" "j
}
}' testipcount | sort -r


The only problem is, i get the correct results, but every line has a 3 on the end:

Freq Source IP
4 SRC=123.4.5.6DPT=9 3
3 SRC=127.0.0.1DPT=1 3
2 SRC=10.0.2.15DPT=1 3
1 SRC=127.0.0.1DPT=3 3
1 SRC=127.0.0.1DPT=2 3
1 SRC=123.4.5.6DPT=8 3
1 SRC=10.0.2.15DPT=3 3
1 SRC=10.0.2.15DPT=2 3

I have no idea where the 3 is coming from. The file i ran it on just contains a mix of source ips and destination ports:

SRC=127.0.0.1 DPT=1
SRC=127.0.0.1 DPT=1
SRC=10.0.2.15 DPT=1
SRC=127.0.0.1 DPT=2
SRC=10.0.2.15 DPT=2
SRC=10.0.2.15 DPT=3
SRC=127.0.0.1 DPT=3
SRC=127.0.0.1 DPT=1
SRC=10.0.2.15 DPT=1
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=8

Last edited by tenaciousbob; 05-24-2007 at 10:13 AM.
 
  


Reply

Tags
bash, logs, script, server


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Hacked server :( and /var/log/messages el_pajaro! Linux - Security 4 04-17-2006 10:28 PM
Deleted /var/log/messages, can't log any files-iptables chingyenccy Linux - Newbie 7 02-27-2005 04:03 PM
How to log conversation between server in /var/log/messages? juris Linux - Software 1 11-23-2004 09:54 AM
Trigger script by message in /var/log/messages? Zoidy Linux - Software 3 01-17-2004 12:03 AM
iptables, changing log file from /var/log/messages acid2000 Linux - Networking 3 03-11-2003 08:38 PM


All times are GMT -5. The time now is 08:08 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration