LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   Bash script for server log (namely var/log/messages) (https://www.linuxquestions.org/questions/programming-9/bash-script-for-server-log-namely-var-log-messages-554743/)

tenaciousbob 05-17-2007 06:28 PM

Bash script for server log (namely var/log/messages)
 
Hi all,

I need a bash script to traverse a filesystem from a server, and look for suspicious activity. This is mainly suspicious activity in the messages log, such as port probing from the same ip address, repeated failed login attempts etc.

Ive managed to traverse the filesystem but could do with a hand on looking for suspicious activity as I am not used to checking server logs.

Any help with the script or how to identify indicators of suspicious activity appreciated.

Thanks,

Bob.

ghostdog74 05-17-2007 06:52 PM

that would depend on what you actually want to find and also understanding of the log format.
for example, if your applications log to file using "failed","Failed","Error" etc, then you might want to use grep/awk/sed on these words..

tenaciousbob 05-17-2007 07:27 PM

Thanks for the quick response.

I had a search through for failed logins, but there doesnt appear to be any. I've been told there has been some port probing, but i wouldnt know the first place to look for that. An example of the log, after server startup, is:

May 2 13:20:53 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=59828 PROTO=TCP SPT=40414 DPT=80 WINDOW=1024 RES=0x00 ACK URGP=0
May 2 13:20:53 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3956 DF PROTO=TCP SPT=80 DPT=40414 WINDOW=0 RES=0x00 RST URGP=0
May 2 13:20:53 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3956 DF PROTO=TCP SPT=80 DPT=40414 WINDOW=0 RES=0x00 RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=38 ID=27772 PROTO=TCP SPT=40390 DPT=984 WINDOW=3072 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=38 ID=27772 PROTO=TCP SPT=40390 DPT=984 WINDOW=3072 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3957 DF PROTO=TCP SPT=984 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3957 DF PROTO=TCP SPT=984 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=19924 PROTO=TCP SPT=40390 DPT=489 WINDOW=1024 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=52 ID=19924 PROTO=TCP SPT=40390 DPT=489 WINDOW=1024 RES=0x00 SYN URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3958 DF PROTO=TCP SPT=489 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0
May 2 13:21:09 localhost kernel: NET : IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=10.0.2.15 DST=10.0.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=3958 DF PROTO=TCP SPT=489 DPT=40390 WINDOW=0 RES=0x00 ACK RST URGP=0

Again, any help either with the script or things to look out for is greatly appreciated.

Thanks,
Bob.

bulliver 05-17-2007 07:50 PM

Any reason you don't want to use one of the many scripts that do this already? Here's a couple:

Swatch: http://swatch.sourceforge.net/
Logwatch: http://www2.logwatch.org:81/

If you are really paranoid you may want to look into a full-blown IDS such as Snort or Nessus.

tenaciousbob 05-17-2007 08:06 PM

Hi,

The script is part of a project im doing, so i have to write the majority myself, although i can adapt scripts already out there. I imagine my script doesn't need to be as polished/complete as those either. Thanks for the suggestion though.

Bob.

bulliver 05-17-2007 08:11 PM

OK, well the two I posted are written in Perl, but you may be able to find things to look for by reading their source.

tenaciousbob 05-17-2007 10:00 PM

Many thanks bulliver. I'll take a look at those. In the meantime, if anyone has any other suggestions they'll be greatly appreciated.

ghostdog74 05-17-2007 10:42 PM

looking at your log , SRC and DST values (even SPT and DPT) may be useful. because you will have a list of internal LAN IPs and IP segments ( i assume you have), you can write script to check SRC values, if they are not in your internal IP list, then you can filter them out. You can also count the number of times the same time value comes out. eg count how many occurences of 13:20:53. ( or even 13:20 within the minute )
etc..the list goes on..

tenaciousbob 05-17-2007 11:03 PM

Thanks ghostdog, that sounds like something i should do, any tips on how to code that?

I was also thinking maybe checking if the same src ip address is probing different ports on the system. If anyone could help with the code for what ghostdog suggested or the probing issue id be grateful.

EDIT: Just so its clear, i need my script to traverse the server filesystem and once it finds the messages log file to read through it for suspicious activity (as mentioned above, things like port probing etc).
Any other files it could chek on for suspicious activity is a bonus, as are other suspicious things to look out for in the messages log.

Thanks,
Bob.

ghostdog74 05-18-2007 02:20 AM

Quote:

Originally Posted by tenaciousbob
Thanks ghostdog, that sounds like something i should do, any tips on how to code that?

what's your scripting experience like?
this is quite a task, but i just give you some suggestion. you can use sed/grep/awk/perl/python to do file processing...i give a short example in awk
Code:

awk '
 {
  for (i=1;i<=NF;i++){
      if ($i ~ /SRC/) {
          sub(/^SRC=/,"",$i)
          srcip[$i]++
      }
  }
 }

END{
 for (i in srcip ){
    print srcip[i] , i
 }
}' logfile

output:
Code:

# ./test.sh
5 127.0.0.1
6 10.0.2.15

the script above just count the number of source ips and displaying them out at the end. so by viewing the output if you get a lot of transactions from a single ip, you can start investigating..
Just an example, but it shows you how you can proceed on. similarly, this can be done in other language tools .

tenaciousbob 05-20-2007 05:48 PM

Thanks for that ghostdog, i tried it but i keep getting a syntax error near line 5 (once you have #!/bin/bash at the top). I have some coding experience, but im stuck on this.

Thanks,
Bob.

ghostdog74 05-20-2007 06:48 PM

what OS are you on, and your awk version? i use GNU awk

tenaciousbob 05-23-2007 06:08 PM

Thanks for that ghostdog - the server im working on has nawk and the script is working perfectly now.

Is there any way of adding to the script so that it would count the ip's as it does now, but also count the destination ports too. So for example if ip 10.0.2.15 probed destination port 984 10 times, it would display '10.0.2.15 probed port 984 10 times'

I tried fiddling about adding another array to count ips which worked, but it obviously didnt make the connection with the ip's too.

Thanks for the help so far,
Bob.

ghostdog74 05-23-2007 08:31 PM

can you show the code you have now?

tenaciousbob 05-24-2007 08:39 AM

Ok, ive got it to count ip's with destination ports. The scipt:

nawk '
{
for (j=1;j<=NF;j++)
{
for (i=1;i<=NF;i++)
{
if ($i ~ /^SRC/)
{
if ($j ~ /^DPT/)
{ srcip[$i, $j]++ }
}
}
}
}

END { print "Freq"" ""Source IP"
for (i in srcip ){
print srcip[i]" "srcip[j] , i" "j
}
}' testipcount | sort -r


The only problem is, i get the correct results, but every line has a 3 on the end:

Freq Source IP
4 SRC=123.4.5.6DPT=9 3
3 SRC=127.0.0.1DPT=1 3
2 SRC=10.0.2.15DPT=1 3
1 SRC=127.0.0.1DPT=3 3
1 SRC=127.0.0.1DPT=2 3
1 SRC=123.4.5.6DPT=8 3
1 SRC=10.0.2.15DPT=3 3
1 SRC=10.0.2.15DPT=2 3

I have no idea where the 3 is coming from. The file i ran it on just contains a mix of source ips and destination ports:

SRC=127.0.0.1 DPT=1
SRC=127.0.0.1 DPT=1
SRC=10.0.2.15 DPT=1
SRC=127.0.0.1 DPT=2
SRC=10.0.2.15 DPT=2
SRC=10.0.2.15 DPT=3
SRC=127.0.0.1 DPT=3
SRC=127.0.0.1 DPT=1
SRC=10.0.2.15 DPT=1
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=9
SRC=123.4.5.6 DPT=8


All times are GMT -5. The time now is 05:26 AM.