LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-20-2019, 07:47 PM   #61
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783

I give up

Code:
grep "404" $loghttp | cut -d ':' -f2- | awk '{print$1}' > iptemp.tmp 
grep "404" $loghttps | cut -d ':' -f2- | awk '{print$1}' >> iptemp.tmp 
if [[ -z "$shd" ]]
then
cat iptemp.tmp | sort -u > iptemp2.tmp
else
cat iptemp.tmp | sort -u | sed -e "/$shd/d" > iptemp2.tmp

I have tried, but I have failed.

horse to water and all that
 
3 members found this post helpful.
Old 10-20-2019, 08:36 PM   #62
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
This is a different flow
it checks every log line and reports the exploit + the IP


no cats echos seds awks

just bash ( and printf )

Code:
#!/bin/bash

unset Patterns
while read -a RE
do
  declare -A Patterns[${RE[0]}]+="${RE[1]}|"
done < definition.conf
# definition.conf "exploit_type regular_expression"
# NOTE they are real regular_expressions not strings !
# Patterns is an associative array, indexed by exploit_type
# regular_expression are appended, with "or" |
# the trailing | is stripped later ${Patterns[$i]%|} 
# One day I'll think of a better way :D

# LogFormat "%h\t%l\t%u\t%t\t\"%r\"\t%>s\t%b" common
# should also fix the awful default date format ;)
while IFS=$'\t' read -r -a Log_line
# [0]="IP" [1]="RFC 1413 aka junk" [2]="userid" [3]="date/time" [4]="request line"
# [5]="server status code" [6]="object size" 
do
  # ${!Patterns[@]} produces a list of the indexes (indices?)
  for exploit in ${!Patterns[@]}
  do
    [[ ${Log_line[4]} =~ ${Patterns[${exploit}]%|} ]] \
      && printf "Exploit %s detected via IP %s\n" \
        ${exploit} \  
        ${Log_line[0]} \
      && break # we break as no need to carry on checking
  done
done < serverlog.tmp
I hope that inspires you

edit
and this will likely blow your mind
Code:
done < <( tail -F serverlog.tmp )
yes, it will now run forever........

edit2

cleaner appending of the REs
Code:
unset Patterns
while read -a RE
do
  [[ ${#Patterns[${RE[0]}]} == 0 ]] \
    && declare -A Patterns[${RE[0]}]+="${RE[1]}" \
    || declare -A Patterns[${RE[0]}]+="|${RE[1]}"
done < definition.conf
# definition.conf "exploit_type regular_expression"
# Patterns is an associative array, indexed by exploit_type
# regular_expression are appended, prefixed | ( RE or )
<snip>
   # no need to remove trailing | , we don't have that anymore
   #[[ ${Log_line[4]} =~ ${Patterns[${exploit}]%|} ]]
    [[ ${Log_line[4]} =~ ${Patterns[${exploit}]} ]] \

Last edited by Firerat; 10-20-2019 at 10:21 PM.
 
1 members found this post helpful.
Old 10-21-2019, 02:21 AM   #63
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 2,768

Rep: Reputation: 1192Reputation: 1192Reputation: 1192Reputation: 1192Reputation: 1192Reputation: 1192Reputation: 1192Reputation: 1192Reputation: 1192
Quote:
Originally Posted by Firerat View Post
I give up

Code:
grep "404" $loghttp | cut -d ':' -f2- | awk '{print$1}' > iptemp.tmp 
grep "404" $loghttps | cut -d ':' -f2- | awk '{print$1}' >> iptemp.tmp 
if [[ -z "$shd" ]]
then
cat iptemp.tmp | sort -u > iptemp2.tmp
else
cat iptemp.tmp | sort -u | sed -e "/$shd/d" > iptemp2.tmp
I have tried, but I have failed.

horse to water and all that
I haven't bothered to look what this code does, but want to take the opportunity to demonstrate some capabilities of the shell that you don't find in books:
Code:
grep -h "404" $loghttp $loghttps |
cut -d ':' -f2- | awk '{print$1}' |
if [[ -z "$shd" ]]
then
  sort -u
else
  sort -u | sed -e "/$shd/d"
fi > iptemp2.tmp
if,while,until,for,case start code blocks where you can redirect stdin,stdout,stderr. The only special thing is that a pipe | to a code block forces it into a sub shell.
You can also have an explicit code block in braces { } where the syntax requires the closing } to occur after a newline or a semicolon.
 
1 members found this post helpful.
Old 10-21-2019, 02:46 AM   #64
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
@MadeInGermany

it is not my code, I was dismayed as I've been trying to teach them to not use that sloppy and redundant code


just look at it


Code:
grep "404" $loghttp | cut -d ':' -f2- | awk '{print$1}' > iptemp.tmp
they are filtering for 404 , cutting and awking the cut, presumably for the IP

Code:
awk '/404/{sub(/:.*/,"",$2);print $2}'
the field number might be incorrect, I really haven't bothered looking at what that sloppy code is doing

but it is completely redundant anyway


I can quickly hack in a 404 filter
Code:
while IFS=$'\t' read -r -a Log_line
# [0]="IP" [1]="RFC 1413 aka junk" [2]="userid" [3]="date/time" [4]="request line"
# [5]="server status code" [6]="object size" 
do
  [[ ${Log_line[5]} != 404 ]] && continue
  for exploit in ${!Patterns[@]}
  do
    [[ ${Log_line[4]} =~ ${Patterns[${exploit}]%|} ]] \
      && printf "Exploit %s detected via IP %s\n" \
        ${exploit} \  
        ${Log_line[0]} \
      && break # we break as no need to carry on checking
  done
or have a case

Code:
  case ${Log_line[4]} in
    404) dummy=yes;;
      *) continue;;
  esac

Last edited by Firerat; 10-21-2019 at 02:50 AM. Reason: ${Log_line[5]} not ${Log_line[4]} from my copypasta
 
1 members found this post helpful.
Old 10-21-2019, 03:39 AM   #65
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
This is probably easier to understand
I have removed one of the arrays in favour of vars

However, an array would make more sense if you have different logs with differing numbers of fields


Code:
#!/bin/bash

unset Patterns
while read -a RE
do
  [[ ${#Patterns[${RE[0]}]} == 0 ]] \
    && declare -A Patterns[${RE[0]}]+="${RE[1]}" \
    || declare -A Patterns[${RE[0]}]+="|${RE[1]}"
done < definition.conf
# definition.conf "exploit_type regular_expression"
# Patterns is an associative array, indexed by exploit_type
# regular_expression are appended, prefixed | ( RE or )

ExploitCheck () {
  for exploit in ${!Patterns[@]}
  do
    [[ ${Request} =~ ${Patterns[$exploit]} ]] \
      && printf "Exploit %s detected via IP %s\n" \
        ${exploit} \
        ${IP} \
      && break
  done
}

# LogFormat "%h\t%l\t%u\t%t\t\"%r\"\t%>s\t%b" common
# should also fix the awful default date format ;)

while IFS=$'\t' read IP RFC1413 UserID Time Request StatusCode ObjectSize
do
  case ${StatusCode} in
    404) ExploitCheck
        ;;
  esac
done < serverlog.tmp
#done < <( tail -F serverlog.tmp )
 
1 members found this post helpful.
Old 10-21-2019, 05:42 AM   #66
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
interesting .

Quote:
I haven't bothered to look what this code does, but want to take the opportunity to demonstrate some capabilities of the shell that you don't find in books:

grep -h "404" $loghttp $loghttps |
cut -d ':' -f2- | awk '{print$1}' |
if [[ -z "$shd" ]]
then
sort -u
else
sort -u | sed -e "/$shd/d"
fi > iptemp2.tmp
It does not work properly , despite the fact that it should , instead grabbing an ip i am grabbing date ?!!!
Anyway , don't bother with this one , i like it the way it is originally .

Basically you both changed the code at your own perspective that right now some codes dont work , or does anything differently .

Well , i figured out one thing .

from original code

Code:
intdet () {
IP=$1
while read LOG
do
    for P in "${Patterns[@]}"
    do  
        [[ ${LOG} =~ "${P#* }" ]] && intr=${P% *} 
    done
done < ipres 
return $?
}
Patterns=()
while read foo
do
    Patterns+=("${foo}")
done < /def/definition.conf
while read IP
do
    intdet $varsel
done <<< ${varsel}
i only need

Code:
intdet () {
while read LOG
do
    for P in "${Patterns[@]}"
    do  
        [[ ${LOG} =~ "${P#* }" ]] && intr=${P% *} 
    done
done < ipres 
return $?
}
Patterns=()
while read foo
do
    Patterns+=("${foo}")
done < /def/definition.conf
intdet
Anyway , please dont bother anymore , the original thread question was solved , this is just an expansion of it that it never should happen .
Thank you all

By the way , i fixed the issue , i just added an extra line to definition.conf with jerbish stuff , and now the read command reads the line 25 . Some how it does not read last line , so i always have to add something in last line that it does not match with anything at all .

Last edited by pedropt; 10-21-2019 at 08:16 AM.
 
Old 10-21-2019, 07:09 AM   #67
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by pedropt View Post
By the way , i fixed the issue , i just added an extra line to definition.conf with jerbish stuff , and now the read command reads the line 25 . Some how it does not read last line , so i always have to add something in last line that it does not match with anything at all .
that is interesting

I'll have to have a think about that
 
Old 10-21-2019, 09:19 AM   #68
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by Firerat View Post
This is probably easier to understand
I have removed one of the arrays in favour of vars

However, an array would make more sense if you have different logs with differing numbers of fields
as a way of explaining ( justifying ) why I favour array over list of vars
It makes checking easier
Code:
#!/bin/bash
list_one () {
cat << EOF
zero one two three four
zero one two three four
EOF
}
list_two () {
cat << EOF
zero one two three 
zero one two three four
zero one two three four five
EOF
}

Vars () {
echo Vars $1
LIST=$1
while read zero one two three four
do
  printf "%s_%s_%s_%s_%s\n" zero=$zero one=$one two=$two three=$three four=$four
done < <( $LIST )
echo
}

Array () {
echo Array $1
LIST=$1
while read -a list
do
  printf "_%s_%s_%s_%s_%s" zero=${list[0]} one=${list=[1]} two=${list[2]} three=${list[3]} four=${list[4]}
  [[ ${#list[@]} != 5 ]] && printf "\t%s\n" "Warning fields incorrect" || printf "\n"
done < <( $LIST )
echo
}

Vars list_one
Vars list_two
Array list_one
Array list_two
you can of course check each $zero $one ... $four is not empty
but simply counting number of array elements is easier
and if you are routinely looking at two logs, one with 5 fields and one with 7 you can count elements and act accordingly ( you should also confirm the extra fields are what is expected and not corruption of the input data )
 
Old 10-21-2019, 10:11 AM   #69
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
I will probably do some tests on all code you posted here , but for now the first approach you did (as i explained in last post) is working like a charm , i have to admit , you did a very good approach on it .
The fact that when it is reading i will always have in advance on definitions file a last line that does not have any influence in the matches just to make it work , it does not have any problem , because if i need to add any new definition to the file then what i have to do is remove last line with sed and then place the new definition and the extra line .
Everything is fine , more you simplify the code or different approaches then it is worst for me to understand it in future , i don't write bash everyday , some times takes months before i write something in bash , and by this it will mean that i will forget all your more complex code sequence and how it works .
If i do it simple and use your first approach then when i open this script to pick up some code to another script then i will understand how it is done .

One thing you can be sure , despite that i am gratefully for your work on this thread , i will grab all the code you wrote to a file here , and maybe when i get some time i will try your different approaches in debug mode , this way i will see what the code is doing .

thanks for everything
 
Old 10-21-2019, 11:04 AM   #70
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783



yeah no problem

worth pointing out that my scripts are pure bash ( besides printf )

My first scripts were essentially lists of commands

then I was adding conditionals ( if then else )
unfortunately I fell into the pipetrain trap, as I was "learning" from bad scripting examples.

I don't actually remember what it was, but I was exposed to real scripts and I re-learned from those

I highly recommend
https://mywiki.wooledge.org/FullBashGuide
and
https://www.tldp.org/LDP/Bash-Beginners-Guide/html/

and of course
http://porkmail.org/era/unix/award.html
that highlights many of the poor code examples you find in too many scripts
 
Old 01-10-2020, 03:29 PM   #71
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
by the way , i had to do some adjustments to this script and find an alternative way in case this routine does not find a match , even if it exists .
What this new routine does is count the number of characters on the line to search , and then uses only a fragment to look into the patterns .
In case this new routine does not work then run the code presented by you guys here .
I created a new script just to do this job , instead adding the code to my main script .
Here it is the code now (i am still doing some adjustments according to my needs or to each case presented)

I had to do this new adjustment because it was not capturing some complicated lines with special characters on then

Code:
#!/bin/bash
#file to check patterns
comp=$1
#Definition File
defile=$2

idptrn () {
while read LOG
do
    for P in "${Patterns[@]}"
    do
        [[ ${LOG} =~ ${P#* } ]] && echo ${P% *}
        #echo "$LOG -> $P"
    done
done < "$comp"
#echo "executed $0 $comp $defile"
return $?
}
splt() {
if [[ "$cnt" -gt "15" && "$cnt" -lt "30" ]]
then
gtpt=$(echo "$rdnmb" | cut -c1-8)
a1=$(grep "$gtpt" "$defile" | awk '{print$1}')
if [[ ! -z "$a1" ]]
then
echo "$a1"
exit 0
fi
elif [[ "$cnt" -gt "31" ]]
then
gtpt=$(echo "$rdnmb" | cut -c1-15 )
a1=$(grep "$gtpt" "$defile" | awk '{print$1}')
if [[ ! -z "$a1" ]]
then
echo "$a1"
exit 0
fi
fi
splt
done
}
help () {
clear
echo "Pattern ID 1.0"
echo "---------------------------"
echo "example : $0 file_to_search  file_with_patterns"
echo "example : $0 /mylog /definition.conf"
echo "----------------------------"
exit 0
}
if [[ -z "$comp" ]]
then
help
elif [[ ! -f "$comp" ]]
then
echo "File to compare not found in : $comp"
exit 0
elif [[ -z "$defile" ]]
then
help
elif [[ ! -f "$defile" ]]
then
echo "Definition file not found in : $defile"
exit 0
fi
nwrtn
Patterns=()
while read foo
do
    Patterns+=("${foo}")
done < "$defile"
idptrn
 
Old 01-14-2020, 08:21 AM   #72
individual
Member
 
Registered: Jul 2018
Posts: 315
Blog Entries: 1

Rep: Reputation: 233Reputation: 233Reputation: 233
Wow, I think this problem was made more difficult that it needed to be. Assuming the requirements are the same as stated in the OP, then the following should do what you need.
Code:
#!/usr/bin/env bash

set -eo pipefail
IFS=$'\n\t'

definitionFile='definitions.conf'
serverFile='server.log'

# find all ip addresses in the server log into an array.
mapfile -t ips < <(grep -Po '\d{1,3}(?:\.\d{1,3}){3}' "$serverFile" | sort -u)

# generate a simple menu using nl to number the lines.
nl -n ln <<< "${ips[*]}"

# ask the user to input a number from 1 to $NUMBER_OF_IPS
read -p "Enter a number from 1 to ${#ips[@]}: " -r choice

# do some input validation.
if [[ $choice -lt 1 ]] || [[ $choice -gt "${#ips[@]}" ]]; then
    echo "Invalid choice"
    exit
fi

# select the ip address that corresponds with choice.
ip="${ips[$((choice - 1))]}"

# harness some grep magic to locate lines with the ip address and pipe that to a second grep
# which will try to match the signature field in the definitions file, e.g. /thinkphp/html/public/index.php.
# the -f flag tells grep to search in a file for patterns. the file in this case is a redirection from
# the cut command.
# the -o flag tells grep to only print the matching portion of the string, which is the signature.
matched="$(grep -F "$ip" $serverFile | grep -o -f <(cut -d' ' -f2- $definitionFile))"

# if we found a match for the chosen ip address AND an exploit signature, find the key for the signature
# by locating it in the definitions file.
if [[ -n "$matched" ]]; then
    echo "Intrusion detected from $ip was : $(grep -Fm1 "$matched" definitions.conf | cut -d' ' -f1)"
else
    echo 'No results were found'
fi
P.S. If you're going to be writing Bash scripts often, please devote some time to studying best practices. It will really help you.

P.P.S. Don't forget to use shellcheck to help you catch mistakes in your scripts.

Last edited by individual; 01-14-2020 at 08:25 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
directory exists but no such directory exists yumito Linux - Newbie 3 06-09-2013 02:02 AM
[SOLVED] Search for a pattern and if it exists change some other text on the line KRevotsk Linux - Newbie 14 04-18-2013 02:23 PM
trying to change part of text with sequence for filename Adol Linux - Newbie 6 01-13-2013 11:02 AM
To exists or not to exists, this is the Q. Inbal Linux - Newbie 3 07-18-2006 06:04 AM
SIOCADDRT: File exists SIOCCADDRT: File Exists Failed to bring up eth0. opsraja Linux - Networking 0 01-10-2005 08:29 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:59 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration