LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-16-2019, 09:14 AM   #16
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783

many ways to skin this cat

assuming retrospective

Code:
#!/bin/bash

patterns=(
/wp-content/plugins/portable-phpmyadmin/wp-pma-mod/
/HNAP1/
/prov/aastra.cfg
/f4bb336d/
/module/action/param1/\${@die\(md5\(HelloThinkPHP\)\)}
/App/\?content=die\(md5\(HelloThinkPHP\)\)
/editBlackAndWhiteList
/0015650000000.cfg
)
# you should fill that array with real patterns

somemagicnumber=3
# how many tests before you action
#TODO weight patterns score using associative array

GetIPs () {
sort -u < <(grep -Eo "[0-9]{1,3}(\.[0-9]{1,3}){3}" server.log ) 
}

CheckPatterns () {
IP=$1
matchcount=0
while read LOG
do
  for P in "${Patterns[@]}"
  do
    [[ $LOG =~ $P ]] && ((matchcount++))
  done
done < <(grep "$IP" server.log )

[[ $matchcount -gt $somemagicnumber ]] \
  || return 0 \
  && Action $IP

return $?
}

Action () {
IP=$1
echo do something based on $IP
return $?
}

while read IP
do
  CheckPatterns $IP \
    || echo some error \
    && echo do something based on $IP
done < <( GetIPs )

all a bit sketchy, but you should get the idea

Edit, the returns in that code are scrappy
it should be easier to write if you have a clear plan of action based on what you do or don't find.

Last edited by Firerat; 10-16-2019 at 09:17 AM.
 
Old 10-16-2019, 09:40 AM   #17
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,371

Rep: Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748
Quote:
This way will work , but eventually i will do a lot of searches with not result that will increase time to the script .
The OP has expressed the need for speed. I will back an awk solution over a bash solution for that requirement.
 
Old 10-16-2019, 11:37 AM   #18
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
I think you are all messing it all up , the reason is because you are comparing all lines from 1 side to the other even if they all match , and i hust want the 1st one that matches , and after the match i want to retrieve from the match the variable 1 in awk from server.conf , witch will tell me what type of intrusion was detected .

The best way to do this is forgetting all the posts we did .
And concentrate on this one i am writing .

make a temporary directory somewhere and write all these files i will post here with their names in that directory .

file : definition.conf

Quote:
DLINK_HNAP_SEC HNAP1/
PHP_Exploitation elrekt.php
PHP_Exploitation TP/html/public/index.php
PHP_Exploitation /thinkphp/html/public/index.php
PHP_Exploitation phppma/index.php
Poly_voip_Info_Disclosure 000000000000.cfg
Poly_voip_Info_Disclosure 0015650000000.cfg
CVE-2019-11581 secure/ContactAdministrators!default.jspa
ThinkPHP_RCE module/action/param1/${@die(md5(HelloThinkPHP))}
UPNP_Info_Disclosure /UPnP/IGD.xml
CVE-2019-16759 ?routestring=ajax/render/widget_php
Fusion_PBX_Disclosure /prov/y000000000007.cfg
Hisilicon_DVDR mnt/custom/ProductDefinition
RDP_Probe mstshash=Administr
vTiger_CRM_CSS /vtigercrm/vtigerservice.php
CVE-2019-11510 /etc/passwd?/dana/html5acc/guacamole/
CVE-2019-17503/17504 /osm/report/
CVE-2019-17132 routestring=profile/upload-profilepicture
file : serverlog.tmp

Quote:
11.11.11.11 - - [00/Oct/2019:07:38:04 +0100] "GET /TP/public/index.php HTTP/1.1" 404 143 "-"
11.11.11.11 - - [00/Oct/2019:07:38:05 +0100] "GET /TP/index.php HTTP/1.1" 404 143 "-"
11.11.11.11 - - [00/Oct/2019:07:38:05 +0100] "GET /thinkphp/html/public/index.php HTTP/1.1"
11.11.11.11 - - [00/Oct/2019:07:38:05 +0100] "GET /html/public/index.php HTTP/1.1" 404 143
22.22.22.22 - - [00/Oct/2019:19:00:25 +0100] "GET ../../mnt/custom/ProductDefinition HTTP"
22.22.22.22 - - [00/Oct/2019:07:54:04 +0100] "GET /.well-known/keybase.txt HTTP/1.1" 200 2054
33.33.33.33 - - [00/Oct/2019:14:14:26 +0100] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173
filename : scriptest

Code:
#!/bin/bash
selip (){
for i in $(seq "$cntip")
do
rdip=$(sed -n ${i}p < iplist)
echo "$i - $rdip"
done
echo ""
echo "Choose an IP to check or write (exit) to quit "
echo -n "Choose from 1 to $cntip : "
read -r ip
if [[ "$ip" == "exit" ]]
then
exit 0
fi
if [[ -z "$ip" ]]
then
echo "Empty value detected"
sleep 3
clear
selip
fi
ckip=$(sed -n ${ip}p iplist)
gtdata=$(grep "$ckip" serverlog.tmp | awk '{print$7}' > tmpfile)
echo "Looking for a match in definitions"
rdln=$(wc -l tmpfile | awk '{print$1}')
if [[ "$rdln" -le "1" ]]
then
readata=$(sed -n 1p tmpfile)

match=$(grep "$readata" definition.conf | awk '{print$1}')
if [[ -z "$match" ]]
then
echo "No results were found"
exit 0
else
echo "Intrusion detected from $ckip was : $match"
exit 0
fi
fi
for i in $(seq "$rdln")
do
readata=$(sed -n ${i}p tmpfile)
match=$(grep "$readata" definition.conf | awk '{print$1}')
if [[ ! -z "$match" ]]
then
echo "Intrusion detected from $ckip was : $match"
exit 0
fi
done
echo "No results were found"
exit 0

}

rm iplist >/dev/null 2>&1
rm tmpfile >/dev/null 2>&1
echo "Checking ips in Log"
echo ""
var1=$(awk '{print$1}' serverlog.tmp | uniq > iplist)
cntip=$(wc -l iplist | awk '{print$1}')
selip
give execution permissions to scriptest and run it

NOTES
if you select the 1 then match was found because is exactly like it is in definition.conf

but despite the fact that option 2 and 3 in script exists also in definition.conf but are not exactly like , then it is not found .

Everyone now can run this example i wrote here to understand what i need .

Last edited by pedropt; 10-16-2019 at 11:38 AM.
 
Old 10-16-2019, 03:34 PM   #19
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Enjoy

Code:
#!/bin/bash
GetIPs () {
    sort -u < <(grep -Eo "[0-9]{1,3}(\.[0-9]{1,3}){3}" serverlog.tmp )
}

CheckPatterns () {
IP=$1
while read LOG
do
    for P in "${Patterns[@]}"
    do  
        [[ ${LOG} =~ ${P#* } ]] && echo ${P% *} matches ${IP}
    done
done < <(grep "$IP" serverlog.tmp )

return $?
}

Patterns=()
while read foo
do
    Patterns+=("${foo}")
done < definition.conf
while read IP
do
    CheckPatterns $IP
done < <( GetIPs )
build on it
 
Old 10-16-2019, 05:03 PM   #20
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
Somehow i believe that there is an issue with the code because it does not give any output .
 
Old 10-16-2019, 05:24 PM   #21
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by pedropt View Post
Somehow i believe that there is an issue with the code because it does not give any output .
I get output here

attached, rename removing .txt ( or add .txt to script )
Attached Files
File Type: txt SomeScript.sh.txt (415 Bytes, 20 views)
File Type: txt definition.conf.txt (774 Bytes, 24 views)
File Type: txt serverlog.tmp.txt (667 Bytes, 21 views)
 
Old 10-17-2019, 09:55 AM   #22
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
Thank you firerat , it works very well .
I did some changes in the code to set user input , instead checking everything .
I can adapt the code now for whatever i need .


Code:
#!/bin/bash


CheckPatterns () {
IP=$1
while read LOG
do
        for P in "${Patterns[@]}"
        do
                [[ ${LOG} =~ ${P#* } ]] && echo ${P% *} matches ${IP}
        done
done < <(grep "$IP" serverlog.tmp )

return $?
}
echo -n "Write IP : "
read -r ipd

Patterns=()
while read foo
do
        Patterns+=("${foo}")
done < definition.conf

while read IP
do
        CheckPatterns $IP 
done < <( echo "$ipd" )
 
Old 10-17-2019, 10:08 AM   #23
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by pedropt View Post
Thank you firerat , it works very well .
I did some changes in the code to set user input , instead checking everything .
I can adapt the code now for whatever i need .


Code:
#!/bin/bash


CheckPatterns () {
IP=$1
while read LOG
do
        for P in "${Patterns[@]}"
        do
                [[ ${LOG} =~ ${P#* } ]] && echo ${P% *} matches ${IP}
        done
done < <(grep "$IP" serverlog.tmp )

return $?
}
echo -n "Write IP : "
read -r ipd

Patterns=()
while read foo
do
        Patterns+=("${foo}")
done < definition.conf

while read IP
do
        CheckPatterns $IP 
done < <( echo "$ipd" )
yeah adapt ,, but EEEEK !!

UUOE !!

echo "$ipd"


just get rid of that while loop, and call teh check
Code:
#!/bin/bash
CheckPatterns () {
IP=$1
while read LOG
do
        for P in "${Patterns[@]}"
        do
                [[ ${LOG} =~ ${P#* } ]] && echo ${P% *} matches ${IP}
        done
done < <(grep "$IP" serverlog.tmp )

return $?
}
Patterns=()
while read foo
do
        Patterns+=("${foo}")
done < definition.conf

echo -n "Write IP : "
read -r ipd
# TODO check to see if $ipd looks like a valid ip address

CheckPatterns $ipd
 
Old 10-17-2019, 11:13 AM   #24
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,371

Rep: Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748Reputation: 2748
Creating a list of IP addresses to choose from requires reading the log file twice.
A bash script with what @Firerat might call a UUOP (Unnecessary use of pipe)
Code:
#!/bin/bash

IPs=$(cut -d" " -f1 server.log | sort -u)

PS3="Your choice (or Ctrl-C to quit): "
select myIP in $IPs; do
  awk -f server.awk -v IP=$myIP server.conf server.log
done
 
Old 10-17-2019, 11:22 AM   #25
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by allend View Post
Creating a list of IP addresses to choose from requires reading the log file twice.
A bash script with what @Firerat might call a UUOP (Unnecessary use of pipe)
Code:
#!/bin/bash

IPs=$(cut -d" " -f1 server.log | sort -u)

PS3="Your choice (or Ctrl-C to quit): "
select myIP in $IPs; do
  awk -f server.awk -v IP=$myIP server.conf server.log
done
bless, I already showed you how to get IPs
Code:
GetIPs(){
sort -u < <(grep -Eo "[0-9]{1,3}(\.[0-9]{1,3}){3}" server.log
}

IPs=($(GetIPs))

PS3="Your choice (or Ctrl-C to quit): "
select myIP in "${IPs[@]}"; do
  awk -f server.awk -v IP=$myIP server.conf server.log
done
 
Old 10-17-2019, 12:49 PM   #26
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
yes i know , everything is ok with the code .
 
Old 10-17-2019, 01:34 PM   #27
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by pedropt View Post
yes i know , everything is ok with the code .
you could look at it the other way round

i.e. look for pattern and get the IP

Code:
#!/bin/bash
serverlog=serverlog.tmp.txt
definition=definition.conf.txt
GetSuspect () {
    for P in "${Patterns[@]}"
    do  
        grep "${P#* }" "${serverlog}"
    done
}
Patterns=()
while read foo
do
    Patterns+=("${foo}")
done < ${definition}

time ( GetSuspect | grep -Eo "[0-9]{1,3}(\.[0-9]{1,3}){3}" )
time ( GetSuspect )
it really depends on your data which is faster

if you process the log by IP you can break once you find a match
if you process the log by Pattern. you may turn up the same IP multiple times
 
Old 10-17-2019, 01:50 PM   #28
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
If i tell you guys wont believe it , and i dont know why !!

the code does not work on my server where it is suppose to run , but it works on my station desktop !!!

both server and desktop have same linux distributions .

This never happened to me .
Any ideas why ? or how to know it doesnt work on server ?

This next image shows 3 windows ,
1st is an ls to the directory
2nd is a nano edit to the script (to show the code)
3rd is the execution of it

https://i.postimg.cc/2jMfFctT/s1.jpg

I checked both serverlog and definition.conf and both of them are normal with the content i posted here .
 
Old 10-17-2019, 02:36 PM   #29
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
off the top of my head I can't think of anything

If awk was involved I would assume it was a mawk vs gawk issue

is the very first line #!/bin/bash
it looks like you might have a blank first line
 
Old 10-17-2019, 03:03 PM   #30
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
Nop , there is no empty 1st line .
This is too much strange , i also downgraded grep version just to see if it was something with grep (just to test) , but it stays the same .
This code is pretty simple and effective for what i need , and now i cant use it !!??
This is too much bad luck for me .
I dont like to mess up repositories in my server , but i will try to upgrade bash from a different repository in another debian distribution to see how it works .
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
directory exists but no such directory exists yumito Linux - Newbie 3 06-09-2013 02:02 AM
[SOLVED] Search for a pattern and if it exists change some other text on the line KRevotsk Linux - Newbie 14 04-18-2013 02:23 PM
trying to change part of text with sequence for filename Adol Linux - Newbie 6 01-13-2013 11:02 AM
To exists or not to exists, this is the Q. Inbal Linux - Newbie 3 07-18-2006 06:04 AM
SIOCADDRT: File exists SIOCCADDRT: File Exists Failed to bring up eth0. opsraja Linux - Networking 0 01-10-2005 08:29 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 10:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration