Search text if some part sequence exists
 

Hi guys , i dont even know how to write the topic to match what i need .

Here it is , i am writing a definition file with errors messages appears in my web log servers .

Example of def.conf

Now in web server log this line could be in many forms but that specific sequence is there , by this i mean :


Now , when i put my script searching the definition log , i will use the variable i have in the log , witch is :
"/index.php/module/action/param1 ${@die(md5(HelloThinkPHP))}"

and i want the script to identify that this line belongs to "/module/action/param1 ${@die(md5(HelloThinkPHP))}" , and then i will retrieve with awk the variable $1 witch is ThinkPHP_RCE.

How can i do this ?

Can you go into a little more detail and give one or two more examples? It sounds like you want to read patterns in from one file and search for them in a second. If that is the case, you might have to escalate to perl to avoid lots of loops in AWK.

I am going to be caned for this, but given
and
then
This reads the def.conf file into two arrays, then processes the def.log file. The use of length() is a gawk extension.

Thanks both of you , Allend is almost there , the problem is i can not rely on 2 last characters found , because it is not enough and a lot of false positives will appear .
One of the difficulties here is that is have more text in the variable than on the file that will provid me the output i want .

If i have a file with definitions like :

1 rttrh/456430/ewrewr/88000
2 3907/weewrerw/2332/ertet

and i send the script to search the definition file above with this variable :

blalbalb/rttrh/456430/ewrewr/88000

then i am stuck because nothing will be found .
Another alternative would be the inverse , witch means picking line by line on definitions file and search on the log , this way will work because the variable will be small :

if i search for :
rttrh/456430/ewrewr/88000

in

blalbalb/rttrh/456430/ewrewr/88000

then i will have a positive output , but will waste a lot of resources and time to do it line by line .

Now , one this that will do the job will be removing the text untile first slash , and then search , if nothing found then remove the text until next front slash .
This way will work , but eventually i will do a lot of searches with not result that will increase time to the script .

Que?
This moves the goal posts from your original post.
Excuse me, but where was this in your original post?

At this forum, you get ideas, not complete solutions.

What is missing for us is: Precisely how much of the matched string can you rely on?

Can you provide a clear example of a single match pattern from the def file, along with a few lines which should match, and a few which should not match. I have tried to see that from your examples already given but without success.

allend , look , i didnt move from my original post , i just give another example .

The definition file is where it will search will be :
is exactly the same as :

Definitions is some file where i will store all the variables to be compared with .
The ip address on first post was just an example , of course that i will not send the ip address to grep , i will send only what i need to search .

-------------------------------------------------------------------

astrogeek , you are right , i tought about that before i made my last post .

from this example :
i can leave out "/index.php" because that name file changes , so i will rely only on
Now what i need is not how to put the 1st line as the second line "by removing everything until the 2nd front slash .
What i need is the fastest way to look into a big file for that combination .
I usually use grep , but for heavy files maybe it would be interesting to use something a little more faster .

However i have here an issue , the problem is that every line is different , and this can not be applied for 1 single case .
i have lines in log like this :
/HNAP1

with is an information disclosure to dlink routers (i believe) , on this case i can not remove until the 1st front slash .
Thinking a little bit better , what i really need is to see if on the beginning of the variable is a file or a directory .

directory = /something
file = /somethiong.php/OTHERSTUFF"

case is file then i will remove until 2nd front slash and use the rest , else use the complete variable .

Basically what i need is a quick way to search .

That still does not specify how much you can rely on very precisely.

Question: Can you rely on there always being a string matching /module/action/param1 in every line you want to search for?

But you have not said what you want to do in these cases. Ignore the line? Search for the line? What?

UPDATE: Think in terms of your thread title:

Define for us exactly the part sequence which exists and is used to trigger the search.

Thanks for the reply astrogeek and everyone else here trying to help and trying to understand what the heck do i need .

Well , first let me post here some real log examples that anyone gets on their servers from attempts of exploiting .

Lets call it Server.log
Now , the definition file contains a sequence that it could be equal or not to what i have in log , this will be this way because by default hackers use automated scripts with potential directories , these scripts they use run a list of potential directories .
For me this means that i dont need to write in definitions file every line , i just need to write one line that i will know that they will use for sure , this way i can identify the technique used .

On the above QUOTE ; there are multiple exploitations they have try , but before start digging the definitions file for what they were after , my script 1st must identify what kind of request was made to the server .

From the above Quote what script must search :

Line 1 = /wp-content/plugins/portable-phpmyadmin/wp-pma-mod/
Line 2 = /HNAP1/
Line 3 = /prov/aastra.cfg
Line 4 = /f4bb336d/
Line 5 = Ignore
Line 6 = /module/action/param1/${@die(md5(HelloThinkPHP))}
Line 7 = /App/?content=die(md5(HelloThinkPHP))
Line 8 = /editBlackAndWhiteList
Line 9 = /0015650000000.cfg

How it should do in code :

if last text of variable is a file , and is a .php then remove that text file and search .
If it does not have any file in the beginning or end then search (Line 2)
if after a directory a file text exists but it is not php then search without removing anything .
if it starts with a filename other than php then search all .

Resuming :

- a)Detect if "anything.php" exists in the beginning or at the end of variable and remove it .
- b) Case a) code is true then execute it and search .
- Case a code is false then search

Now what is more important in the code is a fast search .

Looks a lot like you are reinventing modsecurity...

Your line 5 case seems at odds with your rule "anything.php = remove and search" as stated. How would the script know to ignore it?

What do you want to get as the final output? The lines from the log or simply a count of the matching lines?

How do you intend to use this? Near real time as lines are added to the logs? Once per day/week to extract stats? For reporting purposes or blocking purposes?

There is a lot of relevant info we do not have.

At the very least I think that you have the problem, as stated so far, backwards - instead of searching the logs, mangling the lines then searching the definitions for a match with the mangle, simply search the logs for matches to the second part of the definitions one definition at a time, replace matches, skip others.

That said, I don't think your problem is yet well enough defined as indicated by the line 5 mismatch, and I would suggest looking at a rule set for modsecurity to see what is actually involved in matching common exploits by regular expression.

Given server.log (taking out the space beteen the l and e in portable in what was posted)
and server.conf (escaping the characters used in creating regular expressions)
and server.awk
then

Nice code allend , i will probably have to adjust it and remove the loop .


The comparison will not be directly to server log , before it checks to your code i will remove the

to be only in variable :
This will be done ip by ip , this means that i will choose firstly the ip , then from that your code will start to identify what was that ip doing in server , after your code i will add some other code that in case nothing was found in definitions file (server.conf) , then will ask me to add a new line to definitions file for future detection .
Great code indeed , i was not expecting it was so simple to do it .

I did not yet marked this thread as solved because i am having difficulties to export the code
inside to the script without having an additional instruction file "server.awk" .

Exporting it normally to a script how it should be ?
Example :

var1 = some strings to be matched in server.conf
if the string is matches then stop , else continue checking other lines .

What i am doing here is :
After i select an ip to be checked in webservers log i firstly will grab all the log data from that ip to a temp file , and then i will need this code to compare the ip requests in temp.tmp file with the definitions file "server.conf" , the loop will read the first line of the temp file and will search in the definitions file "server.conf" if it matches , in case something was found then stop there and bring back the results .

Something like this :
After exporting the ip data to a tempfile

Note : temp.tmp file will be a cleaned file with only the requests that ip made , no more data will be there .
An example of temp.tmp file would be this :

you really are making life complicated by using sed the way you are

and I don't understand what you are actually trying to do


The patterns in server.conf need to be patterns , at the moment they are not.

how does IP relate?

Not necessary.
Given server.log
and server.awk
then