LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-15-2013, 10:57 AM   #16
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian Jessie / sid
Posts: 1,471

Rep: Reputation: 444Reputation: 444Reputation: 444Reputation: 444Reputation: 444

learnt a bit ( and I mean a bit ) of c++

PHP Code:
#include <iostream>
#include <fstream>
#include <string>
/* 
/
/ NO ERROR CHECKS OR PROPER STUFF..
/
*/

/*
char inputFilename[] = "Input.log";
int main () {
*/
int main (int argcchar *argv[]) {
    
using std::string;
    
string FEILD;
    
string GET;
    
string MESSAGE;
    
string STRING// I don't know how to 'drop' the rest of a line
    
using std::ifstream;
    
ifstream infile;
    
using std::ios;
    
//infile.open (inputFilename,ios::in);
    
for(int i=1i<argci++) {
        
infile.open (argv[i], ios::in);
        while(!
infile.eof()) {
            
infile >> FEILD;
            if (
FEILD == "GET") {
                
infile >> GET;
                
// 'work' on GET here
            
} else if (FEILD == "Message:") {
                while(
FEILD != "[accuracy") {
                    
infile >> FEILD;
                }
                
infile >> MESSAGE;
                
std::size_t First MESSAGE.find_first_of("\"");
                
std::size_t Last MESSAGE.find_last_of("\"");
                
using std::cout;
                
cout << GET << " " << MESSAGE.substr(First+1,Last-1) << "\n";
                
// instead of 'printing' pass to your fuzzy stuffs
            
} else {
                
// because I have no idea how to just 'drop' it
                
getline(infile,STRING);
            }
        } 
infile.close();
    }

Code:
./a.out /path/to/logs/*.log /path/to/another.log
 
Old 08-16-2013, 02:12 AM   #17
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Read .log format file and get special character from some lines by shell scripting

Hi again dear,

I did not test c++ program but o did test the shell script that you send me. I just make a file that contains two log file. my file just contains two log file that in one of the i Write after GET, http://dvwa.com. i did it myself. but for the other log i put something that was after GET to be itself. after that in the program that was in shell scripting format i did some thing like this:

#!/usr/bin/awk -f
BEGIN{Third=0}{
if ($1 == "GET") {
Get=$2; if ( GET != "Http://dvwa.com" )
Get=9
}
if ($1 == "Message:") {
sub(/^Message:.+\[accuracy/,"",$0);
gsub(/[^[:digit:]]/,"",$1);
Acc=$1;
printf "%d %d %d\n",Acc,Get,Third;
}
}
but it creates 2 output as the same:

8 9 0
8 9 0
and i think it never check the if part. because when i change the dvwa.com to other thing the output is always like this. i do not check the c++ program because i think it raise the complexity of doing my work.
waiting for your rply
 
Old 08-16-2013, 02:24 AM   #18
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian Jessie / sid
Posts: 1,471

Rep: Reputation: 444Reputation: 444Reputation: 444Reputation: 444Reputation: 444
[code]please please please PLEASE[/code] wrap code in code tags......

Yes, no real advantage to the c++ I posted..
Well apart from that fact that you asked a few times how to get the output of awk into your c++ program.. So I learnt a little and now you have an example of how to open a file with your c++ program.


Code:
Get=$2; if ( GET != "Http://dvwa.com" )
What do you think the problem with that is?
Does it match http://dvwa.com ?
 
Old 08-16-2013, 07:08 AM   #19
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Read .log format file and get special character from some lines by shell scripting

I think the problem is that the program do not check the if part and it just always and always put get=9. my logfile was consist of 2 logs like this:
log file:
Quote:
--25814763-B--
GET Http://dvwa.com
Host: 28.99.169.38
Connection: close
Accept: */*
Accept-Charset: *
Accept-Encoding: *;q=0.7
Accept-Language: ondtmsih-rqjk1;q=0.6
Cache-Control: upy='aels'
Cookie2: $Version="388"
Date: Thu, 27 Apr 06 24:36:50 UTC
ETag: "kAPqiaNm18b1MKgZ"
If-Modified-Since: Thu, 18 May 06 01:58:48 GMT
If-Unmodified-Since: Fri, 19 Sep 08 10:20:47 CET
If-Match: *
If-None-Match: "mGN7VPxNspBNWF50vLeI"
Pragma: vthu=ifdlS4il
Authorization: Basic TnJ3MDY6UWVzc29sY2U=
Referer: http://www.hao8.de/0ehnqceo/segfto2z...f/tmdpe2En.avi
TE: deflate;q=0.7,deflate;q=0.1,deflate;q=0.2
User-Agent: rfno5rdhfNnssmascs
UA-CPU: MIPS
Via: FTP/1.8 200.207.173.114
Transfer-Encoding: wria
Warning: 540 www.plsshi.jpeg "rdomaoAwf2ahhud2t" "Fri, 15 Sep 06 13:24:25 CET"

--25814763-F--
--25814763-H--
Message: Access denied with code 406 (phase 2). Pattern match "(^[\"'`\xc2\xb4\xe2\x80\x99\xe2\x80\x98;]+|[\"'`\xc2\xb4\xe2\x80\x99\xe2\x80\x98;]+$)" at ARGS:Settotzeertnl. [file "/usr/local/apachhe/conf/samane_rules/SpiderLabs-owasp-modsecurity-crs-33612c6/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "64"] [id "981318"] [rev "2"] [msg "SQL Injection Attack: Common Injection Testing Detected"] [data "Matched Data: ' found within ARGS:Settotzeertnl: 'pn "] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.7"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB _ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Apache-Error: [file "http_filters.c"] [line 262] [level 3] Unknown Transfer-Encoding: wria, referer: http://www.hao8.de/0ehnqceo/segfto2z...f/tmdpe2En.avi
Action: Intercepted (phase 2)
Stopwatch: 1374417991011250 5572 (- - -)
Stopwatch2: 1374417991011250 5572; combined=867, p1=10, p2=830, p3=0, p4=0, p5=26, sr=0, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.2 (http://www.modsecurity.org/).
Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips DAV/2 PHP/5.4.12
Engine-Mode: "ENABLED"

--25814763-Z--
--25814763-B--
GET w7e/neQhmsdwu7imdb0etet/eT/hsvegbff/EH/niRAvLwGK_L/osLnBWcHRk5oGMI/tmLJFqSww/sSjS6KRJB.html?Settotzeertnl=%27pn+&8nafitm=74LuKUC5t0J&4ttNe=Anmsyusi6&Mf1g-vYqyx=elTTsw&Euoytxp$
Host: 28.99.169.38
Connection: close
Accept: */*
Accept-Charset: *
Accept-Encoding: *;q=0.7
Accept-Language: ondtmsih-rqjk1;q=0.6
Cache-Control: upy='aels'
Cookie2: $Version="388"
Date: Thu, 27 Apr 06 24:36:50 UTC
ETag: "kAPqiaNm18b1MKgZ"
If-Modified-Since: Thu, 18 May 06 01:58:48 GMT
If-Unmodified-Since: Fri, 19 Sep 08 10:20:47 CET
If-Match: *
If-None-Match: "mGN7VPxNspBNWF50vLeI"
Pragma: vthu=ifdlS4il
Authorization: Basic TnJ3MDY6UWVzc29sY2U=
Referer: http://www.hao8.de/0ehnqceo/segfto2z...f/tmdpe2En.avi
TE: deflate;q=0.7,deflate;q=0.1,deflate;q=0.2
User-Agent: rfno5rdhfNnssmascs
UA-CPU: MIPS
Via: FTP/1.8 200.207.173.114
Transfer-Encoding: wria
Warning: 540 www.plsshi.jpeg "rdomaoAwf2ahhud2t" "Fri, 15 Sep 06 13:24:25 CET"

--25814763-F--
--25814763-H--
Message: Access denied with code 406 (phase 2). Pattern match "(^[\"'`\xc2\xb4\xe2\x80\x99\xe2\x80\x98;]+|[\"'`\xc2\xb4\xe2\x80\x99\xe2\x80\x98;]+$)" at ARGS:Settotzeertnl. [file "/usr/local/apachhe/conf/samane_rules/SpiderLabs-owasp-modsecurity-crs-33612c6/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "64"] [id "981318"] [rev "2"] [msg "SQL Injection Attack: Common Injection Testing Detected"] [data "Matched Data: ' found within ARGS:Settotzeertnl: 'pn "] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.7"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB _ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Apache-Error: [file "http_filters.c"] [line 262] [level 3] Unknown Transfer-Encoding: wria, referer: http://www.hao8.de/0ehnqceo/segfto2z...f/tmdpe2En.avi
Action: Intercepted (phase 2)
Stopwatch: 1374417991011250 5572 (- - -)
Stopwatch2: 1374417991011250 5572; combined=867, p1=10, p2=830, p3=0, p4=0, p5=26, sr=0, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.2 (http://www.modsecurity.org/).
Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips DAV/2 PHP/5.4.12
Engine-Mode: "ENABLED"

--25814763-Z--
but the output of this is like this:
Quote:
8 9 0
8 9 0
Can i ask you to check it?
Thanks a lot
 
Old 08-16-2013, 10:15 AM   #20
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,129

Rep: Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876
Quote:
Originally Posted by samasara View Post
...
and i think it never check the if part. because when i change the dvwa.com to other thing the output is always like this. i do not check the c++ program because i think it raise the complexity of doing my work.
waiting for your rply
i think that this description is already unnecessary complicated. partly because the description isnt very clear. seems like you are trying to output 3 variables. lets start with the first.

Acc = the number after the word 'accuracy' on any line beggining with 'Message' which is '8' in your example.

Get = the string in the 2nd field of any line that begins with the word 'GET'; however if the 2nd field is not equal "Http://dvwa.com" then you will replace the value of Get with '9' (notice that Get is not the same thing as GET).

Third is always = '0'.
 
Old 08-16-2013, 10:29 AM   #21
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Read .log format file and get special character from some lines by shell scripting

Hi dear,
You are right. the description that you wrote is ok and right. I know what the program say and i know that the get is not equal to GET. But You see that in my log file i had 2 logs. In one of them in B part after GET is Http://dvwa.com. so the output for first log should not be 8 9 0 it should be some other number that of course we should identify in program for example with else . But for the second log in my log file after GET part i have something like this :
Quote:
w7e/neQhmsdwu7imdb0etet/eT/hsvegbff/EH/niRAvLwGK_L/osLnBWcHRk5oGMI/tmLJFqSww/sSjS6KRJB.html?Settotzeertnl=%27pn+&8nafitm=74LuKUC5t0J&4ttNe=Anmsyusi6&Mf1g-vYqyx=elTTsw&Euoytxp$
So for this log that the GET part is something different from HTTP://dvwa.com the output should be 8 9 0. the problem is that why for both log i have 8 9 0 since for the first one it should not be this.
 
Old 08-16-2013, 10:41 AM   #22
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,129

Rep: Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876Reputation: 876
you never assign GET (Get is not the same thing as GET).

so the interpreter is working correctly:
Code:
...
if ($1 == "GET") {                              # if "GET" is the same as "GET" -- true
Get=$2;                                         # then Get is assigned the value "Http://dvwa.com"
 if ( GET != "Http://dvwa.com" )                # if '\0' is not the same as "Http://dvwa.com" -- true
Get=9                                           # then Get is assigned the value of 9
}
...
check out my comments above.
 
2 members found this post helpful.
Old 08-16-2013, 12:42 PM   #23
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian Jessie / sid
Posts: 1,471

Rep: Reputation: 444Reputation: 444Reputation: 444Reputation: 444Reputation: 444
Quote:
Originally Posted by schneidz View Post
you never assign GET (Get is not the same thing as GET).
DOH!
my fault, I must have done ctrl+p in vim!!
Edit, no wait I added that bit 'inline' via web page

Last edited by Firerat; 08-16-2013 at 01:18 PM.
 
1 members found this post helpful.
Old 08-16-2013, 03:08 PM   #24
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Read .log format file and get special character from some lines by shell scripting

Hi dear users,

I do not now how can i fix this problem? how can i use the program for doing it correctly? How can i change this program to be used well and correctly?

Thanks alot
Regards
 
Old 08-16-2013, 04:00 PM   #25
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian Jessie / sid
Posts: 1,471

Rep: Reputation: 444Reputation: 444Reputation: 444Reputation: 444Reputation: 444
change GET != to Get !=


totally my fault
 
1 members found this post helpful.
Old 08-17-2013, 02:42 AM   #26
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Read .log format file and get special character from some lines by shell scripting

Hi dear users,
Really thanks for your kind and help.

Best Regards
 
Old 09-27-2013, 03:05 AM   #27
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
compare to log file

hi dear users,
really thank you for your last help to me. Now i have another question. last time when i send the output of my last program to another program for further calculation( as you remembered i have something like this Parselog1.awk /pass to *.log | fuzzy code2.cpp : i named this shell my.sh), then i had another shell named my2.sh that get the output of first shell for further calculating. i do this for getting the final output:./my2.sh $(./my.sh). Now i have a question:

I know for getting the latest log that has been created in my log file i should use the command tail -f. what i want to do is each time i want to get the latest log file then send its parameters to the fuzzy code2.cpp program. till now the risk related to that logfile would be calculated. then i pass the output of that to my2.sh to act for further action related to risk that has been calculated(in my2.sh i say for example if risk = 0 restart the machine). now assume that another log has been created. i want each time get the parameters of latest log that has created then i compare its parameter with the last log that we calculate some calculation on it before this new log. if the parameters were the same we again calculate the risk of log with fuzzycode and then if the risk was greater than a threshold we add one to calculated risk(Risk+1) and pass it to other program. i want to know how would it be? how can i do something like this?

the sample of log file is like this:

Quote:
--25814763-B--
GET Http://dvwa.com
Host: 28.99.169.38
Connection: close
Accept: */*
Accept-Charset: *
Accept-Encoding: *;q=0.7
Accept-Language: ondtmsih-rqjk1;q=0.6
Cache-Control: upy='aels'
Cookie2: $Version="388"
Date: Thu, 27 Apr 06 24:36:50 UTC
ETag: "kAPqiaNm18b1MKgZ"
If-Modified-Since: Thu, 18 May 06 01:58:48 GMT
If-Unmodified-Since: Fri, 19 Sep 08 10:20:47 CET
If-Match: *
If-None-Match: "mGN7VPxNspBNWF50vLeI"
Pragma: vthu=ifdlS4il
Authorization: Basic TnJ3MDY6UWVzc29sY2U=
Referer: http://www.hao8.de/0ehnqceo/segfto2z...f/tmdpe2En.avi
TE: deflate;q=0.7,deflate;q=0.1,deflate;q=0.2
User-Agent: rfno5rdhfNnssmascs
UA-CPU: MIPS
Via: FTP/1.8 200.207.173.114
Transfer-Encoding: wria
Warning: 540 www.plsshi.jpeg "rdomaoAwf2ahhud2t" "Fri, 15 Sep 06 13:24:25 CET"

--25814763-F--
--25814763-H--
Message: Access denied with code 406 (phase 2). Pattern match "(^[\"'`\xc2\xb4\xe2\x80\x99\xe2\x80\x98;]+|[\"'`\xc2\xb4\xe2\x80\x99\xe2\x80\x98;]+$)" at ARGS:Settotzeertnl. [file "/usr/local/apachhe/conf/samane_rules/SpiderLabs-owasp-modsecurity-crs-33612c6/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "64"] [id "981318"] [rev "2"] [msg "SQL Injection Attack: Common Injection Testing Detected"] [data "Matched Data: ' found within ARGS:Settotzeertnl: 'pn "] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.7"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB _ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Apache-Error: [file "http_filters.c"] [line 262] [level 3] Unknown Transfer-Encoding: wria, referer: http://www.hao8.de/0ehnqceo/segfto2z...f/tmdpe2En.avi
Action: Intercepted (phase 2)
Stopwatch: 1374417991011250 5572 (- - -)
Stopwatch2: 1374417991011250 5572; combined=867, p1=10, p2=830, p3=0, p4=0, p5=26, sr=0, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.2 (http://www.modsecurity.org/).
Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips DAV/2 PHP/5.4.12
Engine-Mode: "ENABLED"

--25814763-Z--
the parameter that should be compare is the [msg"..................."]part that contains the attack type in section H, the accuracy".." part again in section H, the Get .....part in section B and the Host .... part again in section B. how can i compare these parts each time from latest log with the parameters in last log that has been created before that?
really thank you for your kind and help.

Last edited by samasara; 09-27-2013 at 03:08 AM.
 
Old 09-27-2013, 03:31 AM   #28
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian Jessie / sid
Posts: 1,471

Rep: Reputation: 444Reputation: 444Reputation: 444Reputation: 444Reputation: 444
so
you have a shell script which monitors a log

when new 'records' you analyse them
if safe, no more action
if a risk you send output to a newlog

You have a second shell script which monitors newlog
if safe, no more action
if a risk you send output to 'Fuzzy'

Fuzzy performs further analysis
if safe, no more action
if a risk you 'do more things'


Personally I think I would have Fuzzy 'do the lot'

But honestly, we have written a number of scripts for you
what you ask for now is simply a variation
It is time you sat down an learnt how to write them yourself

I will be happy to help if you show willing yourself

You see the thing is, you are being paid for what *we* are doing for you
 
Old 09-27-2013, 06:55 AM   #29
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Thread: Read .log format file and get special character from some lines by shell scripting

hi firerat,

Thanks a lot for your kind to me. I just want to know how to get the two latest log each time. no more code or anything else.
Thank you
 
Old 09-29-2013, 08:48 PM   #30
samasara
Member
 
Registered: Aug 2013
Posts: 34

Original Poster
Rep: Reputation: Disabled
Read .log format file and get special character from some lines by shell scripting

hi firerat,

i solved my problem myself.

thank you
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Find and Replace character/special character from the file MyRelam Red Hat 8 05-21-2012 12:52 AM
[SOLVED] read from keyboard while reading from file in SHELL SCRIPTING m3ll0 Programming 11 10-30-2010 08:30 AM
Shell script to read lines in a text file and filter user data srimal Linux - Newbie 5 10-21-2009 07:41 AM
Inserting lines into a file through shell scripting false-hopes Linux - General 1 10-22-2005 11:39 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 02:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration