LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Need help with parsing log file (https://www.linuxquestions.org/questions/linux-newbie-8/need-help-with-parsing-log-file-933191/)

sysmicuser 03-07-2012 06:28 AM
Need help with parsing log file
 
Hi Guru's,

I need some assistance from you in regards to parsing a log file.

What my aim or objective is to have a "Report" which would report files which have been added by user and not submitted to repository(Perforce) or any such file which is edited and changes not submitted to repository.

I am not writing here actual script as it is pretty staright forward and command it executes is pretty much p4 diff -f <file_name>...

What I am definitely writing is my grep command of script(set -xv)

Code:
echo "Following files are differing from repository, please submit them to repository"
++ echo 'Following files are differing from repository, please submit them to repository'
Following files are differing from repository, please submit them to repository
grep -E -v "===" -E "file(s) not on client" $LOGS_DIR/p4_opened_files.sh.log > /tmp/$$
++ grep -E -v === -E 'file(s) not on client' test/logs/p4_opened_files.sh.log
grep: file(s) not on client: No such file or directory
mailx -s "Opened files Report" test@testme.com < /tmp/$$
++ mailx -s 'Opened files Report' test@testme.com
Email which I get has following output.
Code:
test/logs/p4_opened_files.sh.log:batch - file(s) not on client.
test/logs/p4_opened_files.sh.log:build_essentials - file(s) not on client.
test/logs/p4_opened_files.sh.log:ccb_batch_restart_worker.sh.20120302 - file(s) not on client.
test/logs/p4_opened_files.sh.log:ccb_spl_ear_undeploy.sh - file(s) not on client.
test/logs/p4_opened_files.sh.log:orig_split_spl_ear.xml - file(s) not on client.
test/logs/p4_opened_files.sh.log:archive - file(s) not on client.
test/logs/p4_opened_files.sh.log:build_essentials - file(s) not on client.
testl/ogs/p4_opened_files.sh.log:check_disk_space.sh_BACKUP - file(s) not on client.
test/logs/p4_opened_files.sh.log:password.txt - file(s) not on client.
test/logs/p4_opened_files.sh.log:549c549
test/logs/p4_opened_files.sh.log:<        ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>        ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:560c560
test/logs/p4_opened_files.sh.log:<        ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>        ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:* - file(s) not on client.
test/logs/p4_opened_files.sh.log:oas_opmn_start.sh - file(s) not on client.
test/logs/p4_opened_files.sh.log:oas_opmn_stop.sh - file(s) not on client.
test/logs/p4_opened_files.sh.log:cdc_jobs - file(s) not on client.
test/logs/p4_opened_files.sh.log:set_bld02.sh - file(s) not on client.
The actual logfile output is.

Code:
batch - file(s) not on client.
build_essentials - file(s) not on client.
ccb_batch_restart_worker.sh.20120302 - file(s) not on client.
ccb_spl_ear_undeploy.sh - file(s) not on client.
orig_split_spl_ear.xml - file(s) not on client.
archive - file(s) not on client.
build_essentials - file(s) not on client.
check_disk_space.sh_BACKUP - file(s) not on client.
password.txt - file(s) not on client.
==== //glide/main/scm/ahscripts/common/build_and_deploy.sh#23 - test/ahscripts/common/build_and_deploy.sh ====
549c549
<        ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
---
>        ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
560c560
<        ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
---
>        ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
* - file(s) not on client.
oas_opmn_start.sh - file(s) not on client.
oas_opmn_stop.sh - file(s) not on client.
cdc_jobs - file(s) not on client.
set_bld02.sh - file(s) not on client.
==== //glide/main/scm/ahscripts/tomcat/tomcat_stop.sh#1 - test/ahscripts/tomcat/tomcat_stop.sh ====
The above output(in email) I do care, issue with it is.
1. it should only output file name and not whole path.
for example
Instead of
test/logs/p4_opened_files.sh.log:password.txt - file(s) not on client.
It should say
password.txt

2. Second Issue is I believe more complex to script

Have a look at actual log file output.
Code:
==== //glide/main/scm/ahscripts/common/build_and_deploy.sh#23 - test/ahscripts/common/build_and_deploy.sh ====
549c549
<        ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
---
>        ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
560c560
<        ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
---
>        ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
* - file(s) not on client.
oas_opmn_start.sh - file(s) not on client.
But in email I get some non-friendly output...

Code:
test/logs/p4_opened_files.sh.log:password.txt - file(s) not on client.
test/logs/p4_opened_files.sh.log:549c549
test/logs/p4_opened_files.sh.log:<        ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>        ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:560c560
test/logs/p4_opened_files.sh.log:<        ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>        ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:* - file(s) not on client.
Instead it should give only name of file that is build_and_deploy.sh


How can I achieve my objective?

Please assist me here.

Many Thanks.

kbp 03-08-2012 06:58 PM
For the first issue you could just install biabam and send the log file as an attachment instead. The path is being shown because of it's relative location to the current working directory of the script, so you'd need to change dir.

In regard to the second issue, you need to add extra filters to your grep to take out the other stuff you don't want

Code:
grep -v xxx | grep -v yyy | grep -v zzz
BTW .. is Glide going to go live some year soon?

sysmicuser 03-08-2012 10:08 PM
hhaha! Glide is already live mate !

Ok, I shall try the technique you described.

sysmicuser 03-09-2012 04:15 AM
@kbp

Code:
grep -v xxx | grep -v yyy | grep -v zzz
Instead I performed
Code:
grep -e -v '===' -e "not on client" p4_opened_files.sh.log
but it gives error such as
Code:
grep: ===: No such file or directory
not so cool about it :(

kbp 03-09-2012 07:45 AM
The pattern needs to be directly after '-e', try this:

Code:
grep -v -e '===' -e "not on client" p4_opened_files.sh.log

sysmicuser 03-10-2012 05:50 PM
@kbp

Thanks for your assistance.
I am still working on it.

Interesting enough !
Code:
grep -v -e '===' -e "not on client" p4_opened_files.sh.log
doesn't give any output!

The final grep which at the moment is working is something like this.
Code:
grep -h "not on client" $LOG_FILE >> /tmp/$$
The out gives name of files but I am want name of directory where file is located. May be I need to use awk some where.
Code:
test.sh - file(s) not on client.

working and still in cooking phase :)


All times are GMT -5. The time now is 03:41 AM.