Need help with parsing log file

sysmicuser · 03-07-2012, 06:28 AM

Hi Guru's,

I need some assistance from you in regards to parsing a log file.

What my aim or objective is to have a "Report" which would report files which have been added by user and not submitted to repository(Perforce) or any such file which is edited and changes not submitted to repository.

I am not writing here actual script as it is pretty staright forward and command it executes is pretty much p4 diff -f <file_name>...

What I am definitely writing is my grep command of script(set -xv)

Code:

echo "Following files are differing from repository, please submit them to repository"
++ echo 'Following files are differing from repository, please submit them to repository'
Following files are differing from repository, please submit them to repository
grep -E -v "===" -E "file(s) not on client" $LOGS_DIR/p4_opened_files.sh.log > /tmp/$$
++ grep -E -v === -E 'file(s) not on client' test/logs/p4_opened_files.sh.log
grep: file(s) not on client: No such file or directory
mailx -s "Opened files Report" test@testme.com < /tmp/$$
++ mailx -s 'Opened files Report' test@testme.com

Email which I get has following output.

Code:

test/logs/p4_opened_files.sh.log:batch - file(s) not on client.
test/logs/p4_opened_files.sh.log:build_essentials - file(s) not on client.
test/logs/p4_opened_files.sh.log:ccb_batch_restart_worker.sh.20120302 - file(s) not on client.
test/logs/p4_opened_files.sh.log:ccb_spl_ear_undeploy.sh - file(s) not on client.
test/logs/p4_opened_files.sh.log:orig_split_spl_ear.xml - file(s) not on client.
test/logs/p4_opened_files.sh.log:archive - file(s) not on client.
test/logs/p4_opened_files.sh.log:build_essentials - file(s) not on client.
testl/ogs/p4_opened_files.sh.log:check_disk_space.sh_BACKUP - file(s) not on client.
test/logs/p4_opened_files.sh.log:password.txt - file(s) not on client.
test/logs/p4_opened_files.sh.log:549c549
test/logs/p4_opened_files.sh.log:<         ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>         ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:560c560
test/logs/p4_opened_files.sh.log:<         ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>         ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:* - file(s) not on client.
test/logs/p4_opened_files.sh.log:oas_opmn_start.sh - file(s) not on client.
test/logs/p4_opened_files.sh.log:oas_opmn_stop.sh - file(s) not on client.
test/logs/p4_opened_files.sh.log:cdc_jobs - file(s) not on client.
test/logs/p4_opened_files.sh.log:set_bld02.sh - file(s) not on client.

The actual logfile output is.

Code:

batch - file(s) not on client.
build_essentials - file(s) not on client.
ccb_batch_restart_worker.sh.20120302 - file(s) not on client.
ccb_spl_ear_undeploy.sh - file(s) not on client.
orig_split_spl_ear.xml - file(s) not on client.
archive - file(s) not on client.
build_essentials - file(s) not on client.
check_disk_space.sh_BACKUP - file(s) not on client.
password.txt - file(s) not on client.
==== //glide/main/scm/ahscripts/common/build_and_deploy.sh#23 - test/ahscripts/common/build_and_deploy.sh ====
549c549
<         ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
---
>         ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
560c560
<         ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
---
>         ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
* - file(s) not on client.
oas_opmn_start.sh - file(s) not on client.
oas_opmn_stop.sh - file(s) not on client.
cdc_jobs - file(s) not on client.
set_bld02.sh - file(s) not on client.
==== //glide/main/scm/ahscripts/tomcat/tomcat_stop.sh#1 - test/ahscripts/tomcat/tomcat_stop.sh ====

The above output(in email) I do care, issue with it is.
1. it should only output file name and not whole path.
for example
Instead of
test/logs/p4_opened_files.sh.logassword.txt - file(s) not on client.
It should say
password.txt

2. Second Issue is I believe more complex to script

Have a look at actual log file output.

Code:

==== //glide/main/scm/ahscripts/common/build_and_deploy.sh#23 - test/ahscripts/common/build_and_deploy.sh ====
549c549
<         ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
---
>         ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
560c560
<         ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
---
>         ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
* - file(s) not on client.
oas_opmn_start.sh - file(s) not on client.

But in email I get some non-friendly output...

Code:

test/logs/p4_opened_files.sh.log:password.txt - file(s) not on client.
test/logs/p4_opened_files.sh.log:549c549
test/logs/p4_opened_files.sh.log:<         ant clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>         ant -v clean build deploy 2>&1  | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:560c560
test/logs/p4_opened_files.sh.log:<         ant clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:---
test/logs/p4_opened_files.sh.log:>         ant -v clean build 2>&1 | tee -a ${IGL_LOGFILE}
test/logs/p4_opened_files.sh.log:* - file(s) not on client.

Instead it should give only name of file that is build_and_deploy.sh

How can I achieve my objective?

Please assist me here.

Many Thanks.

kbp · 03-08-2012, 06:58 PM

For the first issue you could just install biabam and send the log file as an attachment instead. The path is being shown because of it's relative location to the current working directory of the script, so you'd need to change dir.

In regard to the second issue, you need to add extra filters to your grep to take out the other stuff you don't want

Code:

grep -v xxx | grep -v yyy | grep -v zzz

BTW .. is Glide going to go live some year soon?

sysmicuser · 03-08-2012, 10:08 PM

hhaha! Glide is already live mate !

Ok, I shall try the technique you described.

sysmicuser · 03-09-2012, 04:15 AM

@kbp

Code:

grep -v xxx | grep -v yyy | grep -v zzz

Instead I performed

Code:

grep -e -v '===' -e "not on client" p4_opened_files.sh.log

but it gives error such as

Code:

grep: ===: No such file or directory

not so cool about it

kbp · 03-09-2012, 07:45 AM

The pattern needs to be directly after '-e', try this:

Code:

grep -v -e '===' -e "not on client" p4_opened_files.sh.log

sysmicuser · 03-10-2012, 05:50 PM

@kbp

Thanks for your assistance.
I am still working on it.

Interesting enough !

Code:

grep -v -e '===' -e "not on client" p4_opened_files.sh.log

doesn't give any output!

The final grep which at the moment is working is something like this.

Code:

grep -h "not on client" $LOG_FILE >> /tmp/$$

The out gives name of files but I am want name of directory where file is located. May be I need to use awk some where.

Code:

test.sh - file(s) not on client.

working and still in cooking phase