how to extract paragraphs from file in BASH script followed by prefix ! , !! and !!!

nabmufti · 02-08-2008, 05:01 PM

hi all
i am in confusion since last 2 days
i posted thraed yesterday and some friends did help but still i couldnt get solution to my probleml

let it be very clear

i have a long log file of alkatel switch and i have to seperate the minor major and critical alarms shown by ! , !! and !!! respectively

the file format is like this

>> ALAIL;

CMD ALAIL 06095 08-01-30 11-58-23
INP ALAIL 06095 08-01-30 11-58-23
OPTEX=

RES ALAIL 06095 08-01-30 11-58-24
CEN=1/08-01-30/11 H 58 MN 19/STORED ALARMS LIST
PROCESSING TPFFIR ACC

! *A0628/540 /07-12-17/15 H 58/N=7598/TYP=ICT/CAT=SI/EVENT=MAL
/NCEN=MULCT /AM =SMTA1/AGEO=S1-TR01-B03-A085-R000
/TEXAL=AIS/COMPL.INF:
/AF=URMA1
/ ICTRQ AGCA=S1-TR01-B03-A112-R065
/AMET=01-07-02
/AFLR=222-09/CRC=NACT
!!! *A0628/303 /07-12-17/15 H 46/N=7501/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 4/AGEO=BUDLASAN-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=3

!!! *A0628/306 /08-01-23/07 H 20/N=9619/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFCN =CNLM3-4/AGEO=MKRASHID-B02-A003
/TEXAL=CN ISOLATED
!!! *A0628/303 /07-12-17/15 H 46/N=7503/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 12/AGEO=KHOTYWAL-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=0
!!! *A0628/089 /08-01-29/22 H 14/N=1760/TYP=ENP/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 11/AGEO=QADIRPUR-B01
/TEXAL=GENERATING UNIT

!! *A0628/320 /08-01-17/13 H 47/N=8062/TYP=COM/CAT=ID/EVENT=MAL
/NCEN=MULCT /AFUR =URAL- 15/AGEO=S1-TR01-B04
/TEXAL=FAULTY UNIT/COMPL.INF:
* #F0612/T11F14/NCEN=MULCT /08-01-30/11 H 33/NAM=ODCAB /TDA=0001
/N=7064/NIV=2/ENS=006/SENS=013-000-000/P='0228'H/CN=02/PAR='03'H
/EM: AFUR =URAL- 15
!!! *A0628/303 /07-12-17/15 H 46/N=7506/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 7/AGEO=ADABOSAN-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=2

!!! *A0628/303 /07-12-17/15 H 46/N=7507/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 11/AGEO=QADIRPUR-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=2
!! *A0628/320 /08-01-26/06 H 39/N=0566/TYP=COM/CAT=ID/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 14/AGEO=KHOKARAN-B01
/TEXAL=FAULTY UNIT/COMPL.INF:

how wil i extract these paragraphs of alarms in respective alarm file??
i need to extract this log on the basis of prefix ! , !! , !!!

i am using the code to take input from file into script as

Code:

Code:
#!/bin/bash
alarm= `cat ~/alkatel.txt`
echo $alarm | while read line
do 
{
echo $line
}done

i want to have this code in bash only not in perl and any othe language please do me this favor

jschiwal · 02-08-2008, 08:29 PM

Your script simply prints out the file which is what cat does.

You might try sed or awk.

The formatting of the log makes it difficult. If a line where to separate log entries, it would be easier. Here you can insert a line before a log entry and then filter out the critical parts. Instead of a proper sed program that does it all in one, I'll get lazy and cheat. The first sed command simply inserts the lines, while the second filters the entries:

Code:

 sed  '/^!/i\
' log | sed -n '/^!!! /,/^$/p'

The second sed command can write to three respective logs, minor.log, major.log & critical.log.
I found that I couldn't use the form "sed 'sedcommand1;sedcommand2;sedcommand3'" so I needed to use the "-e" argument to separate the three sed command lines.

Code:

 sed  '/^!/i\
' log | sed -n -e '/^!!! /,/^$/w critical.log' -e '/^!! /,/^$/w major.log' -e'/^! /,/^$/w minor.log'

Code:

jschiwal@lmax:~> cat minor.log 
! *A0628/540 /07-12-17/15 H 58/N=7598/TYP=ICT/CAT=SI/EVENT=MAL
/NCEN=MULCT /AM =SMTA1/AGEO=S1-TR01-B03-A085-R000
/TEXAL=AIS/COMPL.INF:
/AF=URMA1
/ ICTRQ AGCA=S1-TR01-B03-A112-R065
/AMET=01-07-02
/AFLR=222-09/CRC=NACT

jschiwal@lmax:~> cat major.log 
!! *A0628/320 /08-01-17/13 H 47/N=8062/TYP=COM/CAT=ID/EVENT=MAL
/NCEN=MULCT /AFUR =URAL- 15/AGEO=S1-TR01-B04
/TEXAL=FAULTY UNIT/COMPL.INF:
* #F0612/T11F14/NCEN=MULCT /08-01-30/11 H 33/NAM=ODCAB /TDA=0001
/N=7064/NIV=2/ENS=006/SENS=013-000-000/P='0228'H/CN=02/PAR='03'H
/EM: AFUR =URAL- 15

!! *A0628/320 /08-01-26/06 H 39/N=0566/TYP=COM/CAT=ID/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 14/AGEO=KHOKARAN-B01
/TEXAL=FAULTY UNIT/COMPL.INF:

jschiwal@lmax:~> cat critical.log 
!!! *A0628/303 /07-12-17/15 H 46/N=7501/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 4/AGEO=BUDLASAN-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=3

!!! *A0628/306 /08-01-23/07 H 20/N=9619/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFCN =CNLM3-4/AGEO=MKRASHID-B02-A003
/TEXAL=CN ISOLATED

!!! *A0628/303 /07-12-17/15 H 46/N=7503/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 12/AGEO=KHOTYWAL-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=0

!!! *A0628/089 /08-01-29/22 H 14/N=1760/TYP=ENP/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 11/AGEO=QADIRPUR-B01
/TEXAL=GENERATING UNIT

!!! *A0628/303 /07-12-17/15 H 46/N=7506/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 7/AGEO=ADABOSAN-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=2

!!! *A0628/303 /07-12-17/15 H 46/N=7507/TYP=COM/CAT=IM/EVENT=MAL
/NCEN=MULCT /AFUR =URAD- 11/AGEO=QADIRPUR-B01
/TEXAL=SINGLE JUNCT/UT OOS/COMPL.INF:
/CN=2

I haven't thouroughly tested this. You might need to add a blank line to the end of the log or test it with a sample where the last entry doesn't end with a blank line. That could cause the ",/^$/" end of range test to fail.

nabmufti · 02-09-2008, 04:21 AM

thanx a lot
the code follwing gave me a way to extract the paragraphs follwing prefix ! , !! , !!! from log file

Code:

 sed  '/^!/i\
' log | sed -n -e '/^!!! /,/^$/w critical.log' -e '/^!! /,/^$/w major.log' -e'/^! /,/^$/w minor.log'

i need your help again

please guide me that how can i trim the required character string in paragraph from log file?????????????

to make you more clear
see
! *A0628/540 /07-12-17/15 H 58/N=7598/TYP=ICT/CAT=SI/EVENT=MAL
/NCEN=MULCT /AM =SMTA1/AGEO=S1-TR01-B03-A085-R000
/TEXAL=AIS/COMPL.INF:
/AF=URMA1
/ ICTRQ AGCA=S1-TR01-B03-A112-R065
/AMET=01-07-02
/AFLR=222-09/CRC=NACT

from the above paragraph i want to extract

date & time i.e. /07-12-17/15 H 58/
NCEN=MULCT
AGEO=S1-TR01-B03-A085-R000
EVENT=MAL
TEXAL=AIS/COMPL.INF:

please guide me that how to deal with these substring extraction from a paragraph in a file

jschiwal · 02-09-2008, 07:05 AM

This can give you an idea on extracting the date on the first line:

Code:

 sed '/^!/s/.*\/\([[:digit:]][[:digit:]]-[[:digit:]][[:digit:]]-[[:digit:]][[:digit:]]\)\/.*/\1/' minor.log

The general pattern is like this:
The first part "/^!/" selects the first line with the date. On the other lines, you may need more patterns to work with.
Are the positions fixed? Is NCEN= a constant or one of numerous possibilities?

Code:

sed '/NCEN=/s/.*\(NCEN=[^ ]*\) .*/\1/'

So far we have:

Code:

 sed -e '/^!/s/.*\/\([[:digit:]][[:digit:]]-[[:digit:]][[:digit:]]-[[:digit:]][[:digit:]]\)\/.*/\1/' -e '/NCEN=/s/.*\(NCEN=[^ ]*\) .*/\1/' minor.log 
07-12-17
NCEN=MULCT
/TEXAL=AIS/COMPL.INF:
/AF=URMA1
/ ICTRQ AGCA=S1-TR01-B03-A112-R065
/AMET=01-07-02
/AFLR=222-09/CRC=NACT

I've been writing it a part at a time and testing it as I go on. It might make sense to create a file with a sed program and call sed this way:
sed -f extract.sed minor.log

And I just noticed looking on the next lines in your sample output that some of the data you want later is contained on the first line I thought I was done with. That means that it is necessary to either use the more advanced commands or sed or use another tool like awk.

Awk might be a better choice. You can extract the information you want from these lines and put them into variables. Then when it reaches a blank line you can print out the variables in the order that you want.

A test awk program like this might help you identify the fields you want to keep:

Code:

awk 'BEGIN { FS="/" }
/^! / { print "1:"$1 " 2:"$2 "3:"$3 " 4:"$4 " 5:"$5 " 6:" $6}' minor.log
1:! *A0628 2:540 3:07-12-17 4:15 H 58 5:N=7598 6:TYP=ICT

The FS is the builtin "Field Separator" variable that awk uses to separate a line into fields. If the values you want are always located in the same fields on each line, you might consider changing the RS (Record Separator) to an empty line. Then you could simply have a line like:

Code:

 awk 'BEGIN {FS="/"; RS=""} { printf "%s/%s/\n%s\n%s\n%s\n%s/%s\n\n", $3, $4, $10, $12, $8, $14, $15}' minor.log 
07-12-17/15 H 58/
NCEN=MULCT 
AGEO=S1-TR01-B03-A085-R000
EVENT=MAL
TEXAL=AIS/COMPL.INF:

Whether this will work depends on how regular the sample pattern you gave is.

nabmufti · 02-10-2008, 09:23 AM

i got the exact solution i was looking for
u r genius