[SOLVED] sed: s command backward references with --regexp-extended

catkin · 05-03-2012, 09:23 AM

I have not been able to get the expected result using --regexp-extended in combination with the s command's \<n> backward references.

The intention is to extract the timestamp components from log files for a specific message (to paste into a spreadsheet for analysis).

This works for a single month

Code:

grep 'incorrect password length' /var/log/* | sed 's/[^:]*:Apr \([0-9]*\) \([^ ]*\).*/\1apr\t\2\tX/'
[snip]
28apr	12:27:07	X

Now I want to do the same for two months. Here's what I tried and the resulting error

Code:

grep 'incorrect password length' /var/log/* | sed --regexp-extended 's/[^:]*:\((Apr|May)\) \([0-9]*\) \([^ ]*\).*/\1\2\t\3\tX/'
sed: -e expression #1, char 57: invalid reference \3 on `s' command's RHS

I tried changing \(( to (\( and similarly for )\) but got the same error.

druuna · 05-03-2012, 09:47 AM

Hi,

Although you don't provide an input example, the following test seems to work:

Code:

sed -r 's/.*((Apr|May)).*/\1X/' infile

Try implementing this for your situation.

Hope this helps.

grail · 05-03-2012, 10:36 AM

I do not have high skills in sed but maybe the data is better suited to awk if you are pulling delimited information out?
If you show some data I would be happy to try an awk solution

catkin · 05-03-2012, 10:55 AM

Thanks both

I didn't provide an input example!

Here's one output line from grep 'incorrect password length' /var/log/*.log

Code:

/var/log/daemon.log:May  1 20:42:53 LS1 smbd[17937]:   smb_pwd_check_ntlmv1: incorrect password length (64)

I tried removing the "\"s as drunna suggested and it solved the error but unexpectedly output the first match twice and not the third match:

Code:

root@LS1:~# mon1=Apr; mon2=May
root@LS1:~# egrep "^($mon1|$mon2) .*incorrect password length" /var/log/*.log \
    | sed --regexp-extended "s/[^:]*:(($mon1|$mon2)) ([0-9]*) ([^ ]*).*/\1\2\t\3\tX/" \
    | sort | uniq
AprApr	29	X
AprApr	30	X
MayMay		X

This worked (but I don't know why; the highlit asterisk is because the 1 in May 1 is left-padded with a space):

Code:

root@LS1:~# egrep "^($mon1|$mon2) .*incorrect password length" /var/log/*.log \
    | sed --regexp-extended "s/[^:]*:(($mon1|$mon2)) *([0-9]*) ([^ ]*).*/\1\3\t\4\tX/" \
    | sort | uniq

firstfire · 05-03-2012, 11:16 AM

Hi.

You don't need double parentheses, this should also work

Code:

sed -r "s/[^:]*:(Apr|May) *([0-9]*) ([^ ]*).*/\1\2\t\3\tX/"

druuna · 05-03-2012, 12:02 PM

Hi,

Would this be a "better" solution:

Code:

awk -v mon1="${mon1}" -v mon2="${mon2}" 'BEGIN{ FS="[ :]*"} /incorrect password length/ { if ( $2 == mon1 || $2 == mon2 ) { print $2, $3 , $4 ":" $5 ":" $6, $7 }}' infile

I'm not entirely clear on what the full intended output should be, the above outputs this:

Code:

$ cat infile
/var/log/daemon.log:Mar  1 20:42:53 LS1 smbd[17937]:   smb_pwd_check_ntlmv1: incorrect password length (64)
/var/log/daemon.log:Apr 11 20:42:53 LS1 smbd[17937]:   smb_pwd_check_ntlmv1: incorrect password length (64)
/var/log/daemon.log:May  1 20:42:53 LS1 smbd[17937]:   smb_pwd_check_ntlmv1: incorrect password length (64)
/var/log/daemon.log:Jun  1 20:42:53 LS1 smbd[17937]:   smb_pwd_check_ntlmv1: incorrect password length (64)
$ mon1=Apr; mon2=May
$ awk -v mon1="${mon1}" -v mon2="${mon2}" 'BEGIN{ FS="[ :]*"} /incorrect password length/ { if ( $2 == mon1 || $2 == mon2 ) { print $2, $3 , $4 ":" $5 ":" $6, $7 }}' infile 
Apr 11 20:42:53 LS1
May 1 20:42:53 LS1

grail · 05-03-2012, 12:41 PM

If I am reading this correctly, you do not need to change FS for awk as the demo data is the output of grep which has prepended the name and the colon:

Code:

awk -v months="(Apr|May)" '/incorrect password length/ && $0 ~ months{print $1,$2,$3,$4}' file

If you do wish to use bash variables for the months, may I suggest an array:

Code:

m=(Apr May)

IFS="|" && awk -v months="(${m[*]})" '/incorrect password length/ && $0 ~ months{print $1,$2,$3,$4}' file

David the H. · 05-03-2012, 12:51 PM

You don't need grep either. sed can filter the lines out of the files directly. (So can awk, as grail demonstrates.)

Assuming the output format is just month/day/time, I like this version:

Code:

sed -nr "/incor.*pass.*length/ s/.*(($mon1|$mon2)[ ]+[0-9]+)[ ]+([0-9:]+).*/\1\t\3\tX/p" /var/log/*.log

#Produces the following output:
May  1  20:42:53        X

If the day needs to be in a separate tab-delimited field from the month, just move the parens around and add back the "\2" to the output.

Code:

sed -nr "/incor.*pass.*length/ s/.*($mon1|$mon2)[ ]+([0-9]+)[ ]+([0-9:]+).*/\1\t\2\t\3\tX/p" /var/log/*.log

May     1       20:42:53        X

druuna · 05-03-2012, 12:51 PM

Hi,

Quote:

Originally Posted by grail

If I am reading this correctly, you do not need to change FS for awk as the demo data is the output of grep which has prepended the name and the colon:

LOL

Of course you are correct.

I don't belief that I overlooked that

BTW: Nice awk solutions!

catkin · 05-03-2012, 09:24 PM

Quote:

Originally Posted by firstfire

Hi.

You don't need double parentheses, this should also work

Code:

sed -r "s/[^:]*:(Apr|May) *([0-9]*) ([^ ]*).*/\1\2\t\3\tX/"

Thanks firstfire

It does work.

My understanding of regexes is defective. According to my understanding Apr|May should match Ap followed by r or M followed by ay but experiments show that is not the case.

In terms of the regex 7 man page the | separates two branches. A branch is one or more pieces, concatenated. A piece is an atom possibly followed by ... . An atom is ... or a single character with no other significance (matching that character).

If backreferencing is not being used on the month names so the regex is "s/[^:]*:Apr|May *([0-9]*) ...", how is the branch after the | terminated? Why does it stop after May?

catkin · 05-03-2012, 09:30 PM

Quote:

Originally Posted by druuna

I'm not entirely clear on what the full intended output should be ...

The intention is to generate a sting in which the first tab-separated field is recognised as a date when pasted into a spreadsheet. I was aiming for the same as your output but with no space between month name and day number but they are effectively the same. Neither work as intended! The day number must come before the month name.

catkin · 05-03-2012, 09:40 PM

Quote:

Originally Posted by grail

If I am reading this correctly, you do not need to change FS for awk as the demo data is the output of grep which has prepended the name and the colon:

Code:

awk -v months="(Apr|May)" '/incorrect password length/ && $0 ~ months{print $1,$2,$3,$4}' file

If you do wish to use bash variables for the months, may I suggest an array:

Code:

m=(Apr May)

IFS="|" && awk -v months="(${m[*]})" '/incorrect password length/ && $0 ~ months{print $1,$2,$3,$4}' file

That's right.

Using an array like that does make it neater.

What is the purpose of IFS="|" and of && ?

ntubski · 05-03-2012, 11:18 PM

Quote:

Originally Posted by catkin

My understanding of regexes is defective. According to my understanding Apr|May should match Ap followed by r or M followed by ay but experiments show that is not the case.

"|" has the lowest precedence.

Quote:

If backreferencing is not being used on the month names so the regex is "s/[^:]*:Apr|May *([0-9]*) ...", how is the branch after the | terminated? Why does it stop after May?

If you remove the parens then the branch is not terminated, ie your 2 branches would be [^:]*:Apr and May *([0-9]*) ...

grail · 05-03-2012, 11:57 PM

Quote:

What is the purpose of IFS="|" and of && ?

When using * for a quoted array it will use the first delimiter in IFS to separate the elements, so by setting IFS to a pipe it gives us the desired output.

&& is to ensure the previous task worked prior to using it.

catkin · 05-04-2012, 12:40 AM

Quote:

Originally Posted by grail

When using * for a quoted array it will use the first delimiter in IFS to separate the elements, so by setting IFS to a pipe it gives us the desired output.

&& is to ensure the previous task worked prior to using it.

Neat.

Hard to imagine the circumstances in which IFS='|' would not work but ...

EDIT:

Code:

c@CW8:/tmp$ IFS='|' && echo true || echo false
bash: IFS: readonly variable
[neither true nor false echoed]