perl on line to take out select values

casperdaghost · 01-22-2010, 12:43 AM

This is a Fix message - it is a type of protocol for transmitting financial data
each number followed by a = sign is a tag - each tag means something.
some tage are moe important than other.

8=FIX.4.1^9=0235^35=D^34=10^43=N^49=VENDOR^50=CUSTOME^ 56=BROKER^52=19980930- 09:25:58^1=XQCCFUND^11=10^21=1^55=EK^48=277461109^22= 1^54=1^38=10000^40=2^44=76.750000^59=0^10=165 18
8=FIX.4.1^9=0235^35=F^34=10^43=N^49=VENDOR^ 50=CUSTOMER^56=BROKER^52=19980930-09:25:58^ 1=XQCCFUND^11=10^21=1^55=EK^48=277461109^ 22=1^54=1^38=10000^40=2^44=76.750000^10=165 24

I have a perl on liner that takes out some of the tags so i can better see them.

cat sample | perl -nle "print /(39=[\w]/'
35=D
35=F

However, I need something that will take out multipe tags all at once ( instaed of hitting each line with the one liner and then cutting and pasting the values to an excel speadsheet).

I need something like this - something that takes out the same tags from each line:
35=D 43=N 56=Broker 54=1
35=F 43=N 56=Broker 54=1

kbp · 01-22-2010, 07:15 AM

This isn't perl - if your data stays in the format you posted this should do the job though:

Code:

cat sample.txt | cut -d^ --output-delimiter=" " -f3,5,8,16 | sed 's/[[:space:]]\{1,\}/ /g'

Output:
35=D 43=N 56=BROKER 54=1
35=F 43=N 56=BROKER 54=1

cheers

casperdaghost · 01-24-2010, 12:40 PM

the first three tags of fix messages are always in the same order - but the rest arr not. Sed and awk will not work, because it goes by column, I need something that will go after the characters itself.
perl -nle 'print if /regex/' works great - for one regex at a time.
I guess I will try making a while loop - that reads in the whole line and splits out the regex one by one.

bartonski · 01-24-2010, 01:37 PM

You need two things to make this work:

An array to hold all of your matching back references
A more sophisticated regular expression which will capture all of the fields that you want to capture.

In this case, I'm using the regular expression alternation operator '|' to specify which fields to pull, and then I'm loading all of the matching backreferences into the array '@matches'

Code:

#! /usr/bin/perl -w
while(<>){
        if ( @matches = $_ =~ /((35|43|54|56)=[^^]+)/g){
                print "@matches\n";
        }
}

and ...

Code:

$ perl  /tmp/match.pl /tmp/sample
35=D 35 43=N 43 56=BROKER 56 54=1 54
35=F 35 43=N 43 56=BROKER 56 54=1 54

... it works!

I wrote this as a full perl script, I'll let you golf it down to a one liner.

casperdaghost · 01-24-2010, 11:06 PM

yeah that is a nice script - look at what i came up with (don't laugh)

while read line
do
echo $line | perl -nle 'print /(35=[A-Z0-9]{1,2})/'
echo $line | perl -nle 'print /(39=[A-Z0-9]{1,2})/'
sleep 3
printf "this line is over\n"
done < fixmessage3
35=8
39=E
this line is over
35=D
this line is over
35=F
this line is over

i was going to post this and ask the forum
for a way to print the values on one line - like this:
35=8 39=E
this line is over
35=D
this line is over
35=F
this line is over

instead of a list.

Anyhow - this perl script looks really good.

just for reference this is fixmessage3

8=FIX.4.2 9=67 35=8 49=PHLX 56=PERS 11=ATOMNOCCC9990900 52=20071123-05:30:00.000
20=3 150=E 39=E 55=MSFT 167=CS 54=1 38=15 40=2 44=15 58=PHLX EQUITY TESTING 59=
0 47=C 32=0 31=0 151=15 14=0 6=0 10=102
8=FIX.4.1 9=0235 35=D 34=10 43=N 49=VENDOR 50=CUSTOME 56=BROKER 52=19980930- 09
:25:58 1=XQCCFUND 11=10 21=1 55=EK 48=277461109 22= 1 54=1 38=10000 40=2 44=76.7
50000 59=0 10=165
8=FIX.4.1 9=0235 35=F 34=10 43=N 49=VENDOR 50=CUSTOMER 56=BROKER 52=19980930-09
:25:58 1=XQCCFUND 11=10 21=1 55=EK 48=277461109 22=1 54=1 38=10000 40=2 44=76.
750000 10=165

chrism01 · 01-24-2010, 11:37 PM

Well, if you ever need to check all (or a lot) of fields, I'd use the split() fn

Code:

$var1 = "fixmsg from post #1";
@arr = split(/\^/, $var1);

# Use this for space separated fixmsg
@arr = split(/ /, $var1);

casperdaghost · 01-26-2010, 10:33 PM

how would i integrate this split function into the perl program

$var1 = "fixmsg from post #1";
@arr = split(/\^/, $var1);

# Use this for space separated fixmsg
@arr = split(/ /, $var1);

kbp · 01-27-2010, 02:52 AM

The split line will break $var1 up into pieces divided on the '^' character, from there you can process the array something like:

foreach (@arr) {

if ( /^35=.*/ ) { print; }
if ( /^53=.*/ ) { print; }
...

}

cheers

bartonski · 01-27-2010, 08:02 AM

better to use alternation:

Code:

foreach(@arr){
    print if(/((35|43|54|56)=/);
}
print "\n";

casperdaghost · 02-01-2010, 12:37 AM

this look svery good thank you