LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 09-18-2013, 01:56 AM   #106
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Thumbs up


Hello Pan64,

Thank you so much. I understand better how it works your code with the explanation. I'll need to see more examples of perl and its syntax
to be able to make more things but I've learned a lot (besides the fact that the help got from all of you).

I've tested the line "my $ha = (length($a)>8) ? Math::BigInt->new("0x$a") : hex($a);" and I've had to add twice this line
inmediately before de line "@result" because if I don't add this line twice I get compilation error in the "else" part of the "while" loop. But it seems is working. I'll test with a real file, the output should be the same, and the time processing a litle bit less I think.

Thanks again.

Best regards
 
Old 09-18-2013, 10:57 AM   #107
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,689

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Well I went back to hash for the Ruby solution (just as it seemed easier and straight forward):
Code:
#!/usr/bin/env ruby

BEGIN{  if ARGV[1].nil?
            $/="\xff\x77"
        else
            $/="\xff#{ARGV[1].hex.chr}"
        end
}

fruit = {   0 => "",
            1 => "",
            "a".hex => "",
            2 => "",
            3 => "",
            6 => "", 
            7 => ""
}

IO.foreach(ARGV[0]){ |l| 

        line = l.unpack('H*')[0]
        next unless line =~ /^(.{6,18})(532064[^f]*).(814[^f]*)/
        printf("%d|%s|%s",$1.hex,$2,$3)

        if line =~ /05(9.{32,34}.*?)940e(.{28})/
            rest = $2

            pos = fruit.clone

            $1.scan(/9([\dab]).{4}(.{2})(.{8})([^f]*)f*(0[01])(0[01])?/).each{
                |y|

                arr = y[1..-1].compact.map{
                    |z|
                    unless z =~ /^814/
                        z.hex
                    else
                        z
                    end
                }
                pos[y[0].hex] = arr.join(",")
            }
            puts "|" + [pos.values.compact,rest.scan(/../).map{|z| z.hex}].join("|")

            pos.clear
        else
            puts
        end
} if File.exists?(ARGV[0])
This one also covers the file being used not existing and allows you to use a different separator (just add as second argument)
 
Old 09-19-2013, 05:21 AM   #108
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello grail,

Thank your for the help one more time. You deserve this too certainly . From your last code I was able to modificate it and handle
2 characters in fruit array. Sure there is a better way to do it, but I got it using fruit array and
Pos hash. Main changes in red.

Was I was wondering is how to empty all the elements of fruit array without to create it again with
Array.new(7,""), because fruit.clear removes all elements, I wanted only to change to "" each element, but
anyway, it works in the way I did.

Code:
#!/usr/bin/env ruby -E BINARY
# -*- encoding: utf-8 -*-

BEGIN{  if ARGV[1].nil?
            $/="\xff\x77".force_encoding("BINARY") 
        else
            $/="\xff#{ARGV[1].hex.chr}".force_encoding("BINARY") 
        end
}

fruit = Array.new(7,"")

pos = {   	"90" => 0,
		"91" => 1,
		"9a" => 2,
		"92" => 3,
		"93" => 4,
		"96" => 5, 
		"97" => 6
}

IO.foreach(ARGV[0]){ |l| 

        line = l.unpack('H*')[0]
        next unless line =~ /^(.{6,18})(532064[^f]*).(814[^f]*)/
        printf("%d|%s|%s",$1.hex,$2,$3)

        if line =~ /05(9.{32,34}.*?)940e(.{28})/
            rest = $2
            $1.scan(/(9[\dab]).{4}(.{2})(.{8})([^f]*)f*(0[01])(0[01])?/).each{
                |y|
                arr = y[1..-1].compact.map{
                    |z|
                    unless z =~ /^814/
                        z.hex
                    else
                        z
                    end
                }
		fruit[pos[y[0]]] = arr.join(",")
            }
		puts "|" + [fruit.compact,rest.scan(/../).map{|z| z.hex}].join("|")
		fruit = Array.new(7,"")		
        else
            puts
        end
} if File.exists?(ARGV[0])
Thanks for all the help to all again, very appreaciated!

Best regards
 
Old 09-19-2013, 12:41 PM   #109
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,689

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
I have not used it, but you could try the fill command:
Code:
fruit.fill("")
I would be interested to know which runs quicker as now you have both an array and a hash created whereas my current example uses a single hash.
 
1 members found this post helpful.
Old 09-20-2013, 02:10 AM   #110
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by grail View Post
I have not used it, but you could try the fill command:
Code:
fruit.fill("")
I would be interested to know which runs quicker as now you have both an array and a hash created whereas my current example uses a single hash.
Hello grail,

Thank you one more time, it works the part array.fill("").

I've tested both scripts and the times are almost the same only 5 minutes takes now, and 7 seconds of difference between both. Please see below.
Code:
-------------------------------------------------
Execution Time 324.0204536 sec (Array and Hash script)
-------------------------------------------------

-------------------------------------------------
Execution Time 331.5304642 sec (Single Hash script)
-------------------------------------------------
Best regards

Last edited by Perseus; 09-20-2013 at 02:11 AM.
 
Old 11-08-2013, 01:06 AM   #111
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello grail/pan64,

An old thread, only I found some issue and maybe you can help me to fix it.

To remember, the file contains blocks separated by FF77. Well, the issue I found is that when the next 3 bytes after a block delimiter
have the same sequence FF77, then the block is being skipped.

Examples of this below, the real block delimiters are FF77 in red. But sometimes the first 3 bytes of any block could contain for example
00FF77532064..... And this FF77 in blue is being confused with real delimiter.

How to fix this? maybe a condition to avoid take as delimiter FF77 if it appears inmediate after another FF77?

Another reference is that always appear FF77 + 3 bytes + 532064.

I hope it is not to difficult to handle this issue and you can help me.

Code:
013300013600013700016600016500017700016900017900009300012200002100010900010a00012600010800012b00002c00002d00002e0000550000560007
2a00002f0000300000930000ff3400800932c90600000000a000800935c90600000000000080093cc90600000000800005910f01020000000d8147451807ffff
ff009310010c0000000d8147451805ffffff0101960f010c0000000d81474518559fffff00940e01020102010001ffffff020102019506000000007000ff7700
ff77532064014041612f81422060002fffff0015000a4800015a0007420001330001360001370001660001650001770001690001790000930001220000210001
0900010a00012600010800012b00002c00002d00002e00005500005600072a00002f0000300000930000ff3400800932c90600000000a000800935c906000000
00000080093cc90600000000800005910f01020000000d8147451825ffffff009310010c0000000d8147451805ffffff0101960f010c0000000d81474518559f
ffff00940e01020102010001ffffff020102019506000000000000ff770ff770532064022280546f81422060003fffff0015000a4800015a0007420001330001
3600013700016600016500017700016900017900009300012200002100010900010a00012600010800012b00002c00002d00002e00005500005600072a00002f
0000300000930000ff3400800932c90600000000a000800935c90600000000000080093cc90600000000800005910f01020000000d8147451905ffffff009310
010c0000000d8147451805ffffff0101960f010c0000000d81474518559fffff00940e01020102010001ffffff020102019506000000000000ff77ff77005320
64022939276f81422060004fffff0015000a4800015a00074200013300013600013700016600016500017700016900017900009300012200002100010900010a
00012600010800012b00002c00002d00002e00005500005600072a00002f0000300000930000ff3400800932c90600000000a000800935c90600000000000080
093cc90600000000800005910f01020000000d8147451844ffffff009310010c0000000d8147451805ffffff0101960f010c0000000d81474518559fffff0094
0e01020102010001ffffff020102019506000000000000ff770ff774532064013741169f81422060354fffff0015000a4800015a000260000133000136000137
00017e00016900006a00007900009300012200002100010900010a00012600010200010400010500010600011000010800012b00002c00002d00002e00005500
005600072a00002f0000300000930000ff3400800932c90688888000a000800935c906000080000000800943c9068888800080000582002e0501000001006500
00000200000200180000000300000300170000000400000400010000000a00ffff0065000000ff77000006532064013741255f81422079900fffff0015000a48
00015a00026000013300013600013700017e00016900006a00007900009300012200002100010900010a00012600010200010400010500010600011000010800
012b00002c00002d00002e00005500005600072a00002f0000300000930000ff3400800932c90688888000a000800935c906000080000000800943c906888880
Thanks in advance.
 
Old 11-08-2013, 02:03 AM   #112
pan64
Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 5,142

Rep: Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364
What I can imagine at this moment: if you can say for example "the length of a record is at least 15 (or ??) bytes" is an acceptable restriction. In that case we can concatenate lines if they were too short.

Code:
# instead of this:
	while ( defined ( my $l = <F> ) ) {
		process ($l);
	}
# insert this at the beginning of the script
my $MINIMAL_RECORD_LENGTH = 15;
# and modify the while cycle:
	while ( defined ( my $l = <F> ) ) {
                while (length $l < $MINIMAL_RECORD_LENGTH) {
                    $l .= $sep;
                    $l .= <F>;
                }
		process ($l);
	}
(not tested, but I hope it helps)
Of course, you can specify another condition, for example if 532064 comes too early, but actually we cannot avoid delimiters in some cases, that is against the logic of the script (= means should be completely rewritten)
 
Old 11-08-2013, 08:31 AM   #113
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello pan64,

Thank you for your answer. I'll test asap.

One last question, I was able to do some addition to ruby code, but I was not able to do the same in perl code. I want to store "rest" variable before printing the output to compare with a string.

I mean, string to compare would be for example compare=||2,13,456|43(2\|\|8)|2|1| and if rest = compare then
Rest=rest+"|A" and if rest is different than compare, then rest=rest + "|B”

I hope make sense
 
Old 11-08-2013, 09:16 AM   #114
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,689

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
I am with pan in that you will need to know what your criteria is for ignoring a split before you can write code to compensate, ie just how far apart will be considered too far or not far enough??

Not sure I follow the new question? You can assign 'rest' to another variable and compare to anything you like so I am not following the exact issue for this part?
 
Old 11-08-2013, 10:21 AM   #115
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello grail,

The less size a block could have is 3+8+8 bytes = 19 bytes. Maybe with that condition could be fix the issue, with 15 as mentioned by pan64 could work too.

Regarding my other question, I was able to modify the part "puts "|" + [fruit.compact,rest.scan(/../).map{|z| z.hex}].join("|")" in your code and assign that to "rest", then, before print "rest", I compare "rest" with string "compare". Compare variable has a regex inside (in red).

So, for ruby I was able to do it, but for perl I haven't had success to replicate the push part "push @result, join '|', map ( hex, unpack("(A2)*", $rest) );" similar to what I did in ruby.

The lines below in blue are those I added in ruby code and works.
Code:
IO.foreach(ARGV[0]){ |l| 
.
.
.
		#puts "|" + [fruit.compact,rest.scan(/../).map{|z| z.hex}].join("|")
		rest = "|" + [fruit.compact,rest.scan(/../).map{|z| z.hex}].join("|")
		
		compare = /\|2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,814752695(59|63|65|94),0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|1/
		rest = ( rest =~ compare ? rest + "|A" : rest + "|B" )
		
		puts rest	
		fruit.fill("")
        else
            puts
        end
}
Thanks in advance for the help.

Last edited by Perseus; 11-08-2013 at 10:25 AM.
 
Old 11-08-2013, 11:12 AM   #116
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,689

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Code:
rest = ( rest =~ compare ? rest + "|A" : rest + "|B" )

rest = rest + "|" + (rest =~ compare ? "A" : "B" )
As for the first part, simply place a length test similar to pan64's and you should be good to go
 
Old 11-08-2013, 11:40 AM   #117
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello grail,

In perl script I understand that the following line prints "result" variable concatenated with "rest" variable and putting in the same operation the delimiter "|".
Code:
push @result, join '|', map ( hex, unpack("(A2)*", $rest) );
The issue is how to replace the part "join '|', map ( hex, unpack("(A2)*", $rest) )" and store it in "rest". Once I have the content stored in "rest", I would be able to compare with string "compare".

Regarding to the other question, would be something like this?
Code:
MINIMAL_RECORD_LENGTH = 15
IO.foreach(ARGV[0]){ |l| 

	line = l.unpack('H*')[0]
		if line.length < MINIMAL_RECORD_LENGTH do
			line = "FF77" + line
		end
	##
	Original code
	## 
} if File.exists?(ARGV[0])
Thanks in advance

Last edited by Perseus; 11-08-2013 at 11:42 AM.
 
Old 11-09-2013, 03:08 AM   #118
pan64
Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian i686 (solaris)
Posts: 5,142

Rep: Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364Reputation: 1364
push, join and map will not print anything, that line only collects the result in the array @result.

$rest = join '|', map ( hex, unpack("(A2)*", $rest);
should work,
push @result, $rest; # or any other variable
also (do not forget to declare: my $rest)
but actually I do not really understand what do you need.
 
Old 11-09-2013, 04:57 AM   #119
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello pan64,

Actually I want to compare elements from 3 to N of array @result. I think I almost near since I found this way.
Code:
$z= join ("|", @result[ 3 .. $#result);
That command it seems to work to store the string I need in $z. I only want now to compare that string in $z with 3 strings that only are different in number in red, like below:
Code:
compare1=2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,81475269559,0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|1
compare2=2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,81475269577,0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|1
compare3=2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,81475269586,0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|1
So, Instead compare 3 times, I want to compare using a regex something like this:
Code:
my $compare = /2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,814752695(59|77|86),0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|1/;
To finally add "A" or "B" to the end of $z depending if matches or not with $compare like this:
Code:
$z= ( $z=~ $compare ? $z . "|A" : $z . "|B" );
But I receive this error:
Code:
Use of uninitialized value $_ in pattern match (m//)
So, I don't know how to define correctly to compare $z with those 3 strings using a regex way similar to above.

Thanks in advance.

Last edited by Perseus; 11-09-2013 at 04:59 AM.
 
Old 11-09-2013, 02:52 PM   #120
Perseus
Member
 
Registered: Oct 2011
Posts: 168

Original Poster
Rep: Reputation: Disabled
Hello pan64,

I finally was able to make the comparison in perl in the long way (if.. else) since I was not able to make it short way, since
I don't know how to assign the pattern to a variable. I did like below.
Code:
$z= join ("|", @result[ 3 .. $#result);	

if($z=~/2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,814752695(59|65|86),0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|11/)
{
	push @result, "A";
	}
else {
	push @result, "B";
}
In ruby I was able to do it in short way like below:
Code:
z = "|" + [fruit.compact,rest.scan(/../).map{|z| z.hex}].join("|")	
compare=/2,13,8147526905,0\|\|\|12,13,8147526905,1,1\|\|12,13,814752695(59|65|86),0\|\|1\|2\|1\|2\|1\|0\|1\|255\|255\|255\|2\|1\|2\|1/
z = ( z=~ compare ? z . "|A" : z . "|B" )
Regarding the other question, the suggestion you said me of modify the "while loop" in post #112 it seems to work, the only issue
is that the number of th first column is not correct, should be the 3 first bytes after the delimiter FF77 converted to decimal, but
is being printed a big number. It seems is taking more characters when converting from hex to dec in first column.

Thanks in advance.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Appending matching strings to specific lines (sed/bash) suntzu Programming 18 09-08-2012 04:29 PM
[SOLVED] search for 2 different strings in 2 diffrent lines threezerous Linux - Newbie 8 07-30-2012 04:42 PM
truncate strings on many lines mufea Linux - Newbie 2 02-23-2012 07:29 AM
How to remove lines and parts of lines from python strings? golmschenk Programming 3 11-27-2009 12:29 AM
Extract lines containing some strings without affectting sequential order cgcamal Programming 7 11-07-2008 12:57 AM


All times are GMT -5. The time now is 10:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration