LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 09-06-2013, 03:39 AM   #46
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled

Hello grail,

The version of Ruby is:
Code:
$ ruby --version
ruby 1.9.3p448 (2013-06-27) [i386-cygwin]
I was trying to run the same script in IRB ruby 2.0.0 in Windows, but I don't know how to run the script in irb, I've
tried with "load script.rb inputfile" but I receive error, it seems is not the correct way to run the code and give the
inputfile argument.
Code:
irb(main):010:0> load 'script.rb' inputfile.txt
SyntaxError: (irb#1):10: syntax error, unexpected tINTEGER, expecting end-of-input
load 'script.rb' inputfile.txt
I haven't changed anything else. When I use the line in blue all works fine, but when I use the line in red
I receive the error metioned in previous post.
Code:
#!/usr/bin/env ruby
BEGIN{	$/="ff77"	}
File.open(ARGV[0])

while gets
	$_.gsub!(/\n/,"")	
	$_.split($/).each{
		|x|
		next unless x =~ /^(.{6,18})(532064[^f]*).(814[^f]*)/
		printf("%d %s %s",$1,$2,$3)
		if x =~ /05(9.{32,34}.*?)940e(.{28})/
			rest = $2
			$1.scan(/9.*?(?=9[1-9]|$)/).each{
			#$1.scan(/9(\d)([1-9][\da-f]|0[e-f]|0[1-9a]).{26,28}(0[0-1]){1,2}/).each{
				|y|
				fruit = { "1" => " APPLES", "3" => " GRAPES", "6" => " PEAR" }
				printf("%s",fruit[y[1]])				

				arr = y[4..-1].scan(/(.{2})(.{2})(.{2})(.{6})([^f]*)f*(.*)/)[0]		
				arr.each_index{
					|i|
					if i > 4 && arr[i].length > 2
						arr << arr[i][2..-1]
						arr[i] = arr[i][0..1]
					end
					arr[i] = arr[i].to_i(16) if i != 4
				}
				printf(" %s",arr * "|")
			}
			printf " PROD_F "
			puts rest.scan(/../).map{|z| z.to_i(16)}.join("|")
		else
			puts
		end
	}
end
The input file is the same as before.
Code:
93114444444c55535f529332939333303693303032353807ffffffffffffffff77000001532064022272619f81422060001fffff0015000a4800015a00074200
013300013600013700016600016500017700016900017900009300012200002100010900010a00012600010800012b00002c00002d00002e0000550000560007
2a00002f0000300000930000ff3400800932c90600000000a000800935c90600000000000080093cc90600000000800005910f01020000000d8147451807ffff
ff009310010c0000000d8147451805ffffff0101960f010c0000000d81474518559fffff00940e01020102010001ffffff020102019506000000007000ff7700
0002532064014041612f81422060002fffff0015000a4800015a0007420001330001360001370001660001650001770001690001790000930001220000210001
0900010a00012600010800012b00002c00002d00002e00005500005600072a00002f0000300000930000ff3400800932c90600000000a000800935c906000000
00000080093cc90600000000800005910f01020000000d8147451825ffffff009310010c0000000d8147451805ffffff0101960f010c0000000d81474518559f
ffff00940e01020102010001ffffff020102019506000000000000ff77000003532064022280546f81422060003fffff0015000a4800015a0007420001330001
3600013700016600016500017700016900017900009300012200002100010900010a00012600010800012b00002c00002d00002e00005500005600072a00002f
0000300000930000ff3400800932c90600000000a000800935c90600000000000080093cc90600000000800005910f01020000000d8147451905ffffff009310
010c0000000d8147451805ffffff0101960f010c0000000d81474518559fffff00940e01020102010001ffffff020102019506000000000000ff770000045320
64022939276f81422060004fffff0015000a4800015a00074200013300013600013700016600016500017700016900017900009300012200002100010900010a
00012600010800012b00002c00002d00002e00005500005600072a00002f0000300000930000ff3400800932c90600000000a000800935c90600000000000080
093cc90600000000800005910f01020000000d8147451844ffffff009310010c0000000d8147451805ffffff0101960f010c0000000d81474518559fffff0094
0e01020102010001ffffff020102019506000000000000ff77000005532064013741169f81422060354fffff0015000a4800015a000260000133000136000137
00017e00016900006a00007900009300012200002100010900010a00012600010200010400010500010600011000010800012b00002c00002d00002e00005500
005600072a00002f0000300000930000ff3400800932c90688888000a000800935c906000080000000800943c9068888800080000582002e0501000001006500
00000200000200180000000300000300170000000400000400010000000a00ffff0065000000ff77000006532064013741255f81422079900fffff0015000a48
00015a00026000013300013600013700017e00016900006a00007900009300012200002100010900010a00012600010200010400010500010600011000010800
012b00002c00002d00002e00005500005600072a00002f0000300000930000ff3400800932c90688888000a000800935c906000080000000800943c906888880
Thanks for the help.
 
Old 09-06-2013, 07:53 AM   #47
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,627

Rep: Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943
Ahhh ... ok i missed it at first but got it now
Code:
$1.scan(/9.*?(?=9[1-9]|$)/).each{
If you print the value of 'y' that scan stores in it you will see that it is a single string
Code:
$1.scan(/9(\d)([1-9][\da-f]|0[e-f]|0[1-9a]).{26,28}(0[0-1]){1,2}/).each{
This line however, as it has back references in it it will actually return an array, so on the first item returned is:
Code:
["1", "0f", "00"]
Now this is based on the 3 back references in your scan. Now if you combine that with the following line:
Code:
arr = y[4..-1].scan(/(.{2})(.{2})(.{2})(.{6})([^f]*)f*(.*)/)[0]
Here we are trying to reference from the fifth element of the array to the end of the array. As the array only holds 3 items it will now be looking at the empty portion of the array,
hence anything not set is considered 'nil'. We then can understand the error message:
Code:
undefined method `scan' for nil:NilClass
Of course the nil object has no method called scan so we have an error.

Hope that all makes sense
 
1 members found this post helpful.
Old 09-08-2013, 05:03 AM   #48
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled
Hello grail,

I'm newbie in ruby and I didn't understand why that error, but with your explanation I understood
and after that I was able to see the different outputs of $1 when I changed the regex.

So, after a few hours testing and thinking I think I was able to modify the script to get a working
one with the long regex and separate directly the elements that I want to print between pipes.

The script is as follow:
Code:
#!/usr/bin/env ruby
BEGIN{	$/="ff77"	}
File.open(ARGV[0])

while gets
	$_.gsub!(/\n/,"")	
	$_.split($/).each{
		|x|
		next unless x =~ /^(.{6,18})(532064[^f]*).(814[^f]*)/
		printf("%d %s %s ","0x"+$1,$2,$3)
		$stdout.flush
		
		if x =~ /ff34.*?05(9.{32,34}.*?)940e(.{28})/
			rest = $2	
			$1.scan(/9(\d)([0-9][\da-f]|0[e-f]|0[1-9a])(.{2})(.{2})(.{8})([^f]*)f*?(0[0-1])(0[0-1])?/).each{		
				|y|		
				fruit = { "1" => " APPLES", "3" => " GRAPES", "6" => " PEAR" }

				if y[7] == nil				
					printf("%s %d|%d|%s|%d",fruit[y[0]],"0x"+y[3],"0x"+y[4],y[5],"0x"+y[6])
					$stdout.flush
				else	
					printf("%s %d|%d|%s|%d|%d",fruit[y[0]],"0x"+y[3],"0x"+y[4],y[5],"0x"+y[6],"0x"+y[7])					
					$stdout.flush
				end						
			}
			printf " PROD_F"
			$stdout.flush
			puts rest.scan(/../).map{|z| z.to_i(16)}.join("|")
			$stdout.flush			
		else
			puts
			$stdout.flush
		end			
	}
end
The evolution of this script to process faster would be if it was able to read the binary directly and avoid to use
of the "xxd" command convertion. But this version will help me a lot in what I have to do.

Many thanks for all the help and patience in this question I posted to all.

Best regards

Last edited by Perseus; 09-08-2013 at 05:05 AM.
 
Old 09-08-2013, 11:49 AM   #49
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,627

Rep: Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943
Please remember to mark as SOLVED once you have a working solution
 
Old 09-08-2013, 12:22 PM   #50
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled
Hello grail,

Only I don't know how to pipe the xxd command with the script. Because the script expects an input argument to the right like below
Code:
ruby script.rb file
I've tried with the following but is not working

Code:
xxd -ps -c 64 binaryfile | ruby script.rb
And with

Code:
xxd -ps -c 64 binary | script.rb
Thanks fot the help

Last edited by Perseus; 09-08-2013 at 12:24 PM.
 
Old 09-08-2013, 01:46 PM   #51
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,627

Rep: Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943
Just like running a normal script (ie bash). make it executable and then:
Code:
xxd -ps -c 64 binary | /path/to/file/script.rb
Sorry, forgot to add, you will need to comment out the File.open line (as were not opening a file anymore)

Last edited by grail; 09-08-2013 at 01:49 PM.
 
1 members found this post helpful.
Old 09-09-2013, 03:51 PM   #52
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled
Hello grail,

Thank you for all the help all this time. It works for me in this way:
Code:
xxd -ps -c 64 binary | ruby script.rb
The only issue is no matter if the input to the ruby script is piped or not, it takes moren
than 2 hours to process a binary of 2GB

Maybe this is an intrinsic issue of Ruby's perfomance speed, since the code only has a few lines
or maybe the use of regex. I don't know.

Best regards.
 
Old 09-10-2013, 03:10 AM   #53
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,627

Rep: Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943Reputation: 2943
An interpreted language will always be slower than something like C / C++ (ie compiled language). A Perl solution may be faster (not sure as not as adept in Perl)??

You may find if you investigate the binary reading option it may be quicker.

Of course a 2GB file will always take some time. Out of curiosity, how long does it take if you redirect to a file first and then run Ruby over it?
If much faster this way I would think then your RAM and CPU may be the bottle neck
 
Old 09-10-2013, 03:26 AM   #54
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled
Hello grail,

Redirecting firt to a file and then run the script takes almost the same. More than 2 hours
only but a little faster than do it piped (a few minutes less).

Regarding read directly the binary in ruby or perl, I'll try to investigate, since I don't know how to
use these regex to read bytes from binary.

Thanks again for the help in this question.

BR.
 
Old 09-10-2013, 03:38 AM   #55
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,482

Rep: Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452
perl can read and evaluate binary (so we do not need xxd), but I would suggest you to use c or c++.
you need to find that delimiter (ff77), construct your lines and convert to hex and finally you can use regexp on those lines.
to speed it up you can try multithreaded app (new thread for every line for example)..

Last edited by pan64; 09-10-2013 at 03:39 AM.
 
Old 09-10-2013, 05:16 AM   #56
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled
Hello pan64,

Thank you for your suggestions, it seems in perl or C or C++ would be faster,
but is more complicated for me since I don't know C nor Perl, only
a little about awk and what I've learned of ruby with grail's
solution and because I knew some awk I could undertand some grail's ruby code.

However following your suggestions I'll try to investigate how to do for a simple
case in perl or C/C++ to be able to reach the output given by grail's ruby code.

Thanks again for all the time and help.

Regards
 
Old 09-10-2013, 06:55 AM   #57
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,482

Rep: Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452
the perl code will be quite similar to the ruby script, you only need to open and read the file using a different method. I hope you can prepare a relatively small binary sample and I will try to help you.
http://stackoverflow.com/questions/8...y-file-in-perl
 
Old 09-10-2013, 12:57 PM   #58
Perseus
Member
 
Registered: Oct 2011
Posts: 179

Original Poster
Rep: Reputation: Disabled
Hello pan64,

Thanks for the availability to help me.

For the attached small binary, redirectiong the output of xxd -ps -c 64 to binary_xxd.txt and then using the
final ruby script below in this way ruby script.rb binary_xxd.txt
Code:
#!/usr/bin/env ruby
BEGIN{	$/="ff77"	}
File.open(ARGV[0])

while gets
	$_.gsub!(/\n/,"")	
	$_.split($/).each{
		|x|
		next unless x =~ /^(.{6,18})(532064[^f]*).(814[^f]*)/
		printf("%d %s %s ","0x"+$1,$2,$3)
		$stdout.flush
		
		if x =~ /ff79.*?05(9.{32,34}.*?)940e(.{28})/
			rest = $2
			$1.scan(/(9[\da-b])([0-9][\da-f]|0[e-f]|0[1-9a])(.{2})(.{2})(.{8})([^f]*)f*?(0[0-1])(0[0-1])?/).each{						
				|y|		
				fruit = { "90" => " APPLE", "91" => " LIME", "92" => " ORANGE", \
				"93" => " GRAPE","96" => " PEAR","97" => " CHERRY"}	

				if y[7] == nil				
					printf("%s %d|%d|%s|%d",fruit[y[0]],"0x"+y[3],"0x"+y[4],y[5],"0x"+y[6])
					$stdout.flush
				else	
					printf("%s %d|%d|%s|%d|%d",fruit[y[0]],"0x"+y[3],"0x"+y[4],y[5],"0x"+y[6],"0x"+y[7])					
					$stdout.flush
				end						
			}
			printf " PROF_F "
			$stdout.flush
			puts rest.scan(/../).map{|z| z.to_i(16)}.join("|")
			$stdout.flush			
		else
			puts
			$stdout.flush
		end			
	}
end
I get the output:
Code:
1 53206445018934550 81474549232
2 53206445018934551 81474554768  APPLE 2|48|8147526905|0 LIME 2|314|81475269559|0 GRAPE 12|159|8147526905|1|1 PEAR 14|235|81475269596|0 CHERRY 1|456345|81475269563|0 PROF_F 0|1|0|0|1|0|1|0|255|255|0|0|1|1
3 53206445018934552 81474557521  LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269565|0 PROF_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
4 53206445018934558 81477380427  LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269594|0 PROF_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
5 53206445018934559 81474663128
Maybe the Perl script could be not too hard to adapt.

Many thanks for the help.

Regards
Attached Files
File Type: txt binary.txt (4.0 KB, 14 views)

Last edited by Perseus; 09-10-2013 at 01:00 PM.
 
Old 09-10-2013, 01:15 PM   #59
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,387

Rep: Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553Reputation: 1553
I wonder if you would see a performance improvement just by removing all those $stdout.flush calls.
 
Old 09-11-2013, 02:16 AM   #60
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 11,482

Rep: Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452Reputation: 3452
I do not think flush will affect performance on a large input file.

Here is the beginning, it will do the first step....
Code:
#!/usr/bin/perl -w

use strict;

my $filename = "binary.data";

sub process {
	my $l = shift;
	my $hex = unpack( 'H*', $l );
	my ($a, $b, $c) = ($hex =~ /^(.{6,18})(532064[^f]*).(814[^f]*)/);
	return if ! defined $c;
#	print "$i: ". length ($l) . ", $a, $b, $c, $hex\n";
	printf("%d %s %s\n",hex($a),$b,$c);
}

do {
	local $/;
	$/ = "\xff\x77";
	open F, $filename or die;
	while ( defined ( my $l = <F> ) ) {
		process ($l);
	}
        close F or die;
}
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Appending matching strings to specific lines (sed/bash) suntzu Programming 18 09-08-2012 04:29 PM
[SOLVED] search for 2 different strings in 2 diffrent lines threezerous Linux - Newbie 8 07-30-2012 04:42 PM
truncate strings on many lines mufea Linux - Newbie 2 02-23-2012 07:29 AM
How to remove lines and parts of lines from python strings? golmschenk Programming 3 11-27-2009 12:29 AM
Extract lines containing some strings without affectting sequential order cgcamal Programming 7 11-07-2008 12:57 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:22 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration