[SOLVED] Get strings distributed along up to 3 lines

Perseus · 09-12-2013, 10:27 AM

Hello pan64,

Thank you. This time it seems to work the if condition and handles it the "9A" too.

I know which fruits to get, but I don't know all the "9X" that could appear in binary,
so, If this is not too complicated, for those values that are not defined in fruit array,
could be printed the value found for example, if I don't have defined "9C" in fruit array
and that value is found, print "fruit_9C". This could help me to enrich each time the code
to consider all cases.

Hello grail,

I've modified to

Code:

File.open(ARGV[0], "rb")

but I get the same error

Code:

$ ruby script.rb binary
script.rb:15:in `gets': encoding mismatch: UTF-8 IO with ASCII-8BIT RS (ArgumentError)
        from script.rb:15:in `gets'
        from script.rb:15:in `<main>'

binary.txt is the binary file, I only put that extension to be able to upload the file to forum, because
without extension is not allowed.

Thanks for all the help.

Regards

pan64 · 09-12-2013, 10:40 AM

so insert just after the while:

Code:

        while ( defined $y[$x] ) {
            if ( ! defined $fruit[hex($y[$x])] ) {
                print "Error: fruit for $y[$x] is missing\n";
                $x += 8;
                next;
            }
	    printf(" %s %d|%d|%s|%d",$fruit[hex($y[$x])],hex($y[$x+3]),hex($y[$x+4]),$y[$x+5],hex($y[$x+6]));
	    printf("|%d",hex($y[$x+7])) if defined $y[$x+7];
	    $x += 8;
	}

be care, not tested, probably does not work
Also you can modify the text printed....

grail · 09-12-2013, 10:49 AM

From what I can find this is an issue with 1.9.3 not handling things quite as smooth as 2.0, hence I receive no errors with or without the change and running it in all the ways you have tried.

Try removing the 'r' and see if that helps:

Code:

File.open(ARGV[0], "b")

Sorry to mess you around a bit but this is a new to me as far as I never tried to open a binary file in a previous version of ruby

grail · 09-12-2013, 10:53 AM

Just saw pan64's post and as another option you could also maybe add to the array a default like 'NO_FRUIT' at that point in the array:

Code:

while ( defined $y[$x] ) {
            $fruit[hex($y[$x])] = "NO_FRUIT" if ( ! defined $fruit[hex($y[$x])] );

	    printf(" %s %d|%d|%s|%d",$fruit[hex($y[$x])],hex($y[$x+3]),hex($y[$x+4]),$y[$x+5],hex($y[$x+6]));
	    printf("|%d",hex($y[$x+7])) if defined $y[$x+7];
	    $x += 8;
	}

I am sure pan64 will correct me if I have used the incorrect variable syntax

pan64 · 09-12-2013, 10:59 AM

Quote:

Originally Posted by grail

I am sure pan64 will correct me if I have used the incorrect variable syntax

perl -c <script> will do that for you.
Otherwise, yes, it should work and also we can print error message to stderr instead of stdout
(using $fruit[hex($y[$x])] several times will degrade performance)

Perseus · 09-12-2013, 11:35 AM

Hello pan64,

It works with that addition!

One question:
I was able to introduce ARG to pass the input file via command line argument doing this:

Code:

my $filename = $ARGV[0];

I was tryin to to the same for the block separator below
[CODE]
$/ = "\xff\x77";

Code:

doing this:
my $SEP = $ARGV[1];
$/ = "\xff\x".$SEP;

But is not working. If possible I'd like to put 77 or any other number here by command line argument.

Hello grail,

I have installed ruby 2.0 o a windows machine, I'll check in that machine the script and let you know.

Thanks for all the help.

pan64 · 09-13-2013, 12:40 AM

Code:

my $SEP = $ARGV[1];
$/ = "\xff" . chr(oct("0x$SEP"));

have you measured the execution times?

Perseus · 09-13-2013, 12:54 AM

Quote:

Originally Posted by pan64

Code:

my $SEP = $ARGV[1];
$/ = "\xff" . chr(oct("0x$SEP"));

have you measured the execution times?

Hello pan64,

Thank you. I've had an internet connection issue to answer before. Yes, I've tested both scripts.

Hello grail and pan64,

I just must to say that both are great scripts!

I've tested both scripts with an input binary of 2GB.

Code:

- Using Ruby script, the output takes 00:08:16 minutes
- Using Perl script, the output takes 00:04:08 minutes

pan64,

With Perl one issue, it runs up to the end and it seems to print all, but during the
execution are printed the following lines and I don't understand the error and I don't know how to fix it:

Code:

Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 1097738.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 1635529.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 1721239.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 3081257.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 3310170.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 4195601.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 4782198.
Hexadecimal number > 0xffffffff non-portable at ./Script.pl line 29, <F> chunk 4834811.

Line 29 is this:

Code:

printf("%d %s|%s",hex($a),$b,$c);

grail,

The Ruby script was only printing up to line 7, due to an hex to decimal convertion, the error was:

Code:

 `printf': invalid value for Integer(): "0000008" (ArgumentError)

I've fixed changing this:

Code:

printf("%d %s %s",$1,$2,$3)

to this:

Code:

printf("%d %s %s","0x"+$1,$2,$3)

After that, the code runs up to the end.

Thanks for all the help.

pan64 · 09-13-2013, 01:23 AM

see here: http://www.perlmonks.org/?node_id=742688 probably you have a 32bit perl or os.
You can try:

Code:

use Math::BigInt; # at the beginning of the script
printf("%s %s|%s",Math::BigInt->new("0x$a"), $b, $c);
# (not tested)

but it will definitely slow down the execution. If those printed lines were acceptable I would ignore these warnings.
no warnings 'portable'; # Support for 64-bit ints required

grail · 09-13-2013, 06:51 AM

Quote:

The Ruby script was only printing up to line 7, due to an hex to decimal convertion, the error was:

Just to clarify, as the start of the strings we are looking at are 0's followed by a number, Ruby has interpreted it as Octal.
If number is to be an int then the following is more correct:

Code:

printf("%d %s %s",$1.to_i,$2,$3)

If it is in fact supposed to be hex then you can process as you have or:

Code:

printf("%d %s %s",$1.to_i(16),$2,$3)

Below are changes for an array version of fruits to see if that runs quicker as suggested by pan64:

Code:

# add the below and remove hash
BEGIN{	$/="\xff\x77"
	fruit = []	}

fruit[0] = "APPLE"
fruit[1] = "LIME"
fruit[2] = "ORANGE"
fruit[3] = "GRAPE"
fruit[6] = "PEAR"
fruit[7] = "CHERRY"

# and change the following
printf(" %s ",fruit[y[0]])

# to
printf(" %s ",fruit[y[0].to_i(16)])

Perseus · 09-14-2013, 02:46 AM

Hello Pan64,

I've tested with "Math::BigInt;". It works, this time don't print the errors, but you're rigth, the execution time increases
from 00:04:08 minutes to 00:07:11 minutes. As you said maybe those warnings could be ignored only.

Hello grail,

I've tried to use the array in the way you said me, but doesn't print the associated values like APPLE, LIME, etc and if the array contains non-numerical values (something similar as what happened before with perl script), i.g., if I put the fruit["9A"]="MELON" I get this error:

Code:

Script.rb:20:in `[]=': no implicit conversion of String into Integer (TypeError)
	from Script.rb:20:in `<main>'

and line 20 contains this:

Code:

fruit["9A"] = "MELON"

And I was not able to test your Ruby script under Windows using IRB, because even when in Ubuntu and IRB has
Ruby 2.0.0, in Windows I'm getting this error:

Code:

C:\Scripts>ruby script.rb binaryfile
script.rb:18:in `gets': encoding mismatch: CP850 IO with UTF-8 RS
(ArgumentError)
        from script.rb:18:in `gets'
        from script.rb:18:in `<main>'

Thanks so much for the help again

grail · 09-14-2013, 09:45 AM

You will not be able to run in irb as it would need to be altered to be a module to be loaded and then executed.

As for the array setup, you would need to perform the same conversion as in the Perl script, but remembering that it only uses the individual character, ie instead of 91 it is just 1,
so instead of 9a it would be just a.

With the above information in hand, to add the letter portions to the array simply use:

Code:

fruit["a".hex] = "MELON"

And to overcome the issue of not having a corresponding entry in the array, make the following change:

Code:

fruit[y[0]] = "MISSING" if fruit[y[0]].nil?

printf(" %s ",fruit[y[0]])

ntubski · 09-14-2013, 10:33 AM

The ruby encoding error can be solved by running with -E BINARY option:

Code:

~/tmp$ ruby bin.rb binary1.txt 
bin.rb:15:in `gets': encoding mismatch: UTF-8 IO with ASCII-8BIT RS (ArgumentError)
	from bin.rb:15:in `gets'
	from bin.rb:15:in `<main>'
~/tmp$ ruby -E BINARY bin.rb binary1.txt 
1 53206445018934550 81474549232
2 53206445018934551 81474554768  2|48|8147526905|0 LIME 2|314|81475269559|0 GRAPE 12|159|8147526905|1|1 PEAR 14|235|81475269596|0 CHERRY 1|456345|81475269563|0 PROD_F 0|1|0|0|1|0|1|0|255|255|0|0|1|1
3 53206445018934552 81474557521 LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269565|0 PROD_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
4 53206445018934558 81477380427 LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269594|0 PROD_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
5 53206445018934559 81474663128
~/tmp$ ruby --version
ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-linux]

Perseus · 09-14-2013, 10:54 AM

Hello grail,

I don't receive anymore the "no implicit convertion error", but even when I've added the line

Code:

fruit["a".hex] = "MELON"

is not printing "MELON" in the output. I'm receiving
this output using the file I've attached in previous post called binary1.txt even when contains the string "9A":

Code:

1 53206445018934550 81474549232 
2 53206445018934551 81474554768  2|48|8147526905|0 LIME 2|314|81475269559|0 GRAPE 12|159|8147526905|1|1 PEAR 14|235|81475269596|0 CHERRY 1|456345|81475269563|0 PROF_F 0|1|0|0|1|0|1|0|255|255|0|0|1|1
3 53206445018934552 81474557521  LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269565|0 PROF_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
4 53206445018934558 81477380427  LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269594|0 PROF_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
5 53206445018934559 81474663128

Perseus · 09-14-2013, 10:57 AM

Quote:

Originally Posted by ntubski

The ruby encoding error can be solved by running with -E BINARY option:

Code:

~/tmp$ ruby bin.rb binary1.txt 
bin.rb:15:in `gets': encoding mismatch: UTF-8 IO with ASCII-8BIT RS (ArgumentError)
	from bin.rb:15:in `gets'
	from bin.rb:15:in `<main>'
~/tmp$ ruby -E BINARY bin.rb binary1.txt 
1 53206445018934550 81474549232
2 53206445018934551 81474554768  2|48|8147526905|0 LIME 2|314|81475269559|0 GRAPE 12|159|8147526905|1|1 PEAR 14|235|81475269596|0 CHERRY 1|456345|81475269563|0 PROD_F 0|1|0|0|1|0|1|0|255|255|0|0|1|1
3 53206445018934552 81474557521 LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269565|0 PROD_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
4 53206445018934558 81477380427 LIME 2|13|8147526905|0 GRAPE 12|13|8147526905|1|1 PEAR 12|13|81475269594|0 PROD_F 1|2|1|2|1|0|1|255|255|255|2|1|2|1
5 53206445018934559 81474663128
~/tmp$ ruby --version
ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-linux]

Hello ntubski,

Thank you, I'll check asap your suggestion.