Problem Putting Hex Doublets into Right Form

telecom_is_me · 07-01-2008, 01:48 PM

I'm attempting to take the following "16 Rows of Doublets:

Code:

f6 f6 f6 28 28 28 01 02 03 00
9c 3c 04 5f 30 59 ed ea e1 8c
8c 00 00 00 00
00 00 00 00 00
00 00
00 00 55 04 5f 30 59 ed
33 ff 37 00 00 00 00 00
00 00
d1 9c 3c 04 5f 30
59 ed ea e1
00 00 00 00 00 00 00 00 00 00
b3 71 aa d1 9c 3c 04 5f 30 59
00 00 00 00 00 00 00 00 00 00

And put it into this form:

Code:

f6 f6 f6 28 28 28 01 02 03 00
9c 3c 04 5f 30 59 ed ea e1 8c
8c 00 00 00 00 00 00 00 00 00
00 00 00 00 55 04 5f 30 59 ed
33 ff 37 00 00 00 00 00 00 00
d1 9c 3c 04 5f 30 59 ed ea e1
00 00 00 00 00 00 00 00 00 00
b3 71 aa d1 9c 3c 04 5f 30 59
00 00 00 00 00 00 00 00 00 00

From the first block of Hex, Rows 3 and 4 become one, Rows 5 and 6 become 1, 7 and 8 become 1, 9 and 10 become 1.

I'm looking for a simple solution in bash or perl If anyones got an idea.

bigearsbilly · 07-01-2008, 03:22 PM

Code:

$ fmt -w 30 <  data.file
f6 f6 f6 28 28 28 01 02 03 00
9c 3c 04 5f 30 59 ed ea e1 8c
8c 00 00 00 00 00 00 00 00 00
00 00 00 00 55 04 5f 30 59 ed
33 ff 37 00 00 00 00 00 00 00
d1 9c 3c 04 5f 30 59 ed ea e1
00 00 00 00 00 00 00 00 00 00
b3 71 aa d1 9c 3c 04 5f 30 59
00 00 00 00 00 00 00 00 00 00

Hko · 07-01-2008, 03:42 PM

Wow, I just wrote this:

Code:

#!/bin/bash

n=1
while read line; do
    test "$line" || continue
    set $line
    while [ "$1" ]; do
        echo -n "$1"
        if [ $n -ge 10 ]; then
            n=0
            echo
        else
            echo -n " "
        fi
        let n++
        shift
    done
done </path/to/your/hex.file

..but bigearsbilly's solution is way more cooler!

bigearsbilly · 07-01-2008, 03:52 PM

i get it right occasionally

telecom_is_me · 07-02-2008, 03:16 PM

Quote:

Originally Posted by bigearsbilly

i get it right occasionally

I don't suppose you would know a way to remove the colums of white space from the stream of hex doublets?

As in, we start with this:

Code:

f6 f6 f6 28 28 28 01 02 03 00
9c 3c 04 5f 30 59 ed ea e1 8c
8c 00 00 00 00 00 00 00 00 00
00 00 00 00 55 04 5f 30 59 ed
33 ff 37 00 00 00 00 00 00 00
d1 9c 3c 04 5f 30 59 ed ea e1
00 00 00 00 00 00 00 00 00 00
b3 71 aa d1 9c 3c 04 5f 30 59
00 00 00 00 00 00 00 00 00 00

f6 f6 f6 28 28 28 01 02 03 00
9c 3c 04 5f 30 59 ed ea e1 8c
8c 00 00 00 00 00 00 00 00 00
00 00 00 00 55 04 5f 30 59 ed
33 ff 37 00 00 00 00 00 00 00
d1 9c 3c 04 5f 30 59 ed ea e1
00 00 00 00 00 00 00 00 00 00
b3 71 aa d1 9c 3c 04 5f 30 59
00 00 00 00 00 00 00 00 00 00

and we end up with this:

Code:

f6f6f628282801020300
9c3c045f3059edeae18c
8c000000000000000000
0000000055045f3059ed
33ff3700000000000000
d19c3c045f3059edeae1
00000000000000000000
b371aad19c3c045f3059
00000000000000000000

f6f6f628282801020300
9c3c045f3059edeae18c
8c000000000000000000
0000000055045f3059ed
33ff3700000000000000
d19c3c045f3059edeae1
00000000000000000000
b371aad19c3c045f3059
00000000000000000000

I was using sed to search for a blank space and replace it with no space, but it has to be looped to work, and I'm concerned that the solution might remove the blank line between the paragraphs of HEX.

Any suggestions?

Mr. C. · 07-02-2008, 03:47 PM

sed doesn't need to be looped:

sed 's/ //g' input

telecom_is_me · 07-02-2008, 04:26 PM

Quote:

Originally Posted by Mr. C.

sed doesn't need to be looped:

sed 's/ //g' input

Mr. C. to the rescue! Thanks Again!

telecom_is_me · 07-02-2008, 04:35 PM

Quote:

Originally Posted by Mr. C.

sed doesn't need to be looped:

sed 's/ //g' input

What if I wanted to make the reverse true and take:

Code:

f6f6f628282801020300
9c3c045f3059edeae18c
8c000000000000000000
0000000055045f3059ed
33ff3700000000000000
d19c3c045f3059edeae1
00000000000000000000
b371aad19c3c045f3059
00000000000000000000

and turn it into this:

Code:

f6 f6 f6 28 28 28 01 02 03 00
9c 3c 04 5f 30 59 ed ea e1 8c
8c 00 00 00 00 00 00 00 00 00
00 00 00 00 55 04 5f 30 59 ed
33 ff 37 00 00 00 00 00 00 00
d1 9c 3c 04 5f 30 59 ed ea e1
00 00 00 00 00 00 00 00 00 00
b3 71 aa d1 9c 3c 04 5f 30 59
00 00 00 00 00 00 00 00 00 00

Mr. C. · 07-02-2008, 04:56 PM

You've been asking a whole series of questions regarding conversions back and forth between various data presentations. These sure sound like homework questions...

telecom_is_me · 07-02-2008, 05:33 PM

Quote:

Originally Posted by Mr. C.

You've been asking a whole series of questions regarding conversions back and forth between various data presentations. These sure sound like homework questions...

LOL.. No, I graduated a month ago with a M.S. Degree in Telecommunications Technologies..

I'm attempting to write a stream interpreter for a custom embedded linux AP. The idea is that the an encrypted hex stream will come to a management system and the management system will convert the hex to binary and put the binary into the right format, then decrypt it's simple encryption algorithm and using various byte values it will be able to check system status information. "It's like SNMP, but well... not!"

It's a pet project that I've wanted to work on for the longest time but I was busily writing my dissertation so I had no time. Now I've got a few months off before I start a new job and I felt like finishing this.

But no, not homework, Telecom doesn't require programming just an intro to linux course that I took 8 years ago "hence why I'm lost with a lot of this."

Mr. C. · 07-02-2008, 05:46 PM

Ok, fair enough. I was mislead by the series of step-by-step questions, rather than the big picture; this is often a tip of a student cramming for homework due. My apologies.

sed 's/$..$/\1 /g; s/ $//'

Hko · 07-02-2008, 05:49 PM

Code:

bash$ cat file.txt
f6f6f628282801020300
9c3c045f3059edeae18c
8c000000000000000000
0000000055045f3059ed
33ff3700000000000000
d19c3c045f3059edeae1
00000000000000000000
b371aad19c3c045f3059
00000000000000000000

Code:

bash$ sed 's/\(..\)/\1 /g' file.txt
f6 f6 f6 28 28 28 01 02 03 00 
9c 3c 04 5f 30 59 ed ea e1 8c 
8c 00 00 00 00 00 00 00 00 00 
00 00 00 00 55 04 5f 30 59 ed 
33 ff 37 00 00 00 00 00 00 00 
d1 9c 3c 04 5f 30 59 ed ea e1 
00 00 00 00 00 00 00 00 00 00 
b3 71 aa d1 9c 3c 04 5f 30 59 
00 00 00 00 00 00 00 00 00 00

telecom_is_me · 07-02-2008, 06:04 PM

I love it when I get two identical responses at the same time, it bring me confidence in the forum.

Mr. C. ... sorry for the step by steps, I may know the big picture but coming up with a procedure for getting my result has been a trip. I've got it down now though I think.

The basic idea is:

1.) Data from Open Source AP Streams in, in Hex
2.) I strip that data down to just the components I want
3.) Convert the remaining hex in dublets to binary
4.) (And this is next

xor the binary cyphertext against a key
5.) Convert the resultant binary back into hex doublets
6.) Compare the doublets to known values of up/down, temp, cpu load, mem load.
7.) Make a nice front end using dialog or zenity

It's taken my this week basically to figure out the process as I play with perl for the first time.

Thanks for all your help.

Mr. C. · 07-02-2008, 06:13 PM

No worries. Sounds like you're on track. Most likely, some of the steps can be collapsed into a single filter. But you have the pieces now.

bigearsbilly · 07-07-2008, 12:43 PM

perl?

use the pack unpack functions for the translations
here are some silly toys I made to play with

hex_read

Code:

#!/usr/local/bin/perl -n

tr/0-9a-fA-F//cd;
print pack "H*" , $_;

hex_write

Code:

#!/usr/local/bin/perl -ns
BEGIN {
    warn "-chomp to ignore new line characters\n" unless $chomp;
    warn "-neat for nice easier spacing\
	(hint: finish off with 'fmt' command)\n" unless $neat;
	warn "\n";

}
chomp if $chomp;
@L =  unpack("H2" x length($_), $_) ;
print $neat? "@L\n":@L;

binary_read

Code:

#!/bin/perl -n

tr/01//cd;
print pack "B*" , $_;

binary_write

Code:

#!/bin/perl -ns
BEGIN {
    warn "-chomp to ignore new line characters\n" unless $chomp;
    warn "-neat for nice easier spacing\
	(hint: finish off with 'fmt' command)\n" unless $neat;
}
chomp if $chomp;
@L = unpack "B8" x length($_), $_ ;

print $neat? "@L\n":@L;