[SOLVED] How to replace 10th and 11th byte in a hex dump

Hello everybody!

I have a feeling that there is a very simple and efficient solution to my task I simply forgot. If I am wrong and you are a mod would you move this to Programming please?


I am working with hex dump looking like that

01 74 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70 ...

I look for a filter statement to replace the n th byte of a long line of data. I can tell n in advance, but would prefer not to hard code it.

My intended usage is

prog1 | missing_command | prog2

Thanks a lot for reading!



Regards, Heraton

You could consider the OR operation....

1010101 OR 0001000 = 1011101

Where the selected bit (the one in the middle) was replaced...

Just a flake of a thought...

Thor

Hope this helps

~[lizard]$ echo $example
01 74 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70
~[lizard]$ missing_command() { local n=$1 r=$2; sed -r "s/(.{$((3*$(($n-1))))})..(.*)/\1$r\2/"; }
~[lizard]$ echo $example | missing_command 1 99
99 74 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70
~[lizard]$ echo $example | missing_command 2 ff
01 ff 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70
~[lizard]$ echo $example | missing_command 10 cc
01 74 58 d9 00 1e 4c 08 00 cc 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70

Here's a simple gawk one-liner to do it:

$ echo "01 74 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70" | gawk -v n=5 -v new=FF '{print $0;$n=new;print $0}' | cat
01 74 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70
01 74 58 d9 FF 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70

Notes:

1. In that example, "prog1" is echo and "prog2," cat
2. The two -v settings are where you specify the byte number (n) and the replacement byte (new). If you were to put this in a bash script, you could use a syntax like -v n=${byte_number} -v new=${replacement_byte}
3. The print $0; is in there just to show the "before" value, a debugging check. Remove it after you test.

Thank you very much for your replies!

In fact both the sed as well as the awk example do perform very well, although I have to admit that the sed thing is out of my league. The awk thing rings a bell and looks pretty familiar, so I guess that was, what I had seen before.

Although I already have figured out some ways to automate replacing 4 or 6 byte in a line of hex dump, I assume I am doing things that would let more advanced scripting people run away in sheer terror.


I could pipe things 4 to 6 times, which solves the problem nicely but is anything but high performance...

cat Data | awk_repl <n> <Val1> | awk_repl <n+1> <Val2> | \
awk_repl <n+2> <Val3> | awk_repl <n+3> <Val14> | cat

I could also use specialised scripts each:

#!/bin/bash
n2=$(($1+1))
n3=$(($n2+1))
n4=$(($n3+1))

gawk -v p1=$1 -v new1=$2 -v p2=$n2 -v new2=$3 -v p3=$n3 -v new3=$4 -v p4=$n4 -v new4=$5 \
$p1=new1;$p2=new2;$p3=new3;$p4=new4;print $0}'

But those scripts look crude and I would have to write one for each length or go all the way to construct the awk expression depending on a length parameter. Although this sounds mad enough to try once, I don't believe that is the way it is meant to be. So basically what I am still longing for is a way to collapse "-v p1=$1 -v new1=$2 ... p4=$n4 -v new4=$5" to something like "from position n to n+3 replace with something that fits in"

Any suggestions would be highly appreciated.


Regards, Heraton

Well, you could do it in a few different ways. If you only want to cover the case of a set of contiguous lines, this might work:

gawk -v start=5 -v new=A1:B2:C3 'BEGIN{n=split(new,New,":")}{for(i=1;i<=n;++i)$(i+start-1)=New[i];print $0}

Here's that applied to your sample string:

$ echo "01 74 58 d9 00 1e 4c 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70" |gawk -v start=5 -v new=A1:B2:C3 'BEGIN{n=split(new,New,":")}{for(i=1;i<=n;++i)$(i+start-1)=New[i];print $0}'
01 74 58 d9 A1 B2 C3 08 00 21 85 69 57 40 00 50 ea 74 58 d9 00 5e 6d cb 61 74 31 70

(Note that, in this simple case, the redundant pipe to cat has been omitted.)

A more general solution would be to prepare a table of byte numbers and new values, read that in, and then apply it to each line of the input file. (You could even use a three column input table with {byte number, old byte, new byte}, and only replace the bytes when all the old bytes match.) That would be possible as a one-line program, but it would be a long line. Here's such a program written so you could put it in a file, make it executable, and use it as you middle command. (Note that the name of the pattern table is hard codded. It could be passed as a variable, but I was getting lazy.)

N.B.: This is untested code.

#!/usr/bin/gawk -f
#
# Read in the conversion table
#  byte expected_value new_value
# ...
BEGIN {
  N=0
  while (getline value < "byte_map_table") {
    ++N
    if (n=split(value,Value) != 3) {
      printf("Error: Line $d of the byte map table (%s) contained %d values, not three.\n",N,value,n)  > "/dev/stderr"
      ++error
      continue
    }
    Byte[N]=Value[1]
    Old[N]=Value[2]
    New[N]=Value[3]
  }
  if (error) exit(1)
}

{
  skip=0
  for (i in Byte) {
    if (Byte[i] < 1 || Byte[i] > NF) {
      printf("Warning: Record %d does not contain a byte in column %d. Skipping it.\n", NR, Byte[i]) > /dev/stderr
      skip=1
    }
    if (tolower($i) != tolower(Old[i])) {
      printf("Warning: Record %d: Byte %d is %s, not $s. Skipping it.\n",NR,i,$i,Old[i]) > "/dev/stderr
      skip=1
    }
  }
  if (!skip) {
    for (i in Byte) {
      $i=New[i]
    }
  }
  print $0
}

Thank you very much for your answer. Once more your solution is perfectly suitable for my task. For now I'll stick with your first proposal as I need to meet a deadline which expires on Thursday and I am already taking night shifts. But I promise to take a closer look at your second proposal next weekend, as I am sure I can learn quite a lot of that.

Regards, Heraton

Just as an alternative, you might like to know that substr() in Perl can also do assignments as well as extractions; neat huh
http://perldoc.perl.org/functions/substr.html

See also pack & unpack
http://perldoc.perl.org/functions/pack.html
http://perldoc.perl.org/functions/unpack.html

Thanks for that advice. I guess I will take a look at that soon. Right now I have to meet a deadline and Perl would not be compatible with my intended audience. Anyway, I guess enlightment is waiting down that road too.

Regards, Heraton