How to edit a file hexadecimally using bash?

SpecB · 02-27-2009, 05:46 PM

I have some (256) files that I want to edit, modifying certain bytes of them. One-by-one method is out of the question since it would take hours, but I also can't seem to find a method of editing the file in hex. I know the command hexdump, but it doesn't really help, or at least I didn't find out how yet.
Someone advised me to use vi and its command :%!xxd to edit in hex, but then again this method would require me to modify each and every file one by one. Maybe if I could use vi in a script it would work, but I didn't find any useful information about that either.

jlinkels · 02-27-2009, 06:11 PM

Use tr. It is perl script, but included in most distributions. You can specify bytes in octal to search/replace.

For example to replace the sequence 0x49 0x44 0x33 with 0x2d0x2d0x2d you give the command:

Code:

cat oldfile | tr "\111\104\063" "\055\055\055"  > newfile

jlinkels

SpecB · 02-27-2009, 07:14 PM

Hmmm, I knew about tr, but didn't know I could use it this way too.
It worked well, thank you!

wje_lq · 02-27-2009, 07:36 PM

jlinkels gave a good solution, but in case someone gets the wrong idea:

Quote:

It is perl script

Not so much, at least in some distributions. In stock Slackware 12.1, I get this:

Code:

wally:~$ file -L $(which tr)
/usr/bin/tr: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), stripped
wally:~$

and Xubuntu (Jaunty Jackelope) gives a similar result.

If there are distributions which implement tr as a Perl script (which would be trivial), I'd be curious to know which ones.

H_TeXMeX_H · 02-28-2009, 03:29 AM

Interesting, but I'm wondering if you could do this with some set space between offsets, and if it can only change the first instance. I needed to do this at one point and ended up writing a C program to do it. For example say you need to match 0x49 then X bytes later you need to match 0x44 and Y bytes later you need to match 0x33, and if all these 3 match then you change each of them to something else. Can that be done or is it too complicated.

Either way, I was just looking for a simpler solution, but I still have the code I wrote for that program, so if you need it say so.

SpecB · 02-28-2009, 04:27 AM

Well maybe you could "jump through" the file's bytes using two cycles for the rows and columns and maybe "cut" or something with a similar effect, setting X and Y as a parameter, and if the bytes are found in the places needed, it could change them.
I didn't have this problem because the filetype is very simple and can be edited using an external program. I set all of the desired bytes to a basically unused number, then just looked for it and changed all of them using the command aforementioned.
(Although I had some problems with the naming because for some reason the script added a ^M non-printable character to the end of the file extensions, probably due to my lazyness at 2am to write a proper cycle and not just use excel's CONCATENATE to generate the needed commands line by line, heh)

jlinkels · 02-28-2009, 06:14 AM

Quote:

Originally Posted by wje_lq

jlinkels If there are distributions which implement tr as a Perl script (which would be trivial), I'd be curious to know which ones.

Arghhh... in Debian Lenny it is an executable, so to see in Etch as well. I am sure I have seen it before in Debian as a Perl script, I looked at the source. For sure.

jlinkels

SpecB · 05-03-2009, 06:18 AM

Code:

cat oldfile | tr "\111\104\063" "\055\055\055"  > newfile

While this code works, I've managed to find a weakness of its: in this specific example, it will change every instance of 0x49, 0x44 and 0x33 into 0x2d, whether or not they are together. But what if, lets say, I only want to change the 0x44 bytes IF they are preceeded by 0x49 and followed by 0x33?
The server I'm contacting to be able to use linux lags like hell, so I can't really try out my ideas, but am I right if I think that a regular expression should do the job?

H_TeXMeX_H · 05-03-2009, 06:38 AM

As I said before a while ago I wrote a program to patch a game in this way ... it's kind of a hack:

Code:

// fix game.exe

#include <stdio.h>
#include <stdlib.h>

int main (void) 
{
	// ! init
	FILE * gfile;
	long fsize;
	char * buffer;
	size_t result;
	
	// init
	unsigned int i = 0;

	// open file read binary
	gfile = fopen ( "game.exe" , "rb" );
	if ( gfile == NULL ) { fputs ("ERROR: failed to open file game.exe !",stderr); exit (1); }

	// obtain file size
	fseek (gfile , 0 , SEEK_END);
	fsize = ftell (gfile);
	rewind (gfile);

	// allocate memory to contain file:
	buffer = (char*) malloc (fsize);
	if ( buffer == NULL ) { fputs ("ERROR: not enough memory to contain file game.exe !",stderr); exit (2); }

	// copy file into buffer
	result = fread (buffer,1,fsize,gfile);
	if ( result != fsize ) { fputs ("ERROR: file game.exe cannot be read !",stderr); exit (3); }

	// close file
	fclose (gfile);

	// search
	for ( i=0; i < fsize; i++)
	{
		if ( buffer[i] == '\x62' && buffer[i+1] == '\x02' && buffer[i-4729] == '\x01')
		{
			printf("The offsets are: %x %x %x\n", i-4729, i, i+1);
			printf("Fixing ...\n");
			buffer[i-4729] = '\x00';
			buffer[i] = '\x55';
			buffer[i+1] = '\x07';
			
			// open file write binary
			gfile = fopen ( "game.exe" , "wb" );
			if ( gfile == NULL ) { fputs ("ERROR: failed to open file game.exe !",stderr); exit (1); }
	
			// write file
			result = fwrite (buffer , 1 , fsize , gfile);
			if ( result != fsize ) { fputs ("ERROR: file game.exe cannot be written !",stderr); exit (3); }
	
			// close file
			fclose (gfile);
			
			break;
		}
	}
		
	// end
	free (buffer);
	return 0;
}

The important part is this statement:

Code:

if ( buffer[i] == '\x62' && buffer[i+1] == '\x02' && buffer[i-4729] == '\x01')

it checks the current index for x62, then the next byte for x02, then 4729 bytes back for x01, so change this statement as you see fit. I'm not a professional programmer and wrote this up in about 1 hour some time ago. I also hardcoded the file name as game.exe, this was done for safety reasons, because it is a dangerous program.

EDIT:
Forgot to mention that the bytes are changed here:

Code:

			buffer[i-4729] = '\x00';
			buffer[i] = '\x55';
			buffer[i+1] = '\x07';

And that it breaks out of the loop after it find one match.