[SOLVED] Searching for a specific hex string or byte array in various images

plasma33 · 05-05-2016, 11:09 PM

Hi guys,

I am doing some research on image processing and I wanted to know if its possible to search for a specific hex string/byte array in various images. It would be great if it gives me a list of images that has that specific string. Basically what grep -r "" does. For some reason grep doesn't do the job. I am not familiar with strings. I did had a look at "man strings" but it didn't help much. Anyways, I would like to look for a specific hex string say "0002131230443" (or even specific byte array i.e. base64 string) in several images in the same folder. Any help would be highly appreciated.

I found this code which exactly does what I want using xxd and grep. Please find the command below:

Code:

xxd -p /your/file | tr -d '\n' | grep -c '22081b00081f091d2733170d123f3114'

FYI:
It'll return 1 if the content matches, 0 else.

xxd -p converts the file to plain hex dump, tr -d '\n' removes the newlines added by xxd, and grep -c counts the number of lines matched.

Does anyone know how to run the code above in a specific directory using batch script. I have around 400 images and I want it to return only 400 if all the 400 images have that particular string. I found this script code below but it runs the same code over and over for 400 times returning either 0 or 1 each time:

Code:

#!/bin/bash

FILES=/FILEPATH/*
for f in $FILES
do
   echo "PROCESSING $f FILES"
   echo "-------------------"
   XXD -u $f | grep ABCD
   echo "-------------------"
done

Thanks guys.

Plasma33

DavidPhillips · 05-06-2016, 12:12 AM

I think you want to add the output of grub for each file. You could pipe the output into a variable and add it up there.

try something like this.

Code:

 xxd -u $f | grep ABCD | awk '{ SUM += $1; print $1} END { print SUM }'

plasma33 · 05-06-2016, 12:37 AM

Hi David,

Thank you for your reply.

I tried your code but its not giving me the output that I need. I am using the following script:

Code:

#!/bin/bash

FILES=/FILEPATH/*
for f in $FILES
do
   echo "PROCESSING $f FILES"
   echo "-------------------"
   xxd -p $f | tr -d '\n' | grep -c '22081b00081f091d2733170d123f3114' | awk '{ SUM += $1; print $1} END { print SUM }'
   echo "-------------------"
done

Its printing something like shown below but over and over and unfortunately its not giving me the final sum output, for instance 400:

Code:

PROCESSING $f FILES
-------------------
1
1
-------------------
PROCESSING $f FILES
-------------------
1
1
-------------------
.
.
.
.
PROCESSING $f FILES
-------------------
1
1
-------------------

Thanks, again.

Plasma33

syg00 · 05-06-2016, 12:42 AM

No need to convert to viewable text just to search it - use awk directly

Code:

awk '{/\x00\x02\x13\x12\x30\x44/ {print FILENAME ; exit}' $f

Note number of "nibbles" must be even. Count could also be added if required.

DavidPhillips · 05-06-2016, 01:17 AM

The output is a sum of the strings in each file.

For all files total you can use something like this.

Code:

cat *png | xxd -p | tr -d '/n' | grep -c '1234' | awk '{ SUM += $1 }; END { print SUM }'

plasma33 · 05-06-2016, 02:27 AM

Hi David,

Thank you for your kind reply.

Its still not working for me. I am getting output stating no such file or directory. All my images have a file extension of TIF but it still says no such file or directory. The output that I get is as follows:

Code:

PROCESSING $f FILES
-------------------
cat: *tif No such file or directory
1
1
-------------------
PROCESSING $f FILES
-------------------
cat: *tif No such file or directory
1
1
-------------------
.
.
.
.
PROCESSING $f FILES
-------------------
cat: *tif No such file or directory
1
1
-------------------

Thanks.

Plasma33

DavidPhillips · 05-06-2016, 11:54 AM

It appears that the files are not there. This would indicate that you need to supply the correct path to the files in your cat command. That should do it.

plasma33 · 05-07-2016, 12:30 AM

Hi David,

Sorry for the late reply. I did that but still its not generating the final total value. My current code looks like the one below:

Code:

cat foldername/foldername/*png | xxd -p | tr -d '/n' | grep -c '1234' | awk '{ SUM += $1 }; END { print SUM }'

I am getting the following output but without any errors this time:

Code:

PROCESSING $f FILES
-------------------
1
-------------------
PROCESSING $f FILES
-------------------
1
-------------------
.
.
.
.
PROCESSING $f FILES
-------------------
1
-------------------

Thanks.

Plasma33

plasma33 · 05-07-2016, 12:52 AM

Quote:

Originally Posted by syg00

No need to convert to viewable text just to search it - use awk directly

Code:

awk '{/\x00\x02\x13\x12\x30\x44/ {print FILENAME ; exit}' $f

Note number of "nibbles" must be even. Count could also be added if required.

Hi syg,

Your code worked perfectly but how do I get the total number of count at the end after the last line of output using "FILENAME". Thanks. I have around 400 images and your code is helping me find that string in all of them and so I need a final output of 400 at the end. How can I do that, please?

Thanks.

Plasma33

DavidPhillips · 05-07-2016, 12:57 AM

I entered and got this output. Output for first run was 1014, second criteria was 46.

Direct copy and paste from terminal.

Your output is confusing because it cannot be generated by that code. Did you insert the line somewhere in your do loop? The do loop only inputs one file at a time.

The total count your looking for is generated by this command without any loop.

Code:

david@david-Dell-System-XPS-L702X /media/david/Windows8/Documents and Settings/david/Pictures $ pwd
/media/david/Windows8/Documents and Settings/david/Pictures

david@david-Dell-System-XPS-L702X /media/david/Windows8/Documents and Settings/david/Pictures $ cat *png | xxd -p $f | tr -d '/n'|grep -c '1234' | awk '{ SUM += $1}; END { print SUM }'
1014

david@david-Dell-System-XPS-L702X /media/david/Windows8/Documents and Settings/david/Pictures $ cat *png | xxd -p $f | tr -d '/n'|grep -c '12345' | awk '{ SUM += $1}; END { print SUM }'
46

plasma33 · 05-07-2016, 01:23 AM

Thanks, David. I tried that but for me its returning 1 instead of 400. How is it working for you?

Plasma33

DavidPhillips · 05-07-2016, 01:25 AM

You can add some more info as well. Still one line of commands, no loops.

Code:

echo 'Total Files ' `ls *png | wc -l` ; echo 'Search string count ' `cat *png | xxd -p $f | tr -d '/n'|grep -c '12345' | awk '{ SUM += $1}; END { print SUM }'`


Total Files  37
Search string count  46

DavidPhillips · 05-07-2016, 01:28 AM

Quote:

Originally Posted by plasma33

Thanks, David. I tried that but for me its returning 1 instead of 400. How is it working for you?

Plasma33

Can you copy my command line above and paste it into your terminal in the path of the png files?

plasma33 · 05-07-2016, 01:48 AM

Quote:

Originally Posted by DavidPhillips

You can add some more info as well. Still one line of commands, no loops.

Code:

echo 'Total Files ' `ls *png | wc -l` ; echo 'Search string count ' `cat *png | xxd -p $f | tr -d '/n'|grep -c '12345' | awk '{ SUM += $1}; END { print SUM }'`


Total Files  37
Search string count  46

Thank you for the wonderful code. It works without any issues but still giving me the Search string count as 1 for some reason. My total files were 3 out of which two files should definitely give me a genuine "Search string count" of 2 but instead its giving me 1. I tried on all the 400 images but grep couldn't handle such a large amount of files and gave me a error of "grep: memory exhausted"
Sigh...

Anyways the code given by syg00 works fine but how do I convert that code according to your way such that it gives me "Total String search count", please. I will give it a shot too.

Code:

awk '{/\x00\x02\x13\x12\x30\x44/ {print FILENAME ; exit}' $f

Thanks.

Plasma33

syg00 · 05-07-2016, 02:10 AM

I would simply pipe it to "wc -l".