LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-05-2016, 11:09 PM   #1
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Rep: Reputation: Disabled
Searching for a specific hex string or byte array in various images


Hi guys,

I am doing some research on image processing and I wanted to know if its possible to search for a specific hex string/byte array in various images. It would be great if it gives me a list of images that has that specific string. Basically what grep -r "" does. For some reason grep doesn't do the job. I am not familiar with strings. I did had a look at "man strings" but it didn't help much. Anyways, I would like to look for a specific hex string say "0002131230443" (or even specific byte array i.e. base64 string) in several images in the same folder. Any help would be highly appreciated.

I found this code which exactly does what I want using xxd and grep. Please find the command below:
Code:
xxd -p /your/file | tr -d '\n' | grep -c '22081b00081f091d2733170d123f3114'
FYI:
It'll return 1 if the content matches, 0 else.

xxd -p converts the file to plain hex dump, tr -d '\n' removes the newlines added by xxd, and grep -c counts the number of lines matched.

Does anyone know how to run the code above in a specific directory using batch script. I have around 400 images and I want it to return only 400 if all the 400 images have that particular string. I found this script code below but it runs the same code over and over for 400 times returning either 0 or 1 each time:
Code:
#!/bin/bash

FILES=/FILEPATH/*
for f in $FILES
do
   echo "PROCESSING $f FILES"
   echo "-------------------"
   XXD -u $f | grep ABCD
   echo "-------------------"
done
Thanks guys.

Plasma33

Last edited by plasma33; 05-05-2016 at 11:59 PM.
 
Old 05-06-2016, 12:12 AM   #2
DavidPhillips
LQ Guru
 
Registered: Jun 2001
Location: South Alabama
Distribution: Fedora / RedHat / SuSE
Posts: 7,163

Rep: Reputation: 58
I think you want to add the output of grub for each file. You could pipe the output into a variable and add it up there.

try something like this.

Code:
 xxd -u $f | grep ABCD | awk '{ SUM += $1; print $1} END { print SUM }'

Last edited by DavidPhillips; 05-06-2016 at 01:30 AM.
 
Old 05-06-2016, 12:37 AM   #3
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Original Poster
Rep: Reputation: Disabled
Hi David,

Thank you for your reply.

I tried your code but its not giving me the output that I need. I am using the following script:
Code:
#!/bin/bash

FILES=/FILEPATH/*
for f in $FILES
do
   echo "PROCESSING $f FILES"
   echo "-------------------"
   xxd -p $f | tr -d '\n' | grep -c '22081b00081f091d2733170d123f3114' | awk '{ SUM += $1; print $1} END { print SUM }'
   echo "-------------------"
done
Its printing something like shown below but over and over and unfortunately its not giving me the final sum output, for instance 400:
Code:
PROCESSING $f FILES
-------------------
1
1
-------------------
PROCESSING $f FILES
-------------------
1
1
-------------------
.
.
.
.
PROCESSING $f FILES
-------------------
1
1
-------------------
Thanks, again.

Plasma33
 
Old 05-06-2016, 12:42 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 20,181

Rep: Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755
No need to convert to viewable text just to search it - use awk directly
Code:
awk '{/\x00\x02\x13\x12\x30\x44/ {print FILENAME ; exit}' $f
Note number of "nibbles" must be even. Count could also be added if required.

Last edited by syg00; 05-06-2016 at 02:42 AM. Reason: missed a bit of the code
 
Old 05-06-2016, 01:17 AM   #5
DavidPhillips
LQ Guru
 
Registered: Jun 2001
Location: South Alabama
Distribution: Fedora / RedHat / SuSE
Posts: 7,163

Rep: Reputation: 58
The output is a sum of the strings in each file.


For all files total you can use something like this.
Code:
cat *png | xxd -p | tr -d '/n' | grep -c '1234' | awk '{ SUM += $1 }; END { print SUM }'

Last edited by DavidPhillips; 05-06-2016 at 01:31 AM.
 
Old 05-06-2016, 02:27 AM   #6
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Original Poster
Rep: Reputation: Disabled
Hi David,

Thank you for your kind reply.

Its still not working for me. I am getting output stating no such file or directory. All my images have a file extension of TIF but it still says no such file or directory. The output that I get is as follows:
Code:
PROCESSING $f FILES
-------------------
cat: *tif No such file or directory
1
1
-------------------
PROCESSING $f FILES
-------------------
cat: *tif No such file or directory
1
1
-------------------
.
.
.
.
PROCESSING $f FILES
-------------------
cat: *tif No such file or directory
1
1
-------------------
Thanks.

Plasma33
 
Old 05-06-2016, 11:54 AM   #7
DavidPhillips
LQ Guru
 
Registered: Jun 2001
Location: South Alabama
Distribution: Fedora / RedHat / SuSE
Posts: 7,163

Rep: Reputation: 58
It appears that the files are not there. This would indicate that you need to supply the correct path to the files in your cat command. That should do it.
 
Old 05-07-2016, 12:30 AM   #8
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Original Poster
Rep: Reputation: Disabled
Hi David,

Sorry for the late reply. I did that but still its not generating the final total value. My current code looks like the one below:
Code:
cat foldername/foldername/*png | xxd -p | tr -d '/n' | grep -c '1234' | awk '{ SUM += $1 }; END { print SUM }'
I am getting the following output but without any errors this time:
Code:
PROCESSING $f FILES
-------------------
1
-------------------
PROCESSING $f FILES
-------------------
1
-------------------
.
.
.
.
PROCESSING $f FILES
-------------------
1
-------------------
Thanks.

Plasma33
 
Old 05-07-2016, 12:52 AM   #9
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Original Poster
Rep: Reputation: Disabled
Total Count, please

Quote:
Originally Posted by syg00 View Post
No need to convert to viewable text just to search it - use awk directly
Code:
awk '{/\x00\x02\x13\x12\x30\x44/ {print FILENAME ; exit}' $f
Note number of "nibbles" must be even. Count could also be added if required.
Hi syg,

Your code worked perfectly but how do I get the total number of count at the end after the last line of output using "FILENAME". Thanks. I have around 400 images and your code is helping me find that string in all of them and so I need a final output of 400 at the end. How can I do that, please?

Thanks.

Plasma33
 
Old 05-07-2016, 12:57 AM   #10
DavidPhillips
LQ Guru
 
Registered: Jun 2001
Location: South Alabama
Distribution: Fedora / RedHat / SuSE
Posts: 7,163

Rep: Reputation: 58
I entered and got this output. Output for first run was 1014, second criteria was 46.

Direct copy and paste from terminal.

Your output is confusing because it cannot be generated by that code. Did you insert the line somewhere in your do loop? The do loop only inputs one file at a time.

The total count your looking for is generated by this command without any loop.



Code:
david@david-Dell-System-XPS-L702X /media/david/Windows8/Documents and Settings/david/Pictures $ pwd
/media/david/Windows8/Documents and Settings/david/Pictures

david@david-Dell-System-XPS-L702X /media/david/Windows8/Documents and Settings/david/Pictures $ cat *png | xxd -p $f | tr -d '/n'|grep -c '1234' | awk '{ SUM += $1}; END { print SUM }'
1014

david@david-Dell-System-XPS-L702X /media/david/Windows8/Documents and Settings/david/Pictures $ cat *png | xxd -p $f | tr -d '/n'|grep -c '12345' | awk '{ SUM += $1}; END { print SUM }'
46

Last edited by DavidPhillips; 05-07-2016 at 01:11 AM.
 
1 members found this post helpful.
Old 05-07-2016, 01:23 AM   #11
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Original Poster
Rep: Reputation: Disabled
Thanks, David. I tried that but for me its returning 1 instead of 400. How is it working for you?

Plasma33
 
Old 05-07-2016, 01:25 AM   #12
DavidPhillips
LQ Guru
 
Registered: Jun 2001
Location: South Alabama
Distribution: Fedora / RedHat / SuSE
Posts: 7,163

Rep: Reputation: 58
You can add some more info as well. Still one line of commands, no loops.

Code:
echo 'Total Files ' `ls *png | wc -l` ; echo 'Search string count ' `cat *png | xxd -p $f | tr -d '/n'|grep -c '12345' | awk '{ SUM += $1}; END { print SUM }'`


Total Files  37
Search string count  46
 
Old 05-07-2016, 01:28 AM   #13
DavidPhillips
LQ Guru
 
Registered: Jun 2001
Location: South Alabama
Distribution: Fedora / RedHat / SuSE
Posts: 7,163

Rep: Reputation: 58
Quote:
Originally Posted by plasma33 View Post
Thanks, David. I tried that but for me its returning 1 instead of 400. How is it working for you?

Plasma33

Can you copy my command line above and paste it into your terminal in the path of the png files?
 
Old 05-07-2016, 01:48 AM   #14
plasma33
LQ Newbie
 
Registered: May 2016
Posts: 13

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by DavidPhillips View Post
You can add some more info as well. Still one line of commands, no loops.

Code:
echo 'Total Files ' `ls *png | wc -l` ; echo 'Search string count ' `cat *png | xxd -p $f | tr -d '/n'|grep -c '12345' | awk '{ SUM += $1}; END { print SUM }'`


Total Files  37
Search string count  46
Thank you for the wonderful code. It works without any issues but still giving me the Search string count as 1 for some reason. My total files were 3 out of which two files should definitely give me a genuine "Search string count" of 2 but instead its giving me 1. I tried on all the 400 images but grep couldn't handle such a large amount of files and gave me a error of "grep: memory exhausted"
Sigh...

Anyways the code given by syg00 works fine but how do I convert that code according to your way such that it gives me "Total String search count", please. I will give it a shot too.
Code:
awk '{/\x00\x02\x13\x12\x30\x44/ {print FILENAME ; exit}' $f
Thanks.

Plasma33
 
Old 05-07-2016, 02:10 AM   #15
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 20,181

Rep: Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755Reputation: 3755
I would simply pipe it to "wc -l".
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
C++ bin array to hex string and back teapottwo Programming 5 12-03-2015 07:43 PM
Migrating Java application from HP/UX to Linux - String to byte array issues JayMac2013 Linux - Newbie 1 04-07-2013 02:30 PM
[SOLVED] searching for hex string in files and folders cosgeazy Linux - Software 16 09-28-2012 07:34 AM
Searching a specific directory for files containing a text string lothwen Linux - Newbie 1 11-20-2010 11:58 AM
Hex output of a hex/ascii input string mlewis Programming 35 04-10-2008 01:05 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:45 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration