[SOLVED] Need to output md5 or sha1 along with fullpath and filesize

hanker · 08-26-2010, 10:03 PM

Hi all,

I am trying to output md5 or sha1 along with fullpath/filename and file size but I dont seem to find a way to do this.

with

Quote:

find . -printf '%s %p'

i can retrieve size and fullpath and filename

however I am not able to merge that info with the md5 or sha1 of the file

my aim is to have a file such as this

6435b607f86b6e6be1e77bb3b1987677d1377275 ./abc/asda/file1.txt 404
6435b607f86b6e6be1e77bb3b987677d13772725 ./abc/asda/file2.txt 1404

also, performance is an issue for me, since i need to get the info out of 10m files (approx 6TB), so commands like find are preferred and less iterations among commands would be great too.

any ideas?

btw i've tried to use something like this

Quote:

find . -type f -printf '%s %p'| xargs awk '{x=system("md5sum "$2)}END {print x" "$2" "$1}'

but variable x contains the return value of the system command md5sum and not the stdout

thanks a million
Hanker

Tinkster · 08-26-2010, 11:46 PM

Hi, welcome to LQ!

Something like

Code:

#!/bin/bash
read file size
md5=$(md5sum $file)
printf "%s\t%s\t$s\n", $md5, $file, $size

Save as md5.sh

And then

Code:

find . -type f  -printf '%s %p\n' | xargs md5.sh

Untested. :}

Cheers,
Tink

David the H. · 08-26-2010, 11:52 PM

Well, first of all, you're right about the system function. It returns the exit value of the command, not the command itself.

But I also see a couple of other errors. First of all, take a look at the output of the find command. You need to specify newlines with printf. And the END section only prints the final values, not once for each line.

In any case, I don't see any easy one-liner way to accomplish what you want. But a simple loop can do the trick.

Code:

while read x; do

     size=${x% *}
     file=${x#* }
     sum="$(md5sum $file 2>/dev/null)"
     sum=${sum%% *}     #strip off filename to get sum only*

     echo "$sum $file $size"

done <<<"$(find . -type f -printf "%s %p\n")"

*Note that the output of md5sum shows the sum, then the filename separated by two spaces. As you seem to want only a single space between each part, I've removed the filename, then printed it separately again on the final line. This step could be removed if you don't mind the extra spaces.

Edit: Hah...tinkster got here first. But I've now added a link too.

ghostdog74 · 08-27-2010, 12:55 AM

bash 4

Code:

#!/bin/bash
shopt -s globstar
for file in /fullpath/**
do
  echo "fullpath: $file"
  echo "filename: ${file##*/}"
  md5=$(md5sum file)
  sha1=$(sha1sum file)
  echo "md5sum : ${md5% *}"
  echo "sha1 : ${sha1% *}"
done

hanker · 08-27-2010, 07:45 AM

Thank you All.

i still have some problems since i am running the script in cygwin
basically i'm getting an error

i've modified the script to understand what is going on

here is the md5.sh

Quote:

#!/bin/bash
read file size
echo "[$file] [$size]"
md5=$(openssl sha1 $file)
printf "[%s] [%s] [%s]\n" $md5 $file $size

the output of

Quote:

find . -type f -printf "%s %p\n"

is

Quote:

124 ./md5.sh
5 ./test.txt

while the error appears in the full command

Quote:

find . -type f -printf "%s %p\n" | xargs md5.sh

Quote:

': not a valid identifierze
[] []
] []9a3ee5e6b4b0d3255bfef95601890afd80709

i'd appreciate some more comments...

BTW Tinkster your approach is the most scalable/effective and that's why i'm using that.

Note that the read command seem to read only "file si" and not "file" and "size"

thanks again
Hanker

grail · 08-27-2010, 09:01 AM

@Tinkster - I am going to need help with this one too as the xargs errors for me with:

Code:

xargs: md5.sh: No such file or directory

This after running:

Code:

find -type f -printf '%s %p\n' | xargs md5.sh

I figure I am missing something but am not real familiar with xargs.

grail · 08-27-2010, 09:08 AM

Quote:

Originally Posted by David the H.

Note that the output of md5sum shows the sum, then the filename separated by two spaces.

I didn't seem to get this affect. I only had one space with the following:

Code:

while read -r s f
do
	echo -n $(md5sum "$f")
	echo " $s"
	echo -n $(sha1sum "$f")
	echo " $s"
done< <(find -type f -printf '%s %p\n')

hanker · 08-27-2010, 10:09 AM

@grail thanks for looking at these:
1. for the first post you probably need to create the md5.sh following Thinkster info at the start of the thread.
2. with the script in your second post i've got

Quote:

$ ./script.sh
./script.sh: line 7: syntax error near unexpected token `done'
./script.sh: line 7: `done << (find -type f -printf '%s %p\n')'

btw i've tried with both "done< <" and "done <<" but no luck...

note i'm using cygwin to run these commands/scripts and i've read on a few blogs that read in cygwin bash does not behave as in ksh, that's probably why i'm stuck...

any other ideas?

hanker · 08-27-2010, 05:09 PM

I can consider this closed in Ubuntu - solution adopted is David the H. thank ye a bunch

grail · 08-28-2010, 03:27 AM

Quote:

1. for the first post you probably need to create the md5.sh following Thinkster info at the start of the thread.

Yes I had created the md5.sh file and it was executable, but it still did not work for me.

Yes there should be a space between the 2 input redirections - <[ ]<
And for me it provides output like:

Quote:

d41d8cd98f00b204e9800998ecf8427e ./file3 0
da39a3ee5e6b4b0d3255bfef95601890afd80709 ./file3 0
50e5682375f97e1fed670905889f04b9 ./logfile 90
c13c5469ac1d7b60fb02a3c42cca18b487128ce2 ./logfile 90