hash all files in subdirs

madivad · 04-30-2008, 10:22 PM

What is the best way to do this?

I see solaris (I'm using developer preview 2) has an md5sum command, but it's not very friendly. it is upset up dirnames (simple error) and will not recurse.

I have tried args and am hitting my head against a brick wall, these are my attempts so far:

1) build a list of files with
find /store/data

This will print a list of files to be hashed

2) pass it through xargs to md5sum
find /store/data | xargs md5sum | tee sums

the tee sums on the end is just so that it prints to screen and a log file to be used later, at least that is the intention

The main problem I am having is that there are whitespaces in the filenames, and the -0 (or --null) of xargs is not implemented in solaris, nor is the -print0 of find.

on directories and files without spaces, it works, but parses by space (and hence breaks up) when a filename or dir has a space. Also, because of filename problems, the listing is halted only a few files into the process, yet without "xargs md5sum", the whole list of all files is processed

Any hints or suggestions?

jlliagre · 05-01-2008, 02:16 AM

Try this:

Code:

find /store/data -type f -exec xargs md5sum {} + | tee sums

madivad · 05-01-2008, 03:13 AM

Quote:

Originally Posted by jlliagre

Try this:

Code:

find /store/data -type f -exec xargs md5sum {} + | tee sums

just doesn't work, it seems xargs is now having a problem with "max arglist" being greater than 2048 characters.

Now that can't be referring to the length of filenames, even with burried directories, I can't see a filename length of 2K characters! LOL

Also, I should point out, there are about 22000 files that this needs to process, is that the problem?

This is the error I am receiving:

Code:

dave@solaris:~# find /solaraid/TB2 -type f -exec xargs md5sum {} +
xargs: a single arg was greater than the max arglist size of 2048 characters
xargs: a single arg was greater than the max arglist size of 2048 characters
xargs: a single arg was greater than the max arglist size of 2048 characters
dave@solaris:~#

and considering it's 22000 files, I am only getting this error 3 times. And when passed to the file (as in: | tee sums), the file is empty

jlliagre · 05-01-2008, 04:12 AM

My mistake, a cut and paste error, this is really what I wanted to suggest:

Code:

find /store/data -type f -exec md5sum {} + | tee sums

madivad · 05-01-2008, 06:34 AM

Quote:

Originally Posted by jlliagre

My mistake, a cut and paste error, this is really what I wanted to suggest:

Code:

find /store/data -type f -exec md5sum {} + | tee sums

No problems, if I had a little better grasp of the ins and outs of unix commands I could have spotted it myself anyway, but then I wouldn't have posted here asking for help ;-P

As always, your help is appreciated immensely.

madivad · 05-01-2008, 07:01 AM

I have just been reading over the find/exec command operation, and with the appended '+'...

Am I reading this correctly?

Quote:

-exec command {} +
This variant of the -exec option runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of invoca-
tions of the command will be much less than the number of
matched files. The command line is built in much the same way
that xargs builds its command lines. Only one instance of '{}'
is allowed within the command. The command is executed in the
starting directory.

Does this mean that all the file names are concatenated to a command line and passed to the md5sum? That os the way I am reading it, and to make it more interesting, that males a command line of over 2,000,000 characters!

That's pretty impressive!

Ahh, re-reading it on preview...
"the total number of invocations of the command will be much less than the number of matched files."

So, it might call the md5sum application 100 or 1000 times, but not the full 22,000 times (the number of files)? or is it called just once?

jlliagre · 05-01-2008, 09:37 AM

md4sum is called the minimum number of times possible. It could be once only if the list of files fit in a command line.