LinuxQuestions.org

LinuxQuestions.org (http://www.linuxquestions.org/questions/index.php)
-   Linux - General (http://www.linuxquestions.org/questions/forumdisplay.php?f=1)
-   -   How can I md5 recursively and place a checksum file in each directory? (http://www.linuxquestions.org/questions/showthread.php?t=800769)

peter1234 04-08-2010 06:55 AM

[SOLVED] How can I md5 recursively and place a checksum file in each directory?
 
Hi All,

I would like to know how to md5 recursively and place the checksum file in each directory, consisting only off checksums from that directory. as if I were to run md5sum in every subdirectory one by one manually.
I tried google for 1 or 2 days but it is difficult to get google understand what exactly I want.

At the moment I use the following:

$ find . -type f 2>/dev/null -exec md5sum {} \; >checksum.txt
(I found this by googling )

This md5 recursively BUT place all the checksums and file names (including path from starting directory) in one file (in this case 'checksum.txt') in the directory where I execute the command from. Problem with this method is if I were to copy a subdirectory to some were I have to manually edit the checksum file and remove all entries that are not relevant to that subdirectory.

I also want to use sha1, 256 & 512 and any other checksum generating programs so a method were I can substitute md5sum with sha1sum and so on will be great.

My shell scrip experience is quite basic.

Thank you for your help,

Peter.



-Zenwalk6, K=2.6.28.7 (stock)

jschiwal 04-08-2010 07:04 AM

You could search for directories instead and run a script or use printf to create a script. For searching for files in a single directory, look at `-maxdepth 1'.

peter1234 04-09-2010 11:52 PM

Thanks for the pointers....

I had a look at 'man find' and found a way to look for subdirectories.

I used the following command,
$ find . -type d 2>/dev/null >subdirectories.list

this is what is inside file 'subdirectories.list'

./dir1
./dir2
./dir3
./dir4/dir7
./dir4/dir7/dir8
./dir4/dir7/dir8/dir9


1st half of the problem is solved (getting a list of subdirectories)

------------------

2nd half of the problem (executing md5sum in each subdirectories or giving the subdirectory path to md5sum)

Method 1
manually editing the 'subdirectories.list' file and add 'md5sum' in front off every directory and '/*' at the end of every directory , this is ok for small number but can take a while when there are lot of subdirectories.

Is there a script that can do it (my script knowledge is at the level off 'echo hello world' level)?


Method 2 (executing 'md5sum' on the same line as 'find' command using option '-execdir')

I tried:
$ find . -type d 2>/dev/null -execdir md5sum >chsum.md5

I get a 0kb file called 'chsum.md5', on the directory where the command was executed but did not find 'chsum.md5' in any subdirectories. so I assume 'md5sum' is executed but the subdirectory paths are not pass on to 'md5sum' or md5sum was not executed at all.

then I tried:
$ find . -type d 2>/dev/null -execdir md5sum * >chsum.md5

Thinking md5sum did not do anything since it was not told to do so, same result as above
0kb 'chsum.md5' file in the directory where the command was executed.

If any one can help with the 2nd half of the problem, Method 1 or 2 would appreciate it very much.

Thank you for your time.

Peter.

I also read about options '-print' and '-printf' (under 'find') but could not figure out how to use it.

mweed 04-10-2010 01:12 AM

Short and sweet. Sorry for lack of any explanation, but it will handle spaces in the filenames and keep the sums files tidy. It depends on the output of file to be consistent. file can be rather inconsistent though. Should probably replace it with ls -l or stat or wc -c. meh.
Code:

find ./ -type d | while read line ; do echo $line ; pushd "$line"; cat /dev/null > md5sums; find ./ -type f -maxdepth 1 -exec md5sum {} \; | grep -vE '^[a-z0-9]{32}  \./md5sums$' >> md5sums; if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi; popd ; done

peter1234 04-15-2010 04:35 AM

:D

Thank you very much for your input 'mweed' :hattip: it works like a charm, one liner that will create md5sum file in every subdirectories below the starting dir.


:newbie:
if it is not asking too much, can you or anyone else tell me what dose, so I can understand :study: and it and apply in other placese.

Tankyou for your time,

Peter

As soon as I figure out how to thank 'mweed' ( I'll add 'Solved' to the title). Bytheway how do you thank some some in LQ ?? :scratch: :newbie:

bakdong 04-15-2010 07:36 AM

Quote:

Originally Posted by peter1234 (Post 3936176)
if it is not asking too much, can you or anyone else tell me what dose, so I can understand :study: and it and apply in other placese.

I'll have a go.....

Code:

find ./ -type d | while read line ;
list all the directories under the current one and pipe the list to the while loop (which will continue to the end of the list)

Code:

do
Start the command secion

Code:

echo $line ;
Echo the first directory name to the output

Code:

pushd "$line";
change to that directory, and push the current directory on to the stack (for use later)

Code:

cat /dev/null > md5sums;
Put nothing into the md5sums file. Could also have just been '> md5sums' ;)

Code:

find ./ -type f -maxdepth 1 -exec md5sum {} \;
list all the filenames in the current directory only, and run md5sum on them

Code:

| grep -vE '^[a-z0-9]{32}  \./md5sums$' >> md5sums;
pipe the output of md5sum through grep to do something I don't quite understand, which is to take the lines that don't match (-v) the extended (-E) expression, which is any line which starts with 32 alphnumeric characters, followed by space dot/md5sums and the end of line. Ah, got it now, this removes the md5sum of the actual 'md5sums' file. This could be made redundant by using the -not option of find:

find ./ -type f -not -name "md5sums" -maxdepth 1 -exec md5sum {} \;

Code:

if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi;
delete the md5sums file if it's empty. Could also have been
'[ -s md5sums ] || rm -f md5sums'
Not sure how platform dependant that is though. I'm using bash.

Code:

popd ;
go back to the previous directory on the stack

Code:

done
finish the command loop and go back for the next line


Quote:

As soon as I figure out how to thank 'mweed' ( I'll add 'Solved' to the title). Bytheway how do you thank some some in LQ ?? :scratch: :newbie:
Just say thank you! Or click on the 'Thanks' button on the post you liked. Or click on the 'This post was helpful' link.... or all three!


That was fun! Thanks, mweed!

peter1234 04-15-2010 10:23 AM

Thank you for the explanation bakdong,:hattip: thing make sense.

With the help off your explanation I made some changes to the command given by 'mweed'

I removed:

Code:

cat /dev/null > md5sums
#did not see why create file 'md5sums' in every dir (maybe I am wrong)

and the corresponding removal of empty 'md5sums' file part

Code:

if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi;
do you (or any one else) see any thing wrong with removing the above parts??

final command after removing bits

Code:

find ./ -type d | while read line ; do echo $line ; pushd "$line"; find ./ -type f -maxdepth 1 -exec md5sum {} \; | grep -vE '^[a-z0-9]{32}  \./md5sums2nd$' >> md5sums2nd; popd ; done
Looking forward to your comment,

Peter

bakdong 04-15-2010 11:55 AM

The idea behind the '> md5sums' bit is to ensure that old 'md5sums' files that may exist in that dir are initialised. You could eliminate a complete line of grep by including that test in the second find, and I'm not sure about your '2nd' suffix on the md5sums filenames.

peter1234 04-15-2010 12:51 PM

Quote:

> md5sums' bit is to ensure that old 'md5sums' files that may exist in that dir are initialised
I see, I did not think off that.....:doh:

I ran the command that 'mweed' gave and I got md5sums in all the subdirectories with files in (as expected) , and after reading your (bakdong) explanation I was looking for ways to make the command shorter, and removed

Code:

cat /dev/null > md5sums
if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi;

and I added the suffix "2nd" to "md5sums" so I can distinguish between md5sum files created from 'mweed' command and my modified command, it serve no other funtion (I cut and past it from command line and forgot to remove the suffix).:redface:

peter1234 04-15-2010 10:48 PM

I am just testing my new signature. :D

rahulrsheth 11-14-2012 01:01 PM

another way
 
find path/folder -type f -iname '*' -exec md5sum {} > checksums.txt \;


All times are GMT -5. The time now is 12:18 PM.