LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-08-2010, 06:55 AM   #1
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Rep: Reputation: 2
[SOLVED] How can I md5 recursively and place a checksum file in each directory?


Hi All,

I would like to know how to md5 recursively and place the checksum file in each directory, consisting only off checksums from that directory. as if I were to run md5sum in every subdirectory one by one manually.
I tried google for 1 or 2 days but it is difficult to get google understand what exactly I want.

At the moment I use the following:

$ find . -type f 2>/dev/null -exec md5sum {} \; >checksum.txt
(I found this by googling )

This md5 recursively BUT place all the checksums and file names (including path from starting directory) in one file (in this case 'checksum.txt') in the directory where I execute the command from. Problem with this method is if I were to copy a subdirectory to some were I have to manually edit the checksum file and remove all entries that are not relevant to that subdirectory.

I also want to use sha1, 256 & 512 and any other checksum generating programs so a method were I can substitute md5sum with sha1sum and so on will be great.

My shell scrip experience is quite basic.

Thank you for your help,

Peter.



-Zenwalk6, K=2.6.28.7 (stock)

Last edited by peter1234; 04-15-2010 at 01:15 PM. Reason: to add 'SOLVED' to the title
 
Old 04-08-2010, 07:04 AM   #2
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
You could search for directories instead and run a script or use printf to create a script. For searching for files in a single directory, look at `-maxdepth 1'.
 
Old 04-09-2010, 11:52 PM   #3
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Original Poster
Rep: Reputation: 2
Thanks for the pointers....

I had a look at 'man find' and found a way to look for subdirectories.

I used the following command,
$ find . -type d 2>/dev/null >subdirectories.list

this is what is inside file 'subdirectories.list'

./dir1
./dir2
./dir3
./dir4/dir7
./dir4/dir7/dir8
./dir4/dir7/dir8/dir9


1st half of the problem is solved (getting a list of subdirectories)

------------------

2nd half of the problem (executing md5sum in each subdirectories or giving the subdirectory path to md5sum)

Method 1
manually editing the 'subdirectories.list' file and add 'md5sum' in front off every directory and '/*' at the end of every directory , this is ok for small number but can take a while when there are lot of subdirectories.

Is there a script that can do it (my script knowledge is at the level off 'echo hello world' level)?


Method 2 (executing 'md5sum' on the same line as 'find' command using option '-execdir')

I tried:
$ find . -type d 2>/dev/null -execdir md5sum >chsum.md5

I get a 0kb file called 'chsum.md5', on the directory where the command was executed but did not find 'chsum.md5' in any subdirectories. so I assume 'md5sum' is executed but the subdirectory paths are not pass on to 'md5sum' or md5sum was not executed at all.

then I tried:
$ find . -type d 2>/dev/null -execdir md5sum * >chsum.md5

Thinking md5sum did not do anything since it was not told to do so, same result as above
0kb 'chsum.md5' file in the directory where the command was executed.

If any one can help with the 2nd half of the problem, Method 1 or 2 would appreciate it very much.

Thank you for your time.

Peter.

I also read about options '-print' and '-printf' (under 'find') but could not figure out how to use it.
 
Old 04-10-2010, 01:12 AM   #4
mweed
Member
 
Registered: Mar 2006
Posts: 33

Rep: Reputation: 17
Short and sweet. Sorry for lack of any explanation, but it will handle spaces in the filenames and keep the sums files tidy. It depends on the output of file to be consistent. file can be rather inconsistent though. Should probably replace it with ls -l or stat or wc -c. meh.
Code:
find ./ -type d | while read line ; do echo $line ; pushd "$line"; cat /dev/null > md5sums; find ./ -type f -maxdepth 1 -exec md5sum {} \; | grep -vE '^[a-z0-9]{32}  \./md5sums$' >> md5sums; if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi; popd ; done
 
1 members found this post helpful.
Old 04-15-2010, 04:35 AM   #5
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Original Poster
Rep: Reputation: 2


Thank you very much for your input 'mweed' it works like a charm, one liner that will create md5sum file in every subdirectories below the starting dir.



if it is not asking too much, can you or anyone else tell me what dose, so I can understand and it and apply in other placese.

Tankyou for your time,

Peter

As soon as I figure out how to thank 'mweed' ( I'll add 'Solved' to the title). Bytheway how do you thank some some in LQ ??
 
Old 04-15-2010, 07:36 AM   #6
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
Quote:
Originally Posted by peter1234 View Post
if it is not asking too much, can you or anyone else tell me what dose, so I can understand and it and apply in other placese.
I'll have a go.....

Code:
find ./ -type d | while read line ;
list all the directories under the current one and pipe the list to the while loop (which will continue to the end of the list)

Code:
do
Start the command secion

Code:
echo $line ;
Echo the first directory name to the output

Code:
pushd "$line";
change to that directory, and push the current directory on to the stack (for use later)

Code:
cat /dev/null > md5sums;
Put nothing into the md5sums file. Could also have just been '> md5sums'

Code:
find ./ -type f -maxdepth 1 -exec md5sum {} \;
list all the filenames in the current directory only, and run md5sum on them

Code:
| grep -vE '^[a-z0-9]{32}  \./md5sums$' >> md5sums;
pipe the output of md5sum through grep to do something I don't quite understand, which is to take the lines that don't match (-v) the extended (-E) expression, which is any line which starts with 32 alphnumeric characters, followed by space dot/md5sums and the end of line. Ah, got it now, this removes the md5sum of the actual 'md5sums' file. This could be made redundant by using the -not option of find:

find ./ -type f -not -name "md5sums" -maxdepth 1 -exec md5sum {} \;

Code:
if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi;
delete the md5sums file if it's empty. Could also have been
'[ -s md5sums ] || rm -f md5sums'
Not sure how platform dependant that is though. I'm using bash.

Code:
popd ;
go back to the previous directory on the stack

Code:
done
finish the command loop and go back for the next line


Quote:
As soon as I figure out how to thank 'mweed' ( I'll add 'Solved' to the title). Bytheway how do you thank some some in LQ ??
Just say thank you! Or click on the 'Thanks' button on the post you liked. Or click on the 'This post was helpful' link.... or all three!


That was fun! Thanks, mweed!
 
1 members found this post helpful.
Old 04-15-2010, 10:23 AM   #7
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Original Poster
Rep: Reputation: 2
Thank you for the explanation bakdong, thing make sense.

With the help off your explanation I made some changes to the command given by 'mweed'

I removed:

Code:
cat /dev/null > md5sums
#did not see why create file 'md5sums' in every dir (maybe I am wrong)

and the corresponding removal of empty 'md5sums' file part

Code:
if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi;
do you (or any one else) see any thing wrong with removing the above parts??

final command after removing bits

Code:
find ./ -type d | while read line ; do echo $line ; pushd "$line"; find ./ -type f -maxdepth 1 -exec md5sum {} \; | grep -vE '^[a-z0-9]{32}  \./md5sums2nd$' >> md5sums2nd; popd ; done
Looking forward to your comment,

Peter
 
Old 04-15-2010, 11:55 AM   #8
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
The idea behind the '> md5sums' bit is to ensure that old 'md5sums' files that may exist in that dir are initialised. You could eliminate a complete line of grep by including that test in the second find, and I'm not sure about your '2nd' suffix on the md5sums filenames.
 
Old 04-15-2010, 12:51 PM   #9
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Original Poster
Rep: Reputation: 2
Quote:
> md5sums' bit is to ensure that old 'md5sums' files that may exist in that dir are initialised
I see, I did not think off that.....

I ran the command that 'mweed' gave and I got md5sums in all the subdirectories with files in (as expected) , and after reading your (bakdong) explanation I was looking for ways to make the command shorter, and removed

Code:
cat /dev/null > md5sums
if [[ $(file md5sums) == "md5sums: empty" ]] ; then rm -f md5sums; fi;
and I added the suffix "2nd" to "md5sums" so I can distinguish between md5sum files created from 'mweed' command and my modified command, it serve no other funtion (I cut and past it from command line and forgot to remove the suffix).
 
Old 04-15-2010, 10:48 PM   #10
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Original Poster
Rep: Reputation: 2
I am just testing my new signature.
 
Old 11-14-2012, 01:01 PM   #11
rahulrsheth
LQ Newbie
 
Registered: Nov 2012
Posts: 1

Rep: Reputation: Disabled
another way

find path/folder -type f -iname '*' -exec md5sum {} > checksums.txt \;
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Help with this please > openssl/md5.h: No such file or directory bigbadbo Linux - Software 1 03-23-2007 03:39 AM
Is MD5 Checksum enough? tededlin Linux - Newbie 2 08-02-2006 11:27 PM
md5 checksum CyberTroniX Linux - Newbie 3 01-16-2006 12:11 PM
Md5 Checksum dna9 Linux - Newbie 2 12-18-2004 11:11 PM
MD5 Checksum codedv Linux - Distributions 5 12-07-2003 08:28 AM


All times are GMT -5. The time now is 07:30 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration