[SOLVED] bash md5sum edit with sed?

Z038 · 01-05-2011, 06:40 PM

I am trying to get a checksum for a file in a subscripted variable in a bash script. md5sum outputs a checksum and the name of the input file. For example:

Code:

eval CSUM$K=\$"(md5sum file)"

This might return something like this:

Code:

3cff5d5c0113959d0be62be34b97e05c file

I want to assign just the checksum to the variable in my shell script and omit the file name that follows. Is there something besides md5sum that will generate a checksum? Or if not, then I was thinking I might be able to extract the checksum without the file name using sed.

Can someone show me how to do this?

jschiwal · 01-05-2011, 06:47 PM

Sed, Awk or cut could do it.

Code:

> md5cksum=($(md5sum Y* | cut -d' ' -f1))
> echo ${md5cksum[0]}
7f60a4e0b2a295f8e12a85cf2ef31c71
> echo ${md5cksum[1]}
8ad942f5605c6bf573cff17e84d23dcb
> echo ${md5cksum[2]}
a15672eab37e8a0d72bba89d117ac7d6

Z038 · 01-05-2011, 07:54 PM

Thank you very much jschiwal. Your example using cut solved my problem.

I've only just started coding in bash, and it's very confusing to me. When I tried this in my do loop initially based on your suggestion:

Code:

CSUM$K=($(md5sum file | cut -d' ' -f1)

I get this

Code:

./test.sh: line 37: syntax error near unexpected token `$(md5sum file
./test.sh: line 37: `    CSUM$K=($(md5sum file | cut -d' ' -f1))'

but this worked

Code:

eval CSUM$K=\$"(md5sum file | cut -d' ' -f1)"

Why do I need to use eval?

David1357 · 01-05-2011, 08:19 PM

Quote:

Originally Posted by Z038

Code:

CSUM$K=($(md5sum file | cut -d' ' -f1)

Remove the first "(" after the "=".

jschiwal · 01-05-2011, 08:42 PM

I don't follow what you want for "CSUM$K". In bash, it will first evaluate $K, add it to CSUM and try to execute the resulting string.
So,
K=A
CSUM$K

will try to run a program CSUMA.

The form
variable=(...)
Will initialize an array variable.
The form $( ... ) will execute the command inside the parenthesis and provide the output as arguments to a command.
It is the same as ` ... `

You access a bash array like:
${variable[index]}
the first element is 0.

You can count the number of elements.
$#{variable[@]}

Code:

md5sum *.pdf >pdf.md5
sums=($(cut -d' ' -f1 pdf.md5))
names=($(cut -d' ' -f3 pdf.md5 ))
echo ${sums[5]}
cbfff584cb597dcc152555736ac3f008
echo ${names[5]}

You could iterate through the elements in a loop like this:

Code:

for ((i=0,i<${#sums[*]};i++)); do
...
...
done

bigearsbilly · 01-06-2011, 04:04 AM

alternatives without using cut

Code:

md5sum $1 |
while read sum file ; do
    echo sum = $sum
done

(
set $(md5sum $1)
    echo sum = $1
)

# if you use a decent shell (ksh) this would work:
md5sum $1 | read sum file
echo sum = $sum

Code:

$ bash 1.sh  demo959.tar.gz  
sum = 34190a633d15d7b9a3d2427a3d82b960
sum = 34190a633d15d7b9a3d2427a3d82b960
sum =

using korn shell much better
$ ksh 1.sh  demo959.tar.gz   
sum = 34190a633d15d7b9a3d2427a3d82b960
sum = 34190a633d15d7b9a3d2427a3d82b960
sum = 34190a633d15d7b9a3d2427a3d82b960

grail · 01-06-2011, 04:15 AM

Or just use substitution:

Code:

out=$(md5sum file)

echo ${out% *}

crts · 01-06-2011, 08:02 AM

Quote:

Originally Posted by grail

Or just use substitution:

Code:

out=$(md5sum file)

echo ${out% *}

Hi,

I would also opt for the substitution. However, there should be two '%' since one '%' will only match the shortest string from the end. Problematic if the filename contains spaces.

Code:

echo ${out%% *}

grail · 01-06-2011, 08:37 AM

@crts - fair call ... I tend to forget how people from the dark side love their space filled naming conventions

Z038 · 01-06-2011, 11:54 PM

Quote:

Originally Posted by jschiwal

I don't follow what you want for "CSUM$K". In bash, it will first evaluate $K, add it to CSUM and try to execute the resulting string.
So,
K=A
CSUM$K

will try to run a program CSUMA.

The form
variable=(...)
Will initialize an array variable.
The form $( ... ) will execute the command inside the parenthesis and provide the output as arguments to a command.
It is the same as ` ... `

You access a bash array like:
${variable[index]}
the first element is 0.

You can count the number of elements.
$#{variable[@]}

Code:

md5sum *.pdf >pdf.md5
sums=($(cut -d' ' -f1 pdf.md5))
names=($(cut -d' ' -f3 pdf.md5 ))
echo ${sums[5]}
cbfff584cb597dcc152555736ac3f008
echo ${names[5]}

You could iterate through the elements in a loop like this:

Code:

for ((i=0,i<${#sums[*]};i++)); do
...
...
done

$K is the control variable in a do loop and it gets incremented by one on each iteration. I'm just trying to assign the md5sum of a file to a new variable each time through the loop. I don't want the variable to contain the file name, just the checksum. The cut command you demonstrated takes care of extracting the checksum without the file name.

The variable I am creating is of the form CSUM$K where $K goes from 1 to x. So I end up with variables like CSUM0 the first time through the loop, then CSUM1 the next time, then CSUM2, CSUM3, ... CSUMx. The value of each one of those variables is a different checksum.

I gather from what you are showing me that what I am doing is not the same as an array. Maybe an array would be easier, but the contents of the file I'm getting the checksum for changes each time through the loop.

I'm going to be trying your examples and some suggestions of others in this thread.

grail · 01-07-2011, 01:46 AM

Yes using your $K idea as the index to an array would make a lot more sense

Quote:

but the contents of the file I'm getting the checksum for changes each time through the loop.

You might need to explain this some more and maybe how you intend to use the checksums if the data keeps changing??

Z038 · 01-07-2011, 09:43 PM

I didn't express that well. I'm monitoring web site pages for changes. I wget several different web pages whose addresses are given by variables SITE0, SITE1, SITE2, etc, and save the initial checksum for each page in a variable like INIT0, INIT1, INIT2, etc. Then I go into a loop and issue another wget for each page, one page per loop iteration. Each time through the loop, I wget the corresponding SITE$K and save the md5sum checksum to CSUM$K, compare it with INIT$K, and send an email if checksum has changed and then assign the new CSUM$K to INIT$K. I then sleep for a while and then loop and check the next SITE$K.

Code:

#!/bin/bash
SITE0=http://www.website0.com/
SITE1=http://www.website1.com/
SITE2=http://www.website2.com/
SITE3=http://www.website3.com/
SITE4=http://www.website4.com/
K=0
while [ $K -lt 5 ]; do
   eval wget \$'SITE'$K --no-cookies -O initial
   eval INIT$K=\$"(md5sum initial | cut -d' ' -f1)"
   ((K++))
done
K=0
while [ $K -lt 5 ]; do
    eval wget \$'SITE'$K --no-cookies -O subsequent
    eval CSUM$K=\$"(md5sum subsequent | cut -d' ' -f1)"
    if [ "$(eval echo \$'INIT'$K)" != "$(eval echo \$'CSUM'$K)" ]; then
       eval echo -e change detected for \$'SITE'$K | mailx -s "SITE was updated" me@wherever.com
       eval INIT$K=CSUM$K
    fi
    sleep 10
    if [ $K == 4 ]; then 
       K=0
    else
       ((K++))
    fi
done

I'm going to try to rewrite this so I can read a list of sites from a file, set my loop counter to the number of records in the file, and then check each site on each iteration of the loop so that I can monitor an arbitrary number of sites on a regular interval. I will try to use arrays to hold the site names and checksums.

I know that monitoring web sites this way will result in a lot of emails, so this may not be an especially practical thing to do, but it is just an exercise to help me learn some techniques. It's a little more interesting than Hello World

grail · 01-08-2011, 01:05 AM

So untested of course seeing as sites are examples, but maybe something like this could work (I assumed it was to run forever, hence while true):

Code:

#!/bin/bash

SITES=( http://www.website0.com/
        http://www.website1.com/
        http://www.website2.com/
        http://www.website3.com/
        http://www.website4.com/ )

for SITE in ${SITES[*]}
do
    INIT+=( $(wget -O - $SITE --no-cookies | md5sum | cut -d' ' -f1) )
done

while true
do
    sleep 10

    for SITE in ${SITES[*]}
    do
        CSUM+=( $(wget -O - $SITE --no-cookies | md5sum | cut -d' ' -f1) )
    done

    for K in ${!SITES[*]}
    do
        if [[ ${INIT[K]} != ${CSUM[K]} ]]
        then
            echo -e change detected for ${SITE[K]} | mailx -s "SITE was updated" me@wherever.com
            ${INIT[K]}=${CSUM[K]}
        fi  
    done
done

Z038 · 01-08-2011, 01:54 AM

That looks much neater than mine, and would not require making changes to hard coded loop values as sites are added. I like it. Thank you. Examples like this are very useful to me because I can relate what you did to my crude efforts in order to understand how it works.

Now if I can find a bash reference or guide that is easier to read and search than this one - http://www.gnu.org/software/bash/man...ef.html#Arrays - I'll be set.

grail · 01-08-2011, 03:32 AM

Well let us see if we can help with that too

I like / use the following:

http://tldp.org/LDP/abs/html/
http://mywiki.wooledge.org/TitleIndex (this one is a great reference for common mistakes that people (and even some sites) make)

This is another I have seen recommended:

http://www.gnu.org/software/bash/manual/bashref.html

One other piece of advice I would give to the code above as you mentioned the list of sites will probably grow is to place them all in a separate file and reference it in the script,
like so:
Plain file for sites: (I'll call it sites)

Code:

http://www.website0.com/
http://www.website1.com/
http://www.website2.com/
http://www.website3.com/
http://www.website4.com/

Change to original code above:

Code:

SITES=( http://www.website0.com/
        http://www.website1.com/
        http://www.website2.com/
        http://www.website3.com/
        http://www.website4.com/ )

# becomes

SITES=( $(< sites) )

Now you only ever need to edit the sites file and the code never needs to be touched