Bash script for old mail maintenance (Maildir)

Juako · 10-06-2011, 11:11 AM

Hi, here's a script i just got from the oven, to list/delete/move old mails from maildir stores. Can be used at any level of a mailstore or filesystem, and will recurse from there.

DON'T EVER USE THIS VERSION - READ BELOW
This version is insecure as it uses "eval" in a way that lefts it subject to be potentially exploited by mail users, I've left this version so the thread retains sense, and as example to you fellow scripters of things NOT to do and mindsets to be aware of (like mine at first ).

A refactored version is down the thread, with some other enhancements too like signal traps, better verbose mode, better mail detection and whatnot.

I still would like to hear advice on enhacements, bugs, or any other input you would like to provide. Thank you!

Code:

#!/bin/bash

# BASICS -----------------------------------------------------------------------
SCRIPT="${0##*/}"
SCRIPTDIR="$(readlink -f "${0%/*}")"
SCRIPTVER=1

# FUNCS ------------------------------------------------------------------------
Help() {
cat << HELP
$SCRIPT version $SCRIPTVER

This script lists, deletes o moves mails with ctime >= to the specified, inside
a path.

Usage:
   $SCRIPT -h
   $SCRIPT <path> <period n> -l
   $SCRIPT <path> <period n> -d [-n] [-v]
   $SCRIPT <path> <period n> -m <destination> [-n] [-v]

   period: -H (hours) -D (days) -W (weeks) -M (months) -Y (years)
   this option requires a numeric argument.

   -h (help) shows this screen
   -l (list) just list the mails with their dates (for parsing)
   -d (delete) deletes mails
   -m (move) moves mails to <destination> preserving folder structure
   -n (not-really) for -d/-m, dry-run mode
   -v (verbose) para -d/-m, shows extra informacion

   Some switches can be grouped (ej. -dnv)

   WARNING
   Actions -d and -m will run WITHOUT ASKING CONFIRMATION, RECURSIVELY into
   <path> !!! use carefully and test first with -n or -l.
HELP

    exit 0
}

Error() {
    exec >&2
    echo "$1"
    echo "For Help: \"$SCRIPT -h\""
    exit 1
}

# START ------------------------------------------------------------------------
shopt -s extglob

declare ORIGIN="" ACTION="" MOVE="" NOTREALLY="" VERBOSE="" PERIOD=""
declare -i NUMBER

# errors at 1st param
[[ "$1" == "-h" || "$2" == "-h" ]] && Help
[[ $# -lt 2 ]] && Error "nothing to do"
ORIGIN="$1"
[[ -z "$ORIGIN" || ! -d "$ORIGIN" ]] && Error "invalid directory: $ORIGIN"
shift

# the rest via getopts
OPTERR=0
while getopts “hldm:nvH:D:W:M:Y:” OPTION;  do
    case $OPTION in
      h) Help;;
      l) [[ -n "$ACTION" ]] && Error "-$OPTION and -$ACTION can't be used simultaneously"
        ACTION="l" ;;
      d) [[ -n "$ACTION" ]] && Error "-$OPTION and -$ACTION can't be used simultaneously"
        ACTION="d" ;;
      m) [[ -n "$ACTION" ]] && Error "-$OPTION and -$ACTION can't be used simultaneously"
        ACTION="m"
        MOVE="$OPTARG"
        [[ -z "$MOVE" || ! -d "$MOVE" ]] && Error "invalid directory for -m: $MOVE"
        ;;
      n) NOTREALLY=1 ;;
      v) VERBOSE=1 ;;
      H|D|W|M|Y) [[ -n "$PERIOD" ]] && Error "-$OPTION and -$IDX can't be used simultaneously"
        [[  -z "${OPTARG##*[!0-9]*}" ]] && Error "invalid number for -$OPTION: $NUMBER"
        NUMBER=$OPTARG
        while IFS="=" read -d" " IDX PERIOD; do
            [[ "$IDX" == "$OPTION" ]] && break 
        done <<< "H=hours D=days W=weeks M=months Y=years"
        ;;
      \?) Error "unrecognized option: $OPTION" ;;
    esac
done

# final checks that are better done after the loop
[[ -z "$ACTION" ]] && Error "nothing to do"
[[ -z "$PERIOD" ]] && Error "missing period parameter"
[[ -z "$NUMBER" || ! $NUMBER -gt 0 ]] && Error "invalid number for -$PERIOD: $NUMBER"

# could use -newerXY, but this is compatible with older finds
DATEREF="$(date -d "$NUMBER $PERIOD ago" +"%Y%m%d0000")"
touch -amt "$DATEREF" "$SCRIPTDIR/$DATEREF.$SCRIPT"
trap "trap - INT TERM EXIT; echo $SCRIPT: exiting... && rm $SCRIPTDIR/$DATEREF.$SCRIPT && exit 0" INT TERM EXIT

# get files, dates, filter mails with "file"
while IFS= read -d $'\0' -r LINE; do
    FILE=${LINE:8}
    DATE=${LINE:0:8}

    [[ "$( file "$FILE" )" =~ "^.* .* mail.*$" ]] && {
        case "$ACTION" in
            'l') echo "$FILE -- $DATE"
                continue 2
                ;;
            'd') ACTION_S="rm -f \"$FILE\""
                [[ $VERBOSE ]] && ACTION_S="rm -fv \"$FILE\""
                ;;
            'm') FILEDIR="$(readlink -f "${FILE%/*}")"
                ACTION_S="if [[ ! -f \"$MOVE$FILEDIR\" ]];then mkdir -p \"$MOVE$FILEDIR\"; fi; mv \"$FILE\" \"$MOVE$FILEDIR\""
                [[ $VERBOSE ]] && ACTION_S="if [[ ! -f \"$MOVE$FILEDIR\" ]];then mkdir -vp \"$MOVE$FILEDIR\"; fi; mv -v \"$FILE\" \"$MOVE$FILEDIR\""
                ;;
        esac

        if [[ $NOTREALLY ]]; then echo "$ACTION_S"
        else eval $ACTION_S
        fi
    }
done< <(find $ORIGIN -type f \! -cnewer "$SCRIPTDIR/$DATEREF.$SCRIPT" -printf "%AD%h/%f\0")

David the H. · 10-06-2011, 12:11 PM

Looks like very clean code to me, at first glance.

I don't have time to look at it carefully right now, but I will make one suggestion.

Variables are designed for holding data, not code. Instead of eval'ng the $ACTION_S variable there at the end, try setting it up as a function, with all the various options you need defined in it. Then you can call that function with the appropriate arguments to execute the version you want.

http://mywiki.wooledge.org/BashFAQ/050
http://mywiki.wooledge.org/BashFAQ/048

Juako · 10-06-2011, 12:59 PM

David, thanks for the review and suggestion.

I'm aware of the longstanding argument against eval and similar constructs, but tbh i never fully agreed that it is always evil to treat code as data and data as code, it gives me awesome freedom to manipulate code this way (might be this freedom the cause of the perceived problems).

In this case I was looking for a short/concise way to either display or execute dinamically generated code based on a flag. I made an assumption too: that this script would be run, or put into some cronjob, mostly by admins, and thus i wouldn't have to worry about untrusted input, which I perceive as one of the really difficult/dangerous situations regarding eval'd code.

Regardless, I'll try to make it based on functions, and see how I can reproduce the wanted behaviour. If you or anyone has suggestions on this task let me know. Thanks!

edit
*facepalms* Just now i read the links (not without my own anti-anti-eval prejudice), and heard of set +x !!! That looks like a better way to do it, not to mention a little bit faster than eval (that I have to admit, eval is slow, always!)

edit 2
Ok. Facepalm again. What about a user crafting a special IMAP folder name and getting into the eval'd code??? Oh man I don't even want to go testing it. The sole idea is enough for me to take your advice an refactor the eval. But this goes to God: WHY YOU GAVE US SUCH A POWERFUL TOOL IF WE CAN'T ALMOST NEVER USE IT WHYYY GOD!!!!!!!!

David the H. · 10-06-2011, 10:46 PM

Glad to have been of service.

Actually I do agree that eval is not always evil, and it does have its uses. But it's precisely because it's so powerful that you need to be very careful about how you apply it. Besides, modern shells have enough other features that there are almost always safer ways to accomplish what you want to do.

So my rule of thumb is...

1) If you can do it another way, then do it that way.
2) If you absolutely must use eval, then be very careful to ensure that it only runs on "clean", predictable code.

In other words, better safe than sorry.

Juako · 10-07-2011, 02:35 PM

changes:

byebye eval
switched from "file" to "fgrep" for mail detection, hope someone can test this and/or come up with enhancements
better verbose mode, plus match counter
other tiny corrections

things i would like to add someday:

more options for filtering mails
more robust mail detection
better cmdline parsing, don't push the current scheme to the limits cause it has them, mainly b/c of limits in getopts itself
more options for what to do with the mail
possibility to retain ALL c|m|a times if desired (ctime without changing system time, would it be possible?)

things i would like you to do:

test it
help me make it better
send me pizza (well I can dream, can't I)

Code:

#!/bin/bash

# BASICS -----------------------------------------------------------------------
SCRIPT="${0##*/}"
SCRIPTDIR="$(readlink -f "${0%/*}")"
SCRIPTVER=2

# FUNCS ------------------------------------------------------------------------
Help() {
cat << HELP
$SCRIPT version $SCRIPTVER

This script lists, deletes o moves mails with ctime >= to the specified, inside
a path.

Usage:
   $SCRIPT -h
   $SCRIPT <path> <period n> -l
   $SCRIPT <path> <period n> -d [-n] [-v]
   $SCRIPT <path> <period n> -m <destination> [-n] [-v]

   period: -H (hours) -D (days) -W (weeks) -M (months) -Y (years)
   this option requires a numeric argument.

   -h (help) shows this screen
   -l (list) just list the mails with their dates (for parsing)
   -d (delete) deletes mails
   -m (move) moves mails to <destination> preserving folder structure
   -n (not-really) for -d/-m, dry-run mode
   -v (verbose) para -d/-m, shows extra informacion

   Some switches can be grouped (ej. -dnv)

   WARNING
   Actions -d and -m will run WITHOUT ASKING CONFIRMATION, RECURSIVELY into
   <path> !!! use carefully and test first with -n or -l.
HELP

    exit 0
}

Error() {
    exec >&2
    echo "$1"
    echo "for Help: \"$SCRIPT -h\""
    exit 1
}

Exit() {
    [[ $VERBOSE ]] && {
        if (( FOUND > 0 )); then echo "$FOUND results found."
        else echo " no results were found."
        fi

        echo "$SCRIPT: exiting..."
    }

    [[ -f "$SCRIPTDIR/$DATEREF.$SCRIPT" ]] && rm -f "$SCRIPTDIR/$DATEREF.$SCRIPT"

    trap - INT TERM EXIT
    exit 0
}

Act_l() {
    echo "$FILE -- $DATE"
}

Act_d() {
    [[ $NOTREALLY ]] && {
        echo "rm -f $VERBOSE \"$FILE\""
        return 0
    }

    rm -f $VERBOSE "$FILE"
}

Act_m() {
    FILEDIR="$(readlink -f "${FILE%/*}")"

    [[ $NOTREALLY ]] && {
        echo "[[ ! -f \"$MOVE$FILEDIR\" ]] && mkdir -p $VERBOSE \"$MOVE$FILEDIR\""
        echo "mv $VERBOSE \"$FILE\" \"$MOVE$FILEDIR\""
        return 0
    }

    [[ ! -f "$MOVE$FILEDIR" ]] && mkdir -p $VERBOSE "$MOVE$FILEDIR"
    mv $VERBOSE "$FILE" "$MOVE$FILEDIR"
}

# START ------------------------------------------------------------------------
shopt -s extglob

declare ORIGIN="" ACTION="" MOVE="" NOTREALLY="" VERBOSE="" PERIOD=""
declare -i NUMBER FOUND=0

# errors at 1st param
[[ "$1" == "-h" || "$2" == "-h" ]] && Help
[[ $# -lt 2 ]] && Error "nothing to do"
ORIGIN="$1"
[[ ! "$ORIGIN" || ! -d "$ORIGIN" ]] && Error "invalid directory: $ORIGIN"
shift

# the rest via getopts
OPTERR=0
while getopts “hldm:nvH:D:W:M:Y:” OPTION;  do
    case $OPTION in
      h) Help ;;
      l|d) [[ "$ACTION" ]] && Error "can't use both -$OPTION and -$ACTION"
        [[ "$OPTARG" ]] && Error "extra arguments for $OPTION: $OPTARG"
        ACTION="$OPTION"
        ;;
      m) [[ "$ACTION" ]] && Error "can't use both -$OPTION and -$ACTION"
        [[ ! "$OPTARG" || ! -d "$OPTARG" ]] && Error "invalid directory for -$OPTION: $OPTARG"
        ACTION="$OPTION"
        MOVE="$OPTARG"
        ;;
      n) NOTREALLY=1 ;;
      v) VERBOSE="-v" ;;
      H|D|W|M|Y) [[ "$PERIOD" ]] && Error "can't use both -$OPTION and -$IDX"
        [[ ! "${OPTARG##*[!0-9]*}" ]] && Error "invalid number for -$OPTION: $NUMBER"
        NUMBER=$OPTARG
        while IFS="=" read -d" " IDX PERIOD; do
            [[ "$IDX" == "$OPTION" ]] && break
        done <<< "H=hours D=days W=weeks M=months Y=years"
        ;;
      \?) Error "unrecognized option: $OPTION" ;;
    esac
done

# final checks that are better done after the loop
[[ ! "$ACTION" ]] && Error "nothing to do"
[[ ! "$PERIOD" ]] && Error "missing period parameter"
! (( $NUMBER )) && Error "invalid number for -$PERIOD: $NUMBER"

# set traps
trap "Exit" INT TERM EXIT

# could use -newerXY, but this is compatible with older finds
DATEREF="$(date -d "$NUMBER $PERIOD ago" +"%Y%m%d0000")"
touch -amt "$DATEREF" "$SCRIPTDIR/$DATEREF.$SCRIPT"

# get files, dates, filter mails
[[ $VERBOSE ]] && echo -n "$SCRIPT: searching mail older than $NUMBER $PERIOD into $ORIGIN ..."
while IFS= read -d $'\0' -r LINE; do
    FILE="${LINE:8}"
    DATE="${LINE:0:8}"

    # this test won't quite work for all kinds of forms in which mail can appear
    # so I've ended up resorting to fgrep, which at least in my system could detect
    # all kinds of mail and also was faster than "file"
    #[[ "$(file -pbi "$FILE")" =~ "^.*message|html|mail|news.*$" ]]

    [[ "$(grep --mmap -FlIie "Message-Id:" -e "Subject:" "$FILE")" ]] && {
        [[ "$VERBOSE" ]] && ! (( FOUND++ )) && echo
        Act_$ACTION
    }
done< <(find "$ORIGIN" -type f \! -newer "$SCRIPTDIR/$DATEREF.$SCRIPT" -printf "%TD%h/%f\0")