LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
LinkBack Search this Thread
Old 10-06-2011, 11:11 AM   #1
Juako
Member
 
Registered: Mar 2010
Posts: 202

Rep: Reputation: 84
Bash script for old mail maintenance (Maildir)


Hi, here's a script i just got from the oven, to list/delete/move old mails from maildir stores. Can be used at any level of a mailstore or filesystem, and will recurse from there.

DON'T EVER USE THIS VERSION - READ BELOW
This version is insecure as it uses "eval" in a way that lefts it subject to be potentially exploited by mail users, I've left this version so the thread retains sense, and as example to you fellow scripters of things NOT to do and mindsets to be aware of (like mine at first ).

A refactored version is down the thread, with some other enhancements too like signal traps, better verbose mode, better mail detection and whatnot.


I still would like to hear advice on enhacements, bugs, or any other input you would like to provide. Thank you!

Code:
#!/bin/bash

# BASICS -----------------------------------------------------------------------
SCRIPT="${0##*/}"
SCRIPTDIR="$(readlink -f "${0%/*}")"
SCRIPTVER=1

# FUNCS ------------------------------------------------------------------------
Help() {
cat << HELP
$SCRIPT version $SCRIPTVER

This script lists, deletes o moves mails with ctime >= to the specified, inside
a path.

Usage:
   $SCRIPT -h
   $SCRIPT <path> <period n> -l
   $SCRIPT <path> <period n> -d [-n] [-v]
   $SCRIPT <path> <period n> -m <destination> [-n] [-v]

   period: -H (hours) -D (days) -W (weeks) -M (months) -Y (years)
   this option requires a numeric argument.

   -h (help) shows this screen
   -l (list) just list the mails with their dates (for parsing)
   -d (delete) deletes mails
   -m (move) moves mails to <destination> preserving folder structure
   -n (not-really) for -d/-m, dry-run mode
   -v (verbose) para -d/-m, shows extra informacion

   Some switches can be grouped (ej. -dnv)

   WARNING
   Actions -d and -m will run WITHOUT ASKING CONFIRMATION, RECURSIVELY into
   <path> !!! use carefully and test first with -n or -l.
HELP

    exit 0
}

Error() {
    exec >&2
    echo "$1"
    echo "For Help: \"$SCRIPT -h\""
    exit 1
}

# START ------------------------------------------------------------------------
shopt -s extglob

declare ORIGIN="" ACTION="" MOVE="" NOTREALLY="" VERBOSE="" PERIOD=""
declare -i NUMBER

# errors at 1st param
[[ "$1" == "-h" || "$2" == "-h" ]] && Help
[[ $# -lt 2 ]] && Error "nothing to do"
ORIGIN="$1"
[[ -z "$ORIGIN" || ! -d "$ORIGIN" ]] && Error "invalid directory: $ORIGIN"
shift

# the rest via getopts
OPTERR=0
while getopts “hldm:nvH:D:W:M:Y:” OPTION;  do
    case $OPTION in
      h) Help;;
      l) [[ -n "$ACTION" ]] && Error "-$OPTION and -$ACTION can't be used simultaneously"
        ACTION="l" ;;
      d) [[ -n "$ACTION" ]] && Error "-$OPTION and -$ACTION can't be used simultaneously"
        ACTION="d" ;;
      m) [[ -n "$ACTION" ]] && Error "-$OPTION and -$ACTION can't be used simultaneously"
        ACTION="m"
        MOVE="$OPTARG"
        [[ -z "$MOVE" || ! -d "$MOVE" ]] && Error "invalid directory for -m: $MOVE"
        ;;
      n) NOTREALLY=1 ;;
      v) VERBOSE=1 ;;
      H|D|W|M|Y) [[ -n "$PERIOD" ]] && Error "-$OPTION and -$IDX can't be used simultaneously"
        [[  -z "${OPTARG##*[!0-9]*}" ]] && Error "invalid number for -$OPTION: $NUMBER"
        NUMBER=$OPTARG
        while IFS="=" read -d" " IDX PERIOD; do
            [[ "$IDX" == "$OPTION" ]] && break 
        done <<< "H=hours D=days W=weeks M=months Y=years"
        ;;
      \?) Error "unrecognized option: $OPTION" ;;
    esac
done

# final checks that are better done after the loop
[[ -z "$ACTION" ]] && Error "nothing to do"
[[ -z "$PERIOD" ]] && Error "missing period parameter"
[[ -z "$NUMBER" || ! $NUMBER -gt 0 ]] && Error "invalid number for -$PERIOD: $NUMBER"

# could use -newerXY, but this is compatible with older finds
DATEREF="$(date -d "$NUMBER $PERIOD ago" +"%Y%m%d0000")"
touch -amt "$DATEREF" "$SCRIPTDIR/$DATEREF.$SCRIPT"
trap "trap - INT TERM EXIT; echo $SCRIPT: exiting... && rm $SCRIPTDIR/$DATEREF.$SCRIPT && exit 0" INT TERM EXIT

# get files, dates, filter mails with "file"
while IFS= read -d $'\0' -r LINE; do
    FILE=${LINE:8}
    DATE=${LINE:0:8}

    [[ "$( file "$FILE" )" =~ "^.* .* mail.*$" ]] && {
        case "$ACTION" in
            'l') echo "$FILE -- $DATE"
                continue 2
                ;;
            'd') ACTION_S="rm -f \"$FILE\""
                [[ $VERBOSE ]] && ACTION_S="rm -fv \"$FILE\""
                ;;
            'm') FILEDIR="$(readlink -f "${FILE%/*}")"
                ACTION_S="if [[ ! -f \"$MOVE$FILEDIR\" ]];then mkdir -p \"$MOVE$FILEDIR\"; fi; mv \"$FILE\" \"$MOVE$FILEDIR\""
                [[ $VERBOSE ]] && ACTION_S="if [[ ! -f \"$MOVE$FILEDIR\" ]];then mkdir -vp \"$MOVE$FILEDIR\"; fi; mv -v \"$FILE\" \"$MOVE$FILEDIR\""
                ;;
        esac

        if [[ $NOTREALLY ]]; then echo "$ACTION_S"
        else eval $ACTION_S
        fi
    }
done< <(find $ORIGIN -type f \! -cnewer "$SCRIPTDIR/$DATEREF.$SCRIPT" -printf "%AD%h/%f\0")

Last edited by Juako; 10-15-2011 at 08:05 PM. Reason: rewording
 
Old 10-06-2011, 12:11 PM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Looks like very clean code to me, at first glance.

I don't have time to look at it carefully right now, but I will make one suggestion.

Variables are designed for holding data, not code. Instead of eval'ng the $ACTION_S variable there at the end, try setting it up as a function, with all the various options you need defined in it. Then you can call that function with the appropriate arguments to execute the version you want.

http://mywiki.wooledge.org/BashFAQ/050
http://mywiki.wooledge.org/BashFAQ/048
 
1 members found this post helpful.
Old 10-06-2011, 12:59 PM   #3
Juako
Member
 
Registered: Mar 2010
Posts: 202

Original Poster
Rep: Reputation: 84
David, thanks for the review and suggestion.

I'm aware of the longstanding argument against eval and similar constructs, but tbh i never fully agreed that it is always evil to treat code as data and data as code, it gives me awesome freedom to manipulate code this way (might be this freedom the cause of the perceived problems).

In this case I was looking for a short/concise way to either display or execute dinamically generated code based on a flag. I made an assumption too: that this script would be run, or put into some cronjob, mostly by admins, and thus i wouldn't have to worry about untrusted input, which I perceive as one of the really difficult/dangerous situations regarding eval'd code.

Regardless, I'll try to make it based on functions, and see how I can reproduce the wanted behaviour. If you or anyone has suggestions on this task let me know. Thanks!

edit
*facepalms* Just now i read the links (not without my own anti-anti-eval prejudice), and heard of set +x !!! That looks like a better way to do it, not to mention a little bit faster than eval (that I have to admit, eval is slow, always!)

edit 2
Ok. Facepalm again. What about a user crafting a special IMAP folder name and getting into the eval'd code??? Oh man I don't even want to go testing it. The sole idea is enough for me to take your advice an refactor the eval. But this goes to God: WHY YOU GAVE US SUCH A POWERFUL TOOL IF WE CAN'T ALMOST NEVER USE IT WHYYY GOD!!!!!!!!

Last edited by Juako; 10-06-2011 at 06:54 PM.
 
Old 10-06-2011, 10:46 PM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Glad to have been of service.

Actually I do agree that eval is not always evil, and it does have its uses. But it's precisely because it's so powerful that you need to be very careful about how you apply it. Besides, modern shells have enough other features that there are almost always safer ways to accomplish what you want to do.

So my rule of thumb is...

1) If you can do it another way, then do it that way.
2) If you absolutely must use eval, then be very careful to ensure that it only runs on "clean", predictable code.

In other words, better safe than sorry.
 
Old 10-07-2011, 02:35 PM   #5
Juako
Member
 
Registered: Mar 2010
Posts: 202

Original Poster
Rep: Reputation: 84
Lightbulb version 2

changes:
  • byebye eval
  • switched from "file" to "fgrep" for mail detection, hope someone can test this and/or come up with enhancements
  • better verbose mode, plus match counter
  • other tiny corrections

things i would like to add someday:
  • more options for filtering mails
  • more robust mail detection
  • better cmdline parsing, don't push the current scheme to the limits cause it has them, mainly b/c of limits in getopts itself
  • more options for what to do with the mail
  • possibility to retain ALL c|m|a times if desired (ctime without changing system time, would it be possible?)

things i would like you to do:
  • test it
  • help me make it better
  • send me pizza (well I can dream, can't I)

Code:
#!/bin/bash

# BASICS -----------------------------------------------------------------------
SCRIPT="${0##*/}"
SCRIPTDIR="$(readlink -f "${0%/*}")"
SCRIPTVER=2

# FUNCS ------------------------------------------------------------------------
Help() {
cat << HELP
$SCRIPT version $SCRIPTVER

This script lists, deletes o moves mails with ctime >= to the specified, inside
a path.

Usage:
   $SCRIPT -h
   $SCRIPT <path> <period n> -l
   $SCRIPT <path> <period n> -d [-n] [-v]
   $SCRIPT <path> <period n> -m <destination> [-n] [-v]

   period: -H (hours) -D (days) -W (weeks) -M (months) -Y (years)
   this option requires a numeric argument.

   -h (help) shows this screen
   -l (list) just list the mails with their dates (for parsing)
   -d (delete) deletes mails
   -m (move) moves mails to <destination> preserving folder structure
   -n (not-really) for -d/-m, dry-run mode
   -v (verbose) para -d/-m, shows extra informacion

   Some switches can be grouped (ej. -dnv)

   WARNING
   Actions -d and -m will run WITHOUT ASKING CONFIRMATION, RECURSIVELY into
   <path> !!! use carefully and test first with -n or -l.
HELP

    exit 0
}

Error() {
    exec >&2
    echo "$1"
    echo "for Help: \"$SCRIPT -h\""
    exit 1
}

Exit() {
    [[ $VERBOSE ]] && {
        if (( FOUND > 0 )); then echo "$FOUND results found."
        else echo " no results were found."
        fi

        echo "$SCRIPT: exiting..."
    }

    [[ -f "$SCRIPTDIR/$DATEREF.$SCRIPT" ]] && rm -f "$SCRIPTDIR/$DATEREF.$SCRIPT"

    trap - INT TERM EXIT
    exit 0
}

Act_l() {
    echo "$FILE -- $DATE"
}

Act_d() {
    [[ $NOTREALLY ]] && {
        echo "rm -f $VERBOSE \"$FILE\""
        return 0
    }

    rm -f $VERBOSE "$FILE"
}

Act_m() {
    FILEDIR="$(readlink -f "${FILE%/*}")"

    [[ $NOTREALLY ]] && {
        echo "[[ ! -f \"$MOVE$FILEDIR\" ]] && mkdir -p $VERBOSE \"$MOVE$FILEDIR\""
        echo "mv $VERBOSE \"$FILE\" \"$MOVE$FILEDIR\""
        return 0
    }

    [[ ! -f "$MOVE$FILEDIR" ]] && mkdir -p $VERBOSE "$MOVE$FILEDIR"
    mv $VERBOSE "$FILE" "$MOVE$FILEDIR"
}

# START ------------------------------------------------------------------------
shopt -s extglob

declare ORIGIN="" ACTION="" MOVE="" NOTREALLY="" VERBOSE="" PERIOD=""
declare -i NUMBER FOUND=0

# errors at 1st param
[[ "$1" == "-h" || "$2" == "-h" ]] && Help
[[ $# -lt 2 ]] && Error "nothing to do"
ORIGIN="$1"
[[ ! "$ORIGIN" || ! -d "$ORIGIN" ]] && Error "invalid directory: $ORIGIN"
shift

# the rest via getopts
OPTERR=0
while getopts “hldm:nvH:D:W:M:Y:” OPTION;  do
    case $OPTION in
      h) Help ;;
      l|d) [[ "$ACTION" ]] && Error "can't use both -$OPTION and -$ACTION"
        [[ "$OPTARG" ]] && Error "extra arguments for $OPTION: $OPTARG"
        ACTION="$OPTION"
        ;;
      m) [[ "$ACTION" ]] && Error "can't use both -$OPTION and -$ACTION"
        [[ ! "$OPTARG" || ! -d "$OPTARG" ]] && Error "invalid directory for -$OPTION: $OPTARG"
        ACTION="$OPTION"
        MOVE="$OPTARG"
        ;;
      n) NOTREALLY=1 ;;
      v) VERBOSE="-v" ;;
      H|D|W|M|Y) [[ "$PERIOD" ]] && Error "can't use both -$OPTION and -$IDX"
        [[ ! "${OPTARG##*[!0-9]*}" ]] && Error "invalid number for -$OPTION: $NUMBER"
        NUMBER=$OPTARG
        while IFS="=" read -d" " IDX PERIOD; do
            [[ "$IDX" == "$OPTION" ]] && break
        done <<< "H=hours D=days W=weeks M=months Y=years"
        ;;
      \?) Error "unrecognized option: $OPTION" ;;
    esac
done

# final checks that are better done after the loop
[[ ! "$ACTION" ]] && Error "nothing to do"
[[ ! "$PERIOD" ]] && Error "missing period parameter"
! (( $NUMBER )) && Error "invalid number for -$PERIOD: $NUMBER"

# set traps
trap "Exit" INT TERM EXIT

# could use -newerXY, but this is compatible with older finds
DATEREF="$(date -d "$NUMBER $PERIOD ago" +"%Y%m%d0000")"
touch -amt "$DATEREF" "$SCRIPTDIR/$DATEREF.$SCRIPT"

# get files, dates, filter mails
[[ $VERBOSE ]] && echo -n "$SCRIPT: searching mail older than $NUMBER $PERIOD into $ORIGIN ..."
while IFS= read -d $'\0' -r LINE; do
    FILE="${LINE:8}"
    DATE="${LINE:0:8}"

    # this test won't quite work for all kinds of forms in which mail can appear
    # so I've ended up resorting to fgrep, which at least in my system could detect
    # all kinds of mail and also was faster than "file"
    #[[ "$(file -pbi "$FILE")" =~ "^.*message|html|mail|news.*$" ]]

    [[ "$(grep --mmap -FlIie "Message-Id:" -e "Subject:" "$FILE")" ]] && {
        [[ "$VERBOSE" ]] && ! (( FOUND++ )) && echo
        Act_$ACTION
    }
done< <(find "$ORIGIN" -type f \! -newer "$SCRIPTDIR/$DATEREF.$SCRIPT" -printf "%TD%h/%f\0")

Last edited by Juako; 10-17-2011 at 12:03 PM. Reason: retouched a few exprs
 
  


Reply

Tags
maildir, maintenance, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Can't get Postfix mail boxes from /var/mail to ~/Maildir phillinux Linux - Server 6 06-01-2010 09:48 AM
Error Retrieving Mail from Linux Mail Server: Unable to Scan $HOME/Maildir tmghendi General 0 06-15-2009 07:35 AM
Bash script for mail quotas Golgo13 Linux - Server 5 11-02-2008 04:07 AM
help: using cron + my bash script --> don't want mail from script beeblequix Linux - General 7 11-23-2007 09:25 PM
Import Mail from /var/spool/mail (sendmail) to a MailDir Format in Postfix shawnbishop Linux - Software 0 04-06-2006 10:44 AM


All times are GMT -5. The time now is 11:48 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration