[SOLVED] Older than $(date +"%m%d%Y" --date="14 days ago") script help

Habitual · 04-03-2013, 09:41 AM

Hello:

I have some homegrown AWS scripts that we use to manage AMIs and Snapshots.
Right now, the requirement is anything older than 14 days - de-register AMI and then delete associated snapshots. That part is all good, however, I need help with this function please

Code:

IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')

As you can see, I have a "$DEL_DATE" and I'd love to see grep "$DEL_DATE" or some other calculated field there, BUT I do not know how to perform the calculation on > "$DEL_DATE"
I am editing the grep by hand to accommodate the requirement presently.

and here's the entire script for grins and giggles.

Code:

#!/bin/bash
# AMN AWS Creds:
# Zone: WEST

# 14 days and older entity removal.
# DEL_DATE=$(date +"%m%d%Y" --date="14 days ago")
# DEL_DATE2=$(date +"%m%d%Y" --date="today")

export EC2_URL=https://ec2.us-west-1.amazonaws.com
export AWS_ACCESS_KEY=
export AWS_SECRET_KEY=
export EC2_HOME=/home/c9admin/ec2/ec2-api-tools/ec2-api-tools-1.6.6.3
export PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME=/usr
IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')
REPORT=/home/c9admin/amn_DELETED_DEREGd.lst
echo "" > "$REPORT"

DE_REGISTER()
{
for i in `echo "$IMAGES"`
	 do ec2-deregister $i >> "$REPORT"
done	
}

DEL_SNAPSHOTS()
{
for i in `echo "$SNAPSHOTS"`
	do ec2-delete-snapshot $i >> "$REPORT"
done
}

MAIL_RESULTS()
{
	mail -s "AMN Image and Snapshot Report" mail@domain.com < "$REPORT"
}


#safety net
if [ "$IMAGES" > 0 ]; then
	DE_REGISTER
	DEL_SNAPSHOTS
	MAIL_RESULTS
else 
	echo "No IMAGES to process" >> "$REPORT"
	MAIL_RESULTS
	exit 
fi
#EOF

Any Questions? I'd be happy to answer them!
Thank you all for your time.

TB0ne · 04-03-2013, 10:40 AM

Quote:

Originally Posted by Habitual

Hello:
I have some homegrown AWS scripts that we use to manage AMIs and Snapshots. Right now, the requirement is anything older than 14 days - de-register AMI and then delete associated snapshots. That part is all good, however, I need help with this function please

Code:

IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')

As you can see, I have a "$DEL_DATE" and I'd love to see grep "$DEL_DATE" or some other calculated field there, BUT I do not know how to perform the calculation on > "$DEL_DATE"
I am editing the grep by hand to accommodate the requirement presently.

Any Questions? I'd be happy to answer them!

What is the output of the "ec2-describe-images" command? And you may have thought of this already, so forgive a potentially stupid question, but since you have the $DEL_DATE, and you want anything older (i.e. LESS THAN) that number string (today, the DEL_DATE command returns 03202013), could you perhaps use awk or a different grep?

Code:

awk '$1 < $DEL_DATE'

or

Code:

grep -E -i "$DEL_DATE"

or remove the date-check from the ec2-describe images command. Clunky, but may work:

Code:

IMAGES=`ec2-describe-images`
SNAPSHOTS=`ec2-describe-images $IMAGES`

DE_REGISTER()
{
for i in `echo "$IMAGES"`
   # Statement here to get the date string into a number
   IMGDATE=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
   # Compare the numbers, since you have $DEL_DATE already
   if [ $IMGDATE -lt $DEL_DATE ]
      do ec2-deregister $i >> "$REPORT"
   fi
done	
}

DEL_SNAPSHOTS()
{
for i in `echo "$SNAPSHOTS"`
   # Same logic here
   SNAPDATE=$(ec2-describe-images $ | grep snap | awk '{print $4}')
   # Compare the numbers, since you have $DEL_DATE already
   if [ $SNAPDATE -lt $DEL_DATE ]
     do ec2-delete-snapshot $i >> "$REPORT"
   fi
done
}

Completely untested, and probably crap. Bash isn't a strong suit with me, and I don't know what the ec2 command is outputting to work with.

Habitual · 04-03-2013, 11:02 AM

TB0ne:
One output okay?

Code:

ec2-describe-images ami-d6604c93
IMAGE	ami-d6604c93	12digit_identifier/04032013_daxtra	12digit_identifier	available	private		x86_64	machine		ebs	paravirtual	xen
BLOCKDEVICEMAPPING	EBS	/dev/sda1		snap-18f14611	10	true	standard	
BLOCKDEVICEMAPPING	EBS	/dev/sdf		snap-1df14614	500	false	standard

I may very well have to use another awk or grep manipulation.
I am sure the answer is quite simple, I'm just not 'seeing' it today.

Thank you,

JJ

Edit: Making some progress with another method of logic...
Anything OVER 15 days to 3[01](ought to cover every month)...

Code:

for i in `seq 15 31` ; do date +"%m%d%Y" --date="$i days ago"; done

03192013
03182013
03172013
03162013
03152013
03142013
03132013
03122013
03112013
03102013
03092013
03082013
03072013
03062013
03052013
03042013
03032013

and then loop those through DE_REGISTER and DEL_SNAPSHOTS...

should work?

Habitual · 04-03-2013, 11:17 AM

Code:

OLD_DATE=$(for i in `seq 15 31` ; do date +"%m%d%Y" --date="$i days ago"; done)
[c9admin@zabbix ~]$ echo "$OLD_DATE"
03192013
03182013
03172013
03162013
03152013
03142013
03132013
03122013
03112013
03102013
03092013
03082013
03072013
03062013
03052013
03042013
03032013
[c9admin@zabbix ~]$ IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep "$OLD_DATE" | awk '{print $1}')
[c9admin@zabbix ~]$

seems to be one method...

Thanks!

TB0ne · 04-03-2013, 01:15 PM

See? I told you my bash skills weren't that great.

Good solution, thanks for sharing.

Habitual · 04-03-2013, 01:23 PM

Well, you're a great listener.

Thanks!

Habitual · 04-03-2013, 02:09 PM

Quote:

Originally Posted by TB0ne

Completely untested, and probably crap. Bash isn't a strong suit with me, and I don't know what the ec2 command is outputting to work with.

No worries.
Testing is my 'job'...
Working through some logic doesn't really require strength in bash, now does it?

I may have to come back to this post if my latest hack doesn't cut it.
Good Job, Well done.

Edit:
This is broken

Code:

if [ "$IMAGES" > 0 ]; then
	DE_REGISTER
	DEL_SNAPSHOTS
	MAIL_RESULTS
else 
	echo "No IMAGES to process" >> "$REPORT"
	MAIL_RESULTS
	exit 
fi

since "$IMAGES" contains strings, I can't do comparison math on it (other than count them...?) I think...

so now, I am favoring

Code:

if [ -n "$IMAGES" ]; then
	DE_REGISTER
	DEL_SNAPSHOTS
	MAIL_RESULTS
else
	echo "No IMAGES to process" >> "$REPORT"
	MAIL_RESULTS
	exit
fi

Still, it could always be "better".

David the H. · 04-03-2013, 05:50 PM

What is ec2-describe-images, a function or script? And what does it's output look like? I'm guessing it reads image metadata or something?

The usual way to get 'older than' files is to use find, but then that usually searches for the last modified time of the files. If you're reading metadata that would be different.

At the least, however, I suggest you take some time to learn how to properly use tools like awk. Chains of multiple cut|grep|sed|awk commands can almost always be reduced to a single instance of sed or awk.

(Examples from the OP)

Code:

# IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')

IMAGES=$( ec2-describe-images | awk '/031[1-8]2013/ { print $2 }' )
#I believe this is equivalent, it's hard to tell without seeing the input.

# SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')

SNAPSHOTS=$( ec2-describe-images $IMAGES | awk '/snap/ {print $4}' )

Here are a few useful sed references:
http://www.grymoire.com/Unix/Sed.html
http://sed.sourceforge.net/grabbag/
http://sed.sourceforge.net/sedfaq.html
http://sed.sourceforge.net/sed1line.txt
http://www.catonmat.net/series/sed-one-liners-explained

Here are a few useful awk references:
http://www.grymoire.com/Unix/Awk.html
http://www.gnu.org/software/gawk/man...ode/index.html
http://www.pement.org/awk/awk1line.txt
http://www.catonmat.net/series/awk-one-liners-explained

One more thing with the above, though. I assume that the IMAGES line results in the variable containing multiple entries. But it's almost never a good idea to store lists of related things in a single variable. As long as you're using bash or another advanced shell, you should use arrays instead. That's what they're designed for.

Code:

mapfile -t IMAGES < <( ec2-describe-images | awk '/031[1-8]2013/ { print $2 }' )

SNAPSHOTS=$( ec2-describe-images "${IMAGES[@]}" | awk '/snap/ {print $4}' )

Next point:

Code:

if [ "$IMAGES" > 0 ]; then
if [ -n "$IMAGES" ]; then

When using advanced shells like bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://mywiki.wooledge.org/BashFAQ/031
http://mywiki.wooledge.org/ArithmeticExpression

In addition, when inside square bracket tests, the symbols '>,<,=' and the like are string comparisons. You have to use '-gt,-lt,-eq' if you want to do numeric comparisons. Another reason why "((..))" is preferable here.

Code:

if (( IMAGES > 0 )); then
if [[ -n $IMAGES ]]; then

Finally, a bit of a meta-point. I see your date strings are in mmddyyyy format. I recently learned that the recommended form is to use the ISO-8601 international standard format of YYYY-MM-DD whenever possible.

If you can update your scripts and filenames to use the standard, it would let the shell to do natural sorting, and would probably make it easier to use globbing or other simple comparative techniques for getting the names you want.

If you don't want to go that way, you should try converting your date strings to epoch seconds (date +%s), and again you'll be able to use simple arithmetic to compare dates.

Habitual · 04-03-2013, 06:50 PM

David the H:

Quote:

ec2-describe-images...And what does it's output look like?

It's part of the Amazon API toolset and see http://www.linuxquestions.org/questi...4/#post4924373

I understand that I should use ISO-8601 format in my creation script. and

Code:

if [[ -n $IMAGES ]]; then

is preferred over:

Code:

if [ -n $IMAGES ]; then

wrt:

Quote:

I recently learned that the recommended form is to use the ISO-8601 international standard format of YYYY-MM-DD whenever possible.
If you can update your scripts and filenames to use the standard, it would let the shell to do natural sorting, and would probably make it easier to use globbing or other simple comparative techniques for getting the names you want.

"other simple comparative techniques". Ouch! Not! That's me.

I'm down with globbing I need to re-read up on it but I hear it's a 'good' thing, and sign me up for 'natural sorting'.

I most certainly can update those scripts since we authored 'em.
It is a simple assignment.

I understand your "chaining" reference.
When I get to "grep x | grep y | grep z" type stuff
I know it's time to write better code.

So I'm hoping this snippet suffices:

Thank you for your reply.
I appreciate your time, and skill.

Habitual · 04-04-2013, 10:08 AM

and now "this"
http://www.linuxquestions.org/questi...6/#post4925042

I'll save you some time

Code:

[[ $variable == 0 ]] or [[ -n $variable ]]

David the H. · 04-06-2013, 04:41 PM

Here's an example of what you can do if your filenames are in standard date order:

Create files for every day in march and april in YYYYmmdd format.
(Yes, it creates a march 31 too, but it's just an example.)

Code:

touch file_2013{03..04}{01..31}.txt

Get the date for 14 days ago in the same format (currently March 24th)

Code:

compdate=$( date -d '-14 days' '+%Y%m%d' )

Loop through the directory and print out every file that has a name earlier than the comparison date. Use a simple parameter substitution to remove all non-digit characters from the name before comparing.

Code:

for fname in *; do
    if (( ${fname//[^0-9]} < compdate )); then
        echo "$fname"
    fi
done

The output:

Code:

file_20130301.txt
file_20130302.txt
file_20130303.txt
file_20130304.txt
file_20130305.txt
file_20130306.txt
file_20130307.txt
file_20130308.txt
file_20130309.txt
file_20130310.txt
file_20130311.txt
file_20130312.txt
file_20130313.txt
file_20130314.txt
file_20130315.txt
file_20130316.txt
file_20130317.txt
file_20130318.txt
file_20130319.txt
file_20130320.txt
file_20130321.txt
file_20130322.txt
file_20130323.txt

Notice how you can use simple gt/lt comparisons on the dates when they're in the non-hyphenated format. (Actually, this would still work anyway, as the hyphens would be removed from the name as well during comparison).