LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 04-03-2013, 09:41 AM   #1
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Older than $(date +"%m%d%Y" --date="14 days ago") script help


Hello:

I have some homegrown AWS scripts that we use to manage AMIs and Snapshots.
Right now, the requirement is anything older than 14 days - de-register AMI and then delete associated snapshots. That part is all good, however, I need help with this function please

Code:
IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')
As you can see, I have a "$DEL_DATE" and I'd love to see grep "$DEL_DATE" or some other calculated field there, BUT I do not know how to perform the calculation on > "$DEL_DATE"
I am editing the grep by hand to accommodate the requirement presently.

and here's the entire script for grins and giggles.
Code:
#!/bin/bash
# AMN AWS Creds:
# Zone: WEST

# 14 days and older entity removal.
# DEL_DATE=$(date +"%m%d%Y" --date="14 days ago")
# DEL_DATE2=$(date +"%m%d%Y" --date="today")

export EC2_URL=https://ec2.us-west-1.amazonaws.com
export AWS_ACCESS_KEY=
export AWS_SECRET_KEY=
export EC2_HOME=/home/c9admin/ec2/ec2-api-tools/ec2-api-tools-1.6.6.3
export PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME=/usr
IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')
REPORT=/home/c9admin/amn_DELETED_DEREGd.lst
echo "" > "$REPORT"

DE_REGISTER()
{
for i in `echo "$IMAGES"`
	 do ec2-deregister $i >> "$REPORT"
done	
}

DEL_SNAPSHOTS()
{
for i in `echo "$SNAPSHOTS"`
	do ec2-delete-snapshot $i >> "$REPORT"
done
}

MAIL_RESULTS()
{
	mail -s "AMN Image and Snapshot Report" mail@domain.com < "$REPORT"
}


#safety net
if [ "$IMAGES" > 0 ]; then
	DE_REGISTER
	DEL_SNAPSHOTS
	MAIL_RESULTS
else 
	echo "No IMAGES to process" >> "$REPORT"
	MAIL_RESULTS
	exit 
fi
#EOF
Any Questions? I'd be happy to answer them!
Thank you all for your time.

Last edited by Habitual; 04-03-2013 at 10:12 AM. Reason: terminated string '"14 days ago'
 
Old 04-03-2013, 10:40 AM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 21,959

Rep: Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827
Quote:
Originally Posted by Habitual View Post
Hello:
I have some homegrown AWS scripts that we use to manage AMIs and Snapshots. Right now, the requirement is anything older than 14 days - de-register AMI and then delete associated snapshots. That part is all good, however, I need help with this function please

Code:
IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')
As you can see, I have a "$DEL_DATE" and I'd love to see grep "$DEL_DATE" or some other calculated field there, BUT I do not know how to perform the calculation on > "$DEL_DATE"
I am editing the grep by hand to accommodate the requirement presently.

Any Questions? I'd be happy to answer them!
What is the output of the "ec2-describe-images" command? And you may have thought of this already, so forgive a potentially stupid question, but since you have the $DEL_DATE, and you want anything older (i.e. LESS THAN) that number string (today, the DEL_DATE command returns 03202013), could you perhaps use awk or a different grep?
Code:
awk '$1 < $DEL_DATE'
or
Code:
grep -E -i "$DEL_DATE"
or remove the date-check from the ec2-describe images command. Clunky, but may work:
Code:
IMAGES=`ec2-describe-images`
SNAPSHOTS=`ec2-describe-images $IMAGES`

DE_REGISTER()
{
for i in `echo "$IMAGES"`
   # Statement here to get the date string into a number
   IMGDATE=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')
   # Compare the numbers, since you have $DEL_DATE already
   if [ $IMGDATE -lt $DEL_DATE ]
      do ec2-deregister $i >> "$REPORT"
   fi
done	
}

DEL_SNAPSHOTS()
{
for i in `echo "$SNAPSHOTS"`
   # Same logic here
   SNAPDATE=$(ec2-describe-images $ | grep snap | awk '{print $4}')
   # Compare the numbers, since you have $DEL_DATE already
   if [ $SNAPDATE -lt $DEL_DATE ]
     do ec2-delete-snapshot $i >> "$REPORT"
   fi
done
}
Completely untested, and probably crap. Bash isn't a strong suit with me, and I don't know what the ec2 command is outputting to work with.
 
Old 04-03-2013, 11:02 AM   #3
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
TB0ne:
One output okay?
Code:
ec2-describe-images ami-d6604c93
IMAGE	ami-d6604c93	12digit_identifier/04032013_daxtra	12digit_identifier	available	private		x86_64	machine		ebs	paravirtual	xen
BLOCKDEVICEMAPPING	EBS	/dev/sda1		snap-18f14611	10	true	standard	
BLOCKDEVICEMAPPING	EBS	/dev/sdf		snap-1df14614	500	false	standard
I may very well have to use another awk or grep manipulation.
I am sure the answer is quite simple, I'm just not 'seeing' it today.

Thank you,

JJ

Edit: Making some progress with another method of logic...
Anything OVER 15 days to 3[01](ought to cover every month)...

Code:
for i in `seq 15 31` ; do date +"%m%d%Y" --date="$i days ago"; done
03192013
03182013
03172013
03162013
03152013
03142013
03132013
03122013
03112013
03102013
03092013
03082013
03072013
03062013
03052013
03042013
03032013

and then loop those through DE_REGISTER and DEL_SNAPSHOTS...

should work?

Last edited by Habitual; 04-03-2013 at 11:11 AM.
 
Old 04-03-2013, 11:17 AM   #4
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
Code:
OLD_DATE=$(for i in `seq 15 31` ; do date +"%m%d%Y" --date="$i days ago"; done)
[c9admin@zabbix ~]$ echo "$OLD_DATE"
03192013
03182013
03172013
03162013
03152013
03142013
03132013
03122013
03112013
03102013
03092013
03082013
03072013
03062013
03052013
03042013
03032013
[c9admin@zabbix ~]$ IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep "$OLD_DATE" | awk '{print $1}')
[c9admin@zabbix ~]$
seems to be one method...

Thanks!
 
Old 04-03-2013, 01:15 PM   #5
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 21,959

Rep: Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827Reputation: 5827
See? I told you my bash skills weren't that great. Good solution, thanks for sharing.
 
Old 04-03-2013, 01:23 PM   #6
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
Well, you're a great listener.

Thanks!
 
Old 04-03-2013, 02:09 PM   #7
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by TB0ne View Post
Completely untested, and probably crap. Bash isn't a strong suit with me, and I don't know what the ec2 command is outputting to work with.
No worries.
Testing is my 'job'...
Working through some logic doesn't really require strength in bash, now does it?

I may have to come back to this post if my latest hack doesn't cut it.
Good Job, Well done.

Edit:
This is broken
Code:
if [ "$IMAGES" > 0 ]; then
	DE_REGISTER
	DEL_SNAPSHOTS
	MAIL_RESULTS
else 
	echo "No IMAGES to process" >> "$REPORT"
	MAIL_RESULTS
	exit 
fi
since "$IMAGES" contains strings, I can't do comparison math on it (other than count them...?) I think...

so now, I am favoring
Code:
if [ -n "$IMAGES" ]; then
	DE_REGISTER
	DEL_SNAPSHOTS
	MAIL_RESULTS
else
	echo "No IMAGES to process" >> "$REPORT"
	MAIL_RESULTS
	exit
fi
Still, it could always be "better".

Last edited by Habitual; 04-03-2013 at 04:51 PM.
 
Old 04-03-2013, 05:50 PM   #8
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,842

Rep: Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004
What is ec2-describe-images, a function or script? And what does it's output look like? I'm guessing it reads image metadata or something?

The usual way to get 'older than' files is to use find, but then that usually searches for the last modified time of the files. If you're reading metadata that would be different.

At the least, however, I suggest you take some time to learn how to properly use tools like awk. Chains of multiple cut|grep|sed|awk commands can almost always be reduced to a single instance of sed or awk.

(Examples from the OP)
Code:
# IMAGES=$(ec2-describe-images | awk '{print $2 " "$3}' | grep 031[1-8]2013 | awk '{print $1}')

IMAGES=$( ec2-describe-images | awk '/031[1-8]2013/ { print $2 }' )
#I believe this is equivalent, it's hard to tell without seeing the input.

# SNAPSHOTS=$(ec2-describe-images $IMAGES | grep snap | awk '{print $4}')

SNAPSHOTS=$( ec2-describe-images $IMAGES | awk '/snap/ {print $4}' )
Here are a few useful sed references:
http://www.grymoire.com/Unix/Sed.html
http://sed.sourceforge.net/grabbag/
http://sed.sourceforge.net/sedfaq.html
http://sed.sourceforge.net/sed1line.txt
http://www.catonmat.net/series/sed-one-liners-explained

Here are a few useful awk references:
http://www.grymoire.com/Unix/Awk.html
http://www.gnu.org/software/gawk/man...ode/index.html
http://www.pement.org/awk/awk1line.txt
http://www.catonmat.net/series/awk-one-liners-explained


One more thing with the above, though. I assume that the IMAGES line results in the variable containing multiple entries. But it's almost never a good idea to store lists of related things in a single variable. As long as you're using bash or another advanced shell, you should use arrays instead. That's what they're designed for.

Code:
mapfile -t IMAGES < <( ec2-describe-images | awk '/031[1-8]2013/ { print $2 }' )

SNAPSHOTS=$( ec2-describe-images "${IMAGES[@]}" | awk '/snap/ {print $4}' )

Next point:

Code:
if [ "$IMAGES" > 0 ]; then
if [ -n "$IMAGES" ]; then
When using advanced shells like bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://mywiki.wooledge.org/BashFAQ/031
http://mywiki.wooledge.org/ArithmeticExpression

In addition, when inside square bracket tests, the symbols '>,<,=' and the like are string comparisons. You have to use '-gt,-lt,-eq' if you want to do numeric comparisons. Another reason why "((..))" is preferable here.


Code:
if (( IMAGES > 0 )); then
if [[ -n $IMAGES ]]; then

Finally, a bit of a meta-point. I see your date strings are in mmddyyyy format. I recently learned that the recommended form is to use the ISO-8601 international standard format of YYYY-MM-DD whenever possible.

If you can update your scripts and filenames to use the standard, it would let the shell to do natural sorting, and would probably make it easier to use globbing or other simple comparative techniques for getting the names you want.

If you don't want to go that way, you should try converting your date strings to epoch seconds (date +%s), and again you'll be able to use simple arithmetic to compare dates.
 
Old 04-03-2013, 06:50 PM   #9
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
David the H:

Quote:
ec2-describe-images...And what does it's output look like?
It's part of the Amazon API toolset and see http://www.linuxquestions.org/questi...4/#post4924373

I understand that I should use ISO-8601 format in my creation script. and
Code:
if [[ -n $IMAGES ]]; then
is preferred over:
Code:
if [ -n $IMAGES ]; then
wrt:
Quote:
I recently learned that the recommended form is to use the ISO-8601 international standard format of YYYY-MM-DD whenever possible.
If you can update your scripts and filenames to use the standard, it would let the shell to do natural sorting, and would probably make it easier to use globbing or other simple comparative techniques for getting the names you want.
"other simple comparative techniques". Ouch! Not! That's me.

I'm down with globbing I need to re-read up on it but I hear it's a 'good' thing, and sign me up for 'natural sorting'.

I most certainly can update those scripts since we authored 'em.
It is a simple assignment.

I understand your "chaining" reference.
When I get to "grep x | grep y | grep z" type stuff
I know it's time to write better code.

So I'm hoping this snippet suffices:

Thank you for your reply.
I appreciate your time, and skill.

Last edited by Habitual; 04-03-2013 at 07:01 PM.
 
Old 04-04-2013, 10:08 AM   #10
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374

Original Poster
Blog Entries: 37

Rep: Reputation: Disabled
and now "this"
http://www.linuxquestions.org/questi...6/#post4925042

I'll save you some time
Code:
[[ $variable == 0 ]] or [[ -n $variable ]]
 
Old 04-06-2013, 04:41 PM   #11
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,842

Rep: Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004Reputation: 2004
Here's an example of what you can do if your filenames are in standard date order:


Create files for every day in march and april in YYYYmmdd format.
(Yes, it creates a march 31 too, but it's just an example.)
Code:
touch file_2013{03..04}{01..31}.txt
Get the date for 14 days ago in the same format (currently March 24th)
Code:
compdate=$( date -d '-14 days' '+%Y%m%d' )
Loop through the directory and print out every file that has a name earlier than the comparison date. Use a simple parameter substitution to remove all non-digit characters from the name before comparing.

Code:
for fname in *; do
    if (( ${fname//[^0-9]} < compdate )); then
        echo "$fname"
    fi
done
The output:

Code:
file_20130301.txt
file_20130302.txt
file_20130303.txt
file_20130304.txt
file_20130305.txt
file_20130306.txt
file_20130307.txt
file_20130308.txt
file_20130309.txt
file_20130310.txt
file_20130311.txt
file_20130312.txt
file_20130313.txt
file_20130314.txt
file_20130315.txt
file_20130316.txt
file_20130317.txt
file_20130318.txt
file_20130319.txt
file_20130320.txt
file_20130321.txt
file_20130322.txt
file_20130323.txt
Notice how you can use simple gt/lt comparisons on the dates when they're in the non-hyphenated format. (Actually, this would still work anyway, as the hyphens would be removed from the name as well during comparison).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
"How to rename files with current date stamp appended into name using shell script?" nandi Linux - Newbie 4 10-05-2011 10:36 AM
How to print out "n" days ago's year, month and day? ArthurHuang Programming 3 02-10-2011 08:33 PM
Rotate Image (Script) without changing "modified" date tg3793 Linux - General 2 11-20-2010 02:32 AM
Perl: Turning a seconds-since-the-epoch number into "X days ago" or similar AlucardZero Programming 2 08-26-2010 05:59 PM
Monthly Archiving Script... help with "date" & "cron" Supp0rtLinux Linux - Software 3 01-03-2003 09:29 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 09:20 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration