LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 02-26-2013, 09:42 AM   #1
nitya
LQ Newbie
 
Registered: Feb 2013
Location: India
Distribution: RedHat, Cent, Fedora, Ubuntu
Posts: 21

Rep: Reputation: Disabled
Thumbs up Find out timestamp line and delete if it is older than particular/specific days using bash script


Hi All,
I have a file with MAC addresses in which few lines are having epoch value in end of the line and few lines don't have any epoch value. Now I want to grep only those lines which are having epoch value. So could anybody please help me how to grep only epoch value lines?
Please see the below example file.
Code:
$ cat mac_address
00:11:22:aa:bb:c1  # Asus WiFi
00:11:22:aa:bb:c2  # Belkin WiFi
00:11:22:aa:bb:c3  # Test WiFi1
00:11:22:aa:bb:c4  # Test WiFi2
00:11:22:aa:bb:c5  # Test WiFi3
00:11:22:aa:bb:c6  # Nitya VM1
00:11:22:aa:bb:c7  # Nitya VM2
00:11:22:aa:bb:c8  # Nitya VM3
00:11:22:aa:bb:c9  # Rama # 1361471401
00:11:22:aa:bb:d1  # Sita # 1361471401
00:11:22:aa:bb:d2  # Lakshman # 1361471401
00:11:22:aa:bb:d3  # Anjaneya # 1361471401
00:11:22:aa:bb:d4  # Krishna # 1361471401
00:11:22:aa:bb:d5  # Arjuna # 1361730601
00:11:22:aa:bb:d6  # Bheema # 1361730601
00:11:22:aa:bb:d7  # Amar # 1361730601
00:11:22:aa:bb:d8  # Akbar # 1361730601
00:11:22:aa:bb:d9  # Antony # 1361730601
00:11:22:aa:bb:e1  # Karna # 1361817001
00:11:22:aa:bb:e2  # Suyodhana # 1361817001
00:11:22:aa:bb:e3  # Brahma # 1361817001
00:11:22:aa:bb:e4  # Vishnu # 1361817001
00:11:22:aa:bb:e5  # Maheshwara # 1361817001
00:11:22:aa:bb:e6  # Shakti # 1361817001
00:11:22:aa:bb:e7  # Om Sai # 1361817001

Last edited by nitya; 02-27-2013 at 11:43 AM.
 
Old 02-26-2013, 09:58 AM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
I take it you just want to extract the lines that have the timestamps? All that takes is an appropriate regex.

Code:
grep -E '# [0-9]+$' mac_address
(This may even be a bit of overkill, but better safe than sorry.)

I recommend taking the time to learn something about regular expressions. You'll be glad you did.

Here are a few regular expressions tutorials:
http://mywiki.wooledge.org/RegularExpression
http://www.grymoire.com/Unix/Regular.html
http://www.regular-expressions.info/
 
1 members found this post helpful.
Old 02-26-2013, 09:59 AM   #3
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,797
Blog Entries: 4

Rep: Reputation: 285Reputation: 285Reputation: 285
Invoke:
Code:
awk '$5 !~ /\#/ {if(NF>4) print $5}' mac_address

Last edited by shivaa; 02-26-2013 at 10:10 AM. Reason: Little better code
 
Old 02-26-2013, 10:32 AM   #4
nitya
LQ Newbie
 
Registered: Feb 2013
Location: India
Distribution: RedHat, Cent, Fedora, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: Disabled
Respected David the H. Sir, I am very lucky, you came to this thread.
Code:
grep -E '# [0-9]+$' mac_address
The above code worked successfully.
Actually I want to find out only those MAC address which are having epoch value(The time we added) and added 90 days ago and delete those lines from mac_address file and then reload the squid3 service. This is because whichever MAC address are given in this file will be having internet access for a couple of days/weeks. So we are giving internet access for maximum 90 days after that those device should not access internet. If anybody wanted internet access even after 90 days then we will add those MAC addresses manually with a new timestamp.
But I don't know shell script to achieve this so begging your kind help. So could you please provide a bash script for this i.e searching timestamp, calculating days with current timestamp and if it is reached 90 or more than 90 days then those lines should delete from mac_address file and run the "service squid3 reload" command.
Thanks in advance for your kind help.

Last edited by nitya; 02-27-2013 at 11:40 AM.
 
Old 02-26-2013, 10:37 AM   #5
nitya
LQ Newbie
 
Registered: Feb 2013
Location: India
Distribution: RedHat, Cent, Fedora, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by shivaa View Post
Invoke:
Code:
awk '$5 !~ /\#/ {if(NF>4) print $5}' mac_address
Hi shivaa, Very thanks for your kind reply. But it is not extracting whole line instead extracting only end of the field of epoch lines. But I was expecting to extract whole lines as from David The H. command.
 
Old 02-26-2013, 10:48 AM   #6
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,797
Blog Entries: 4

Rep: Reputation: 285Reputation: 285Reputation: 285
Simply do:
Code:
awk '{if(NF>4) print $0}' mac_address
 
Old 02-26-2013, 11:19 AM   #7
nitya
LQ Newbie
 
Registered: Feb 2013
Location: India
Distribution: RedHat, Cent, Fedora, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: Disabled
Thanks for the reply Shivva,
Quote:
Originally Posted by shivaa View Post
Simply do:
Code:
awk '{if(NF>4) print $0}' mac_address
No, This is extracting all lines which are having more than 4 fields in a line. But if I add something more like below then it extract that line also which I don't want.
Code:
00:11:22:aa:bb:c1  # Asus WiFi # some more text
But
Code:
grep -E '# [0-9]+$' mac_address
command can skip such lines even if it is having many fields and print only those lines which are having epoch value.

Last edited by nitya; 02-26-2013 at 12:02 PM.
 
Old 02-27-2013, 11:17 AM   #8
nitya
LQ Newbie
 
Registered: Feb 2013
Location: India
Distribution: RedHat, Cent, Fedora, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: Disabled
Hi All,
After reading many threads (especially from David the H.) from linuxquestions.org, I could try for bash script and worked successfully.
For that first I found the epoch value for 90 days using
Code:
expr $(date -d "90 days" +%s) - $(date +%s)
And according the above I could write a small script as below
Code:
# cat find_timestamp_and_delete_lines 
#!/bin/bash
max_age="7776000"
mac_file="/etc/squid3/mac_address"
#squid_file="/etc/squid3/squid.conf"
backup_dir="/root/backup/squid3"

###run this script if only $mac_file is exist and not empty.
if [[ -s "$mac_file" ]]; then
###backup $mac_file file and keep only 7 backup files
  cd "$backup_dir"
  cur_files=$( ls | wc -l )
  if [[ $cur_files -ge 7 ]]; then
     files2remove=$( expr "$cur_files" - 6 )
     rm $( ls -rt | head -n"$files2remove" )
  fi
  cp "$mac_file" "$backup_dir"/mac_address-$(date '+%d-%b-%Y-%H:%M:%S')
###search timestamp lines, count it's age and delete if it is older than 90 days
  timestamp=( $( grep -E '# [0-9]+$' "$mac_file" | awk '{ print $NF }' ) )
  for i in "${!timestamp[@]}" ; do
	now=$( date +%s )
	cur_age=$( expr "$now" - "${timestamp[i]}" 2>/dev/null )
		if [[ $cur_age -ge $max_age ]]; then
			echo "Below MAC address was added 90 days ago so deleted from "$mac_file" file
$(grep "${timestamp[i]}" "$mac_file" 2>/dev/null)" | mail -s "MAC address deleted" me@mydomain.com
			line2delete=( $( grep "${timestamp[i]}" "$mac_file" ) )
			sed -i "/$line2delete/d" "$mac_file"
		fi; done
  service squid3 reload
else
  echo ""$mac_file" file doesn't exist!"
fi
To test the above script I have changed timestamp to 90 days back for Anjaneya, Akbar and Suyodhana in mac_address file. Executed script and found working fine by sending mail if it deletes any MAC address.
Code:
# cat /etc/squid3/mac_address
00:11:22:aa:bb:c1  # Asus WiFi
00:11:22:aa:bb:c2  # Belkin WiFi
00:11:22:aa:bb:c3  # Test WiFi1
00:11:22:aa:bb:c4  # Test WiFi2
00:11:22:aa:bb:c5  # Test WiFi3
00:11:22:aa:bb:c6  # Nitya VM1
00:11:22:aa:bb:c7  # Nitya VM2
00:11:22:aa:bb:c8  # Nitya VM3
00:11:22:aa:bb:c9  # Rama # 1361471401
00:11:22:aa:bb:d1  # Sita # 1361471401
00:11:22:aa:bb:d2  # Lakshman # 1361471401
00:11:22:aa:bb:d3  # Anjaneya # 1345573801
00:11:22:aa:bb:d4  # Krishna # 1361471401
00:11:22:aa:bb:d5  # Arjuna # 1361730601
00:11:22:aa:bb:d6  # Bheema # 1361730601
00:11:22:aa:bb:d7  # Amar # 1361730601
00:11:22:aa:bb:d8  # Akbar # 1329849999
00:11:22:aa:bb:d9  # Antony # 1361730601
00:11:22:aa:bb:e1  # Karna # 1361817001
00:11:22:aa:bb:e2  # Suyodhana # 1329849001
00:11:22:aa:bb:e3  # Brahma # 1361817001
00:11:22:aa:bb:e4  # Vishnu # 1361817001
00:11:22:aa:bb:e5  # Maheshwara # 1361817001
00:11:22:aa:bb:e6  # Shakti # 1361817001
00:11:22:aa:bb:e7  # Om Sai # 1361817001
Hi All, Please let me know if I have done anything wrong in codes.
Once agagin thank you all and Thanks David The H.

Last edited by nitya; 02-27-2013 at 08:59 PM.
 
Old 03-02-2013, 07:57 AM   #9
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Quote:
Originally Posted by nitya View Post
So could you please provide a bash script for this i.e searching timestamp, calculating days with current timestamp and if it is reached 90 or more than 90 days then those lines should delete from mac_address file and run the "service squid3 reload" command.
If you need to calculate relative values, then you're definitely going to want to use awk instead (or another language like perl that has full mathematic ability).

Code:
awk 'BEGIN{ time = systime() - (86400*90) } ( $NF ~ /^[0-9]+$/ && $NF < time ) { next } { print }' mac_address
This will print out every line except ones where the final field is a string of numbers smaller than the epoch - 90 days.

I'll let you flesh out the rest of the script yourself.


Edit the next day: After looking back at this again with fresh eyes, we can simplify the above code even more:

Code:
awk '! ( $NF ~ /^[0-9]+$/ && $NF < systime() - 7776000 )' "$mac_address"
I started by removing the BEGIN section and moved the systime calculation directly into the test. And instead of using next to skip the matching lines, I inverted the test logic with "!" ("not"), so now it will only execute on non-matching lines instead. Finally, since the default action on a true match is to print, we can dump that keyword as well.

Last edited by David the H.; 03-03-2013 at 02:40 AM. Reason: as stated
 
1 members found this post helpful.
Old 03-02-2013, 08:55 AM   #10
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Now for a few quick comments on the script you posted.

1)
Code:
###backup $mac_file file and keep only 7 backup files
  cd "$backup_dir"
  cur_files=$( ls | wc -l )
  if [[ $cur_files -ge 7 ]]; then
     files2remove=$( expr "$cur_files" - 6 )
     rm $( ls -rt | head -n"$files2remove" )
  fi
This section would be more safely and cleanly handled with an array:

Code:
  cd "$backup_dir"
  cur_files=( * )
  num=$(( ${#cur_files[@]} - 7 ))
  (( num > 0 )) && rm "${cur_files[@]:0:$num}"
In particular parsing ls for filenames or metadata is highly discouraged.

And when using bash or ksh, it's recommended to use ((..)) for numerical tests.

http://mywiki.wooledge.org/ArithmeticExpression

2)
Code:
  timestamp=( $( grep -E '# [0-9]+$' "$mac_file" | awk '{ print $NF }' ) )
I just demonstrated this above, but in any case this is a useless use of grep. awk is powerful enough to do almost all text processing itself.

It's also not the cleanest way to load an array, since it relies on shell word-splitting. I suggest using mapfile instead, with a process substitution.

Code:
mapfile -t timestamp < <( awk '$NF ~ /^[0-9]+$/ { print $NF }' "$mac_file" )
3)
Code:
  for i in "${!timestamp[@]}" ; do
	now=$( date +%s )
This is just a minor point, but unless you need the exact "now" value for each timestamp, it would be more efficient to set the variable once before the loop, rather than resetting it again on every iteration inside it.

4)
Code:
$(grep "${timestamp[i]}" "$mac_file" 2>/dev/null)" | mail -s "MAC address deleted" me@mydomain.com
"$(..)" is used for command substitution, that is, inserting the output of one command inside another one. The grep command above should not have them.

If you really need to run a command in a subshell, without expansion, use "(..)" brackets instead.

5)
Code:
line2delete=( $( grep "${timestamp[i]}" "$mac_file" ) )
sed -i "/$line2delete/d" "$mac_file"
This is using array setting syntax to set line2delete. With the above, "$line2delete" will only hold the first word of the output (it's equal to "${line2delete[0]}").

Assuming the grep output is only one line, use simple command substitution instead.

Code:
line2delete=$( grep "${timestamp[i]}" "$mac_file" )
Finally, this whole script has a rather complex and inefficient code flow, IMO. A better approach would be to simply set the comparison time to a variable first, then loop through the file, using read's ability to load each line into an array, and compare the last field to the timestamp variable.

Or just use awk, as I posted above.

Last edited by David the H.; 03-02-2013 at 09:27 AM. Reason: added a sub-point
 
1 members found this post helpful.
Old 09-26-2014, 02:30 PM   #11
nitya
LQ Newbie
 
Registered: Feb 2013
Location: India
Distribution: RedHat, Cent, Fedora, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: Disabled
Hi Guru,
I was not aware of your reply. I am really sorry....... for this. And still that script is working fine but still I would like to follow your suggestion.
Thansk a lot.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Exctracting lines with grep leve.gr Programming 2 01-08-2012 07:44 PM
grep lines before and after magnum81 Programming 3 07-29-2011 05:52 PM
[SOLVED] Need help in displaying particular lines via grep khandu Programming 9 04-08-2011 10:03 AM
Getting only unique lines using grep.. sph90457 Linux - Newbie 6 07-05-2007 09:31 AM
Grep lines gsibble Linux - General 1 04-12-2004 02:30 AM


All times are GMT -5. The time now is 09:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration