[SOLVED] Removing duplicates with offset (datetime)

hattori.hanzo · 10-20-2010, 01:35 AM

I have a column of datetime entries which I sorted to removed duplicate entries. I have still lots of entries which are adjacent to each other by 1 second. How would I go about removing any entries which have an offset of the previous or after entry by 1 second?

Code:

2010-10-19 00:00:59;
2010-10-19 00:04:38; <--
2010-10-19 00:04:39; <--
2010-10-19 00:05:27; 
2010-10-19 00:10:14; <--
2010-10-19 00:10:15; <--
2010-10-19 00:10:31;
2010-10-19 00:12:14;
2010-10-19 00:14:04; <--
2010-10-19 00:14:05; <--
2010-10-19 00:14:06; <--
2010-10-19 00:15:19;

Jerry Mcguire · 10-20-2010, 05:11 AM

Convert the date-time into seconds since 1970. You then have a list of numbers, e.g.
List A:

Code:

10 19 20 30 31 32 40

Add 1 to the list:
List B:

Code:

11 20 21 31 32 33 41

For each 'b' in List B,
If 'b' is found in List A,
Then remove 'b' and 'b'-1 in List A.

The remainders in List A is the timestamps you need.

colucix · 10-20-2010, 05:39 AM

A solution in GNU awk:

Code:

{
  datespec = gensub(/[-:;]/," ","g")
  if ( mktime(datespec) - pre > 1 ) print
  pre = mktime(datespec)
}

grail · 10-20-2010, 05:49 AM

Wasn't sure from the description if you want to print the first one found or the last. This does the last:

Code:

awk -F"[:;]" 'NR > 1 && (sec > $(NF-1)+1 || sec < $(NF-1)-1){print line;last=sec}{line=$0;sec=$(NF-1)}END{if(sec > last + 1 || sec < last -1)print}' file

grail · 10-20-2010, 05:58 AM

Well that looks a lot better ... I always forget about the time functions

hattori.hanzo · 10-20-2010, 07:28 PM

Thank you colucix and grail. Much appreciated.