bash filter output

aristosv · 11-03-2020, 03:37 AM

I am trying to create a filter that will read mobile phone numbers in a calendar, filter only the information I need and output it in a file.

So basically I need it to find the numbers that

- start with 94, 95, 96, 97 and 99
- are 8 digits long
- ignore any spaces in the number
- ignore country prefix (+357 or 00357)

Basically something more robust than what I currently have. Because now it picks up anything that contains (not starts with) 94, 95, 96, 97 and 99 and doesn't ignore country code.

Code:

gcalcli --config-folder $dir/auth agenda --tsv \
 "$(date -d 'now + 61 minutes')" "$(date -d 'now + 119 minutes')" \
 | sed 's/\(.\) /\1/g' \
 | grep -e 94 -e 95 -e 96 -e 97 -e 99 \
 | sed -E 's/^(.*:.{3}).*([[:digit:]]{8}).*/\1\2/' \
 | awk '{print $1 " " $2 " " $5}' \
 | awk -F'[- ]' '{ print $3"-"$2"-"$1" "$4" "$5 }' \
>> $dir/events/$(date +"%d_%m_%y")/today_$nexthour

pan64 · 11-03-2020, 03:49 AM

basically you need to use only one tool, for example grep, awk, perl, python and construct a single regexp which will do the job.
Would be nice to see an example for input and output. How to know what is a correct country prefix?

aristosv · 11-03-2020, 04:13 AM

The calendar input could be something as messy as this

Click image for larger version

Name: 1.png
Views: 10
Size: 16.2 KB
ID: 34464

The command output, without any filters is this

Code:

# gcalcli --config-folder $dir/auth/ agenda --tsv "$(date -d 'now + 61 minutes')" "$(date -d 'now + 119 minutes')"
2020-11-03	13:00	2020-11-03	14:00	99123456
2020-11-03	13:00	2020-11-03	14:00	94123456
2020-11-03	13:00	2020-11-03	14:00	95123456
2020-11-03	13:00	2020-11-03	14:00	4563453453543654567
2020-11-03	13:00	2020-11-03	14:00	+357678423683
2020-11-03	13:00	2020-11-03	14:00	00357584903
2020-11-03	13:00	2020-11-03	14:00	+35 799 1276 454
2020-11-03	13:30	2020-11-03	14:30	97123456

I need my output to be

Code:

03-11-2020 13:00 99123456
03-11-2020 13:00 94123456
03-11-2020 13:00 95123456
03-11-2020 13:30 97123456

The problem is that the person entering the phone numbers in the calendar might add the country prefix (which is always +357 or 00357 or simply 357) with or without spaces, or the phone number with or without spaces, or anything they can imagine.

What I need to do, is do my best to figure out if the information entered is a mobile phone number or not.

All mobile phone numbers start with 94, 95, 96, 97 and 99 so thats a good start. After that there's an additional 6 digits, without any spaces, but that doesn't stop the person entering the information from adding spaces. So I need to remove them. And I can simply ignore the country prefix, because I don't use it when sending an SMS. (thats the purpose of this script.)

pan64 · 11-03-2020, 05:35 AM

ok. what I see: you can use a single awk to:
1. get/parse date, time
2. put the end of the line from column 32(?) into a variable
3. remove spaces, country prefix in this variable
4. check ^9[45679] and skip line if not needed
5. print the result in the required format.

you can do the same in awk/perl/python/whatever or even in plain bash, choose your preferred language.
avoid those pipe chains if possible.

Turbocapitalist · 11-03-2020, 06:57 AM

I'd second the use of AWK. The above steps would cover it, but since you have tabs separating the fields things are much easier. You can set the FS (input field separator) pattern to be just a plain tab.

Code:

awk -F "\t" ...

awk ... FS="\t"

Then it is a matter of making a pattern to select on the fifth while manipulating the contents of the first and fifth fields and printing the results.

Perl works, too, but that goes without saying.

aristosv · 11-03-2020, 07:03 AM

Thanks for the suggestions. The code I posted took me about a month to figure out, and its not really working as I wanted it to. Any chance of helping me out with some code snippets? Not looking for ready-made stuff, but it will take me a long time to figure out your suggestions.

pan64 · 11-03-2020, 07:24 AM

For date and time:

Code:

awk -F '[\t :-]+' '{ printf "y:%i, m:%i, d:%02i, h:%i, m:%i\n", $1, $2, $3, $4, $5 }'

for the phone number:

Code:

{ number = substr($0, 32) # take the last field
  number = gensub("regexp", "", number) # remove country
  number = gensub("regexp", "", number) # remove spaces and whatever
  if match (number, "^9[456789]")        # check first two chars
}

Turbocapitalist · 11-03-2020, 08:07 AM

I'm wondering about the actual output of the "Google Calendar Command Line Interface" utility you have shown:

Code:

gcalcli --config-folder $dir/auth agenda --tsv \
 "$(date -d 'now + 61 minutes')" "$(date -d 'now + 119 minutes')"

I interpreted the --tsv to mean tab-separated-values. However, what you posted in #3 above has other than tabs and newlines for field and record separators. That will throw a wrench into the works until addressed.

aristosv · 11-03-2020, 08:09 AM

Thats the actual output of the command. It's a copy/paste from the terminal.

Turbocapitalist · 11-03-2020, 08:23 AM

Nevermind, I seem to have gotten mixed on things there.

It's tempting to try to use a positive lookbehind pattern, even if the variable width ones are still experimental:

Code:

#!/usr/bin/perl

use strict;
use warnings;

# read stdin until no more input                                                      
while (<>) {
    # split the special variable for default input into named variables                            
    my ($date1, $time1, $date2, $time2, $phone) = split(/\t/, $_);

    next unless ($phone);

    $phone =~ s/\s//g;          # ignore any spaces                             
    $phone =~ s/\+357/00357/;   # standardize country code                      

    # swap the day and the year around                                          
    $date1 = join('-',reverse(split(/-/, $date1)));

    # use positive lookbehind pattern to                                        
    # find an 8-digit number starting with 94 - 99                              
    # if it is by itself or if it follows the right country code                   
    if ($phone =~ m/(?<=^00357|^)9[4-9][0-9]{6}$/) {
        print qq($date1\t$time1\t$phone\n);
    }
}