CCTV wood for trees problem

Entropy1024 · 08-28-2018, 05:27 AM

I have a CCTV camera that stores a JPG each time it sees movement. After a day I have around 700+ images, most of these are just just 1-4 images of a cat walking past etc. Every now and again I get a person walk into the image and hanging around. For these I get say 10+ images and these are the ones I really want to see. However they are buried in with the noise of everyday movement.

I'm looking for some way of picking these notable events out.

How can I go about searching the JPGs for clusters of say 10+ images within a 1 minute window?

I'm guessing some sort of for next loop searching every minute and count how many JPGs within that loop. But not sure of the syntax. Also if 5 of these images was at 12:00 and 6 more at 12:01 it would not trigger an alert.

Many thanks for any help.

hydrurga · 08-28-2018, 06:07 AM

Personally, I would have a trigger counter, initially set to zero. I would loop through the images in ascending time order and check the timestamps. If the timestamp of the current image is less than x seconds later than the previous one then I would increase the counter by one, otherwise I would reset the counter. If, on resetting the counter, it was greater than y then I would print out the details of the first timestamp in the sequence. x and y would be declared at the start of the program.

This doesn't quite fit in with your proposal (there is no window as such), but it goes by the theory that it's the elapsed time between individual triggers that matters.

If you go ahead with the window idea that you proposed, you should use a rolling window, not search minutes like 12:00 and 12:01.

Personally, I would implement both your idea and mine, and then compare them to see which provides the better output.

In any case, you should write some code and provide it here - others can then help you with it.

Entropy1024 · 08-28-2018, 06:17 AM

Quote:

Originally Posted by hydrurga

Personally, I would have a trigger counter, initially set to zero. I would loop through the images in ascending time order and check the timestamps. If the timestamp of the current image is less than x seconds later than the previous one then I would increase the counter by one, otherwise I would reset the counter. If, on resetting the counter, it was greater than y then I would print out the details of the first timestamp in the sequence. x and y would be declared at the start of the program.

This doesn't quite fit in with your proposal (there is no window as such), but it goes by the theory that it's the elapsed time between individual triggers that matters.

If you go ahead with the window idea that you proposed, you should use a rolling window, not search minutes like 12:00 and 12:01.

Personally, I would implement both your idea and mine, and then compare them to see which provides the better output.

In any case, you should write some code and provide it here - others can then help you with it.

Like very much the rolling counter idea of yours. Still daunted as to how I code this. Will certainly give it a go. Will post any progress I make here.
Thanks again for the help.

michaelk · 08-28-2018, 07:34 AM

One idea would be to save the file names in an array. If the next file time difference is greater then your window time replace otherwise add name. When the number of images in the array is 10+ and exceeds your time window then copy the images to somewhere else and start over. The stat command provides an easy way to get timestamps in epoch time.

syg00 · 08-28-2018, 07:55 AM

stat is pretty expensive - "ls -tlr --time-style=full-iso <some-pattern>" should suffice.

Entropy1024 · 08-28-2018, 08:27 AM

Quote:

Originally Posted by michaelk

One idea would be to save the file names in an array. If the next file time difference is greater then your window time replace otherwise add name. When the number of images in the array is 10+ and exceeds your time window then copy the images to somewhere else and start over. The stat command provides an easy way to get timestamps in epoch time.

That sounds a great idea but no idea how to do this.

So far I have got the script to read all the files in todays JPG folder and compare the two times.
Note: As these files are downloaded from a FTP site once a minute all the timestamps are the same for that group of images downloaded. Therefore I'm usung the filenames which have times in them.

Filenames appear as 'DomeCCTV_216576543_20180828141354846_MOTION_DETECTION.jpg'. the cut command extracts the 141354846 portion of the filename, ie the time.
A simple subtraction calculates the difference between the two files.
I was thinking of adding the last 10 time differences then dividing them by 10 and if the average difference is less than x then do something. Guessing your array could help with this, but no idea how to achieve it. Can you please give an example?

Here is the script so far:

Code:

#!/bin/bash
DATENOW=$(date "+%y%m%d")
echo "Date "$DATENOW
FILES=/home/tim/Videos/domecctv/$DATENOW/*.jpg
cd /home/tim/Videos/domecctv/$DATENOW/

for f in $FILES
        do
                file1_time=$(echo $f | cut -c 61-69)
                file2_time=$(echo $g | cut -c 61-69)


                diff_time=$( expr $file2_time - $file1_time )
                echo "File1 "$file1_time" File2 "$file2_time"  Time Diff="$diff_time
                g=$f

                #if total of the last 10 time differences divide by 10 less than 1000 then;
                #do stuff


        done

michaelk · 08-28-2018, 08:36 AM

As always there are many ways to accomplish the same thing. bash due to its many loop pitfalls isn't really the best choice when iterating a directory by time but just trying to suggest something fairly simple.

The OP would probably want to use ls -tlr --time-style=+%s *.jpg so no time conversion is required.
or
stat --printf %Y filename

I forgot that typically the picture file names are time stamped and automatically sort by time...

Let me think on that a bit and will post an idea shortly.

hydrurga · 08-28-2018, 08:42 AM

Just a note that you can extract the actual EXIF creation date timestamp from a JPG file. There are quite a few programs that can do this, including Imagemagick and Exiftool.

e.g.

Imagemagick:

Code:

identify -format %[EXIF:DateTime] photo.jpg

Exiftool:

Code:

exiftool -p '$DateTimeOriginal' photo.jpg

michaelk · 08-28-2018, 11:08 AM

Not real code nor tested but you should get the idea.

Code:

for f in $FILES
do
  curr_time=$(echo $f | cut -c 61-69)

  if (( $count == 0 )); then # save first file
     f_array=() # initialize array
     count=1
     f_array+=($f) # add file name to array
     start_time=$curr_time
     continue
  fi

  diff_time=$( expr $curr_time - $start_time )
  if (( $diff_time>60 && $count < 5 )); then  # must be the cat
      f_array=() # initialize array
     count=1
     f_array+=($f) # add file name to array
     start_time=$curr_time
     continue
  fi 
     
  if (( $diff_time>60 && $count > 10 )); then # save for further investigation
     for f in "${f_array[@]}"
     do
	# do something with each file in list
     done

     f_array=() # start over / initialize array
     count=1
     f_array+=($f) # add file name to array
     start_time=$curr_time
     continue
  fi

  f_array+=($f)
  count=$count+1
  
done

Entropy1024 · 08-30-2018, 02:09 PM

Quote:

Originally Posted by michaelk

Not real code nor tested but you should get the idea.

Code:

for f in $FILES
do
  curr_time=$(echo $f | cut -c 61-69)

  if (( $count == 0 )); then # save first file
     f_array=() # initialize array
     count=1
     f_array+=($f) # add file name to array
     start_time=$curr_time
     continue
  fi

  diff_time=$( expr $curr_time - $start_time )
  if (( $diff_time>60 && $count < 5 )); then  # must be the cat
      f_array=() # initialize array
     count=1
     f_array+=($f) # add file name to array
     start_time=$curr_time
     continue
  fi 
     
  if (( $diff_time>60 && $count > 10 )); then # save for further investigation
     for f in "${f_array[@]}"
     do
	# do something with each file in list
     done

     f_array=() # start over / initialize array
     count=1
     f_array+=($f) # add file name to array
     start_time=$curr_time
     continue
  fi

  f_array+=($f)
  count=$count+1
  
done

Thanks for help. I did not read your script till I created the probably terrible script below. However it works. Not perfect but does considerably cut down on the noise.

Code:

  #!/bin/bash
# 29aug18
# Find cluster of events in short window of time


DATENOW=$(date "+%y%m%d")
FILES=/home/tim/Videos/domecctv/$DATENOW/*.jpg
cd /home/tim/Videos/domecctv/$DATENOW/

EVENTS=0
rm -f /home/tim/Videos/domecctv/event/*.jpg


EVENTSROLL=4
TIMEDIFF=5000

TOTALEVENTS=0

g="/home/tim/Videos/domecctv/180828/192.168.0.92_01_20180828000636229_MOTION_DETECTION.jpg"

for f in $FILES
        do
                #Extract time from current and previous filename
                #file1_time=$(echo $f | cut -c 56-64)
                #file2_time=$(echo $g | cut -c 56-64)

                file1_timea=$(echo $f | sed 's/^.*20'$DATENOW'/20'$DATENOW'/')
                file1_time=${file1_timea::-21}
                #echo "File1 = "$file1_time

                file2_timea=$(echo $g | sed 's/^.*20'$DATENOW'/20'$DATENOW'/')
                file2_time=${file2_timea::-21}
                #echo "File2 = "$file2_time

                #Calculate difference in time
                diff_time=$( expr $file1_time - $file2_time )
                #echo "File1 "$file1_time" File2 "$file2_time"  Time Diff="$diff_time

                #Move Current file to $g
                g=$f

                        #Find fils close to each other in time
                        if [ $diff_time -lt $TIMEDIFF ]
                                then
                                        echo "Less than "$TIMEDIFF" in a row ="$EVENTS
                                        mkdir -p /home/tim/Videos/domecctv/event
                                        cp $f /home/tim/Videos/domecctv/event/
                                        cp $g /home/tim/Videos/domecctv/event/
                                        ((EVENTS++))
                                else
                                        #If more then x in a row
                                        if [ $EVENTS -gt $EVENTSROLL ]
                                        then
                                        echo "More than "$EVENTSROLL" in a row. Event triggered"
                                        mkdir -p /home/tim/Pictures/domecctv/$DATENOW/
                                        cp /home/tim/Videos/domecctv/event/* /home/tim/Pictures/domecctv/$DATENOW/
                                        ((TOTALEVENTS++))

                                        fi

                                rm -f /home/tim/Videos/domecctv/event/*
                                EVENTS=0
                        fi

        done
echo "Total Events = "$TOTALEVENTS