Find and move files based on time intervals

fobistech · 05-20-2017, 12:15 PM

Hi,

I have a situation where there are about 100K-150K files in folders divided by month. These are snapshot images that were captured by a Synology Surveillance Station. Problem is we have way too many files to work with as the system was capturing about every 10 seconds. The images are needed to ultimately create a time lapse sequence over about a 12-15 month period.

So I was trying to see if there might be someone who could help me to see if a command could be used to search these folders and move only files that are in 60 minutes intervals to a designated smaller folder? The captures though are not completely in 24 hours intervals per day. The setting was to capture during daylight hours like between 8am-6pm. Hope this makes sense.

Going forward, we reduced the capture time to avoid this problem but needing to deal with the older captures.

BW-userx · 05-20-2017, 12:27 PM

man find

Code:

find /PathLookingIn -mtime 'n=T' -name "fileType" -exec mv -t {} /pathTo/ \;

60 minutes intervals = 1 every 60min
Math required.
Have to get a start time on 'the first file in folder' then a patten of the 'two' files to know it is the same sequence then compare it against the 'first file'.

if their is a 60 minute gap between the two.

If yes move, (one or both) if no skip go to next 'batch' in same or different folder?

so I think it is going to take a little more then just one command.

Defiantly need more information on data that is being worked on, and how it is actually being stored, ie. same dir or dir/sub-dir or different dirs etc.

kakistocrat · 05-20-2017, 12:35 PM

The find utility can also search by minute, see -mmin for that.

Code:

find /PathLookingIn -mmin -15 -type f -exec mv -t /pathTo/ {} \;

You could build a series of searches for successively later intervals:

Code:

seq 0 30 1440

Edit: fixed mv arguments.

BW-userx · 05-20-2017, 02:03 PM

hum

Code:

userx%voider ⚡ ~ ⚡> sudo find / -mmin 60 -type f > min60
find: ‘/proc/3534’: No such file or directory
find: ‘/proc/3536/task/3536/fdinfo/6’: No such file or directory
find: ‘/proc/3536/fdinfo/5’: No such file or directory
find: ‘/run/user/1000/gvfs’: Permission denied
userx%voider ⚡ ~ ⚡> cat min60
(file shortened)
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/BD6E7DB29B38AB5181B280348D5EB65D972C10EC
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/73A5001B5681C4F0073D6814F66380313F718B06
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/2F2E87B2F213EFC154DAADDB9EE63DA3D26B7B8A
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/7F325FDF5591A53CC051F4D461CE1E3606C11541
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/D66E5A0EADE4A3769AB7C0F952F70DD1B53EC0A0
userx%voider ⚡ ~ ⚡> status /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
bash: status: command not found
userx%voider ⚡ ~ ⚡> stat /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
  File: /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
  Size: 3520            Blocks: 8          IO Block: 4096   regular file
Device: 807h/2055d      Inode: 3937228     Links: 1
Access: (0600/-rw-------)  Uid: ( 1000/   userx)   Gid: ( 1000/   userx)
Access: 2017-05-20 12:59:13.370586112 -0400
Modify: 2017-05-20 12:59:13.370586112 -0400
Change: 2017-05-20 12:59:13.370586112 -0400
 Birth: -
userx%voider ⚡ ~ ⚡>

file min60 created 1:59:xx PM
file checked is 12:59:xx
that takes from current time back 60 mins.

so my word problem I believe still stands to get files 60 minutes apart then move them somewhere else.

kakistocrat · 05-20-2017, 02:15 PM

Quote:

Originally Posted by BW-userx

that takes from current time back 60 mins.

so my word problem I believe still stands to get files 60 minutes apart then move them somewhere else.

Yes, I see now. My example is relative to the current time, but the time offset does not have to be in round units. Even the offset can be made relative to the current time. But maybe that sets in the possibility of a kind of race condition for things.

I'm thinking that the easier way then to get absolute blocks of time might be with perl and CPAN's File::Find module.

BW-userx · 05-20-2017, 02:56 PM

Quote:

Originally Posted by Turbocapitalist

Yes, I see now. My example is relative to the current time, but the time offset does not have to be in round units. Even the offset can be made relative to the current time. But maybe that sets in the possibility of a kind of race condition for things.

I'm thinking that the easier way then to get absolute blocks of time might be with perl and CPAN's File::Find module.

yes but still a basis for comparison needs to be set due to a whole bunch of files already created mixed within a bunch more that will not have the same intervals.

He has one bunch taken in 10 second intervals for over a years worth, then changed to every 60 minutes over a 24hr period non stop.

if they are all held within the same directory. does he want to go back to the very first photo taken then everyone after that every 60 minutes later (intervals) then save all of them and get rid of the rest?

or just pull out the ones that he started taking every 60 minutes and save them somewhere else, and do whatever to the others prior to the changing of the time photos are taken?

he needs to chime in again to see.

fobistech · 05-20-2017, 04:11 PM

Thanks for the helpful info. I am trying to come up to speed on doing stuff on the command line. I can do most rudimentary stuff but in this case as the situation is a bit complicated, I realize I need help.

1. I had about 7 months of these images all sitting in one folder.
2. Through the find command using the mtime parameter, I managed to move then out into folders by month/30 days. There are 6 folders from June thru to Dec.
3. However, there are still way too many files in those folders.
4. Since this is for time lapse use, we only need snapshots on an hourly basis.
5. The snapshots were set to run from like 7am in the morning to 6pm. I could give you specific start and ending times if that would help.
6. So I don't mind doing a command or multi-command on each months folder to move out images that are about 60 minutes apart more or less from the beginning of the snapshot sequence.

Please let me know if I you need any further details from me.

Thank you.

John

kakistocrat · 05-20-2017, 11:41 PM

You also have the -newerXY option in find which can be used to find an absolute date + time if used with "m" and "t" . Here files from 2017-05-20 are grouped by their hour of last modification:

Code:

d='2017-05-20'; 
for h in $(seq 0 1 23); 
do 
       echo -e "\n $h\n"; 
       find /tmp/X/ -type f \
           -newermt "$d $h:00:00" \
           \! -newermt "$d $h:59:59" \
           -print; 
done;

Is that how your files are grouped? Can the times to search for be predicted automatically?

See about -newerXY

Code:

man find

fobistech · 05-21-2017, 07:05 AM

Ok I need to get the drive back and will work on some of these examples and post back with my results.

fobistech · 05-22-2017, 06:49 PM

So I have folders set aside by month. Inside those folders are a ton of images which were taken in 10 second intervals. I want to scan those folders and extract images that are spaced 60 minutes in time and put them in a different folder. The starting time and/or reference is not critical. These are going to be used in a time-lapse sequence over a 12-16 month period so the data by day/month is not critical in terms of those relative times. Hope that makes more sense.

There are not that many folders so if I can find a command that will work for a single month, I can then apply that command with slightly different variables to the others.

syg00 · 05-22-2017, 07:27 PM

Just extract every 360'th image - ignore the time.
KISS.

BW-userx · 05-22-2017, 07:36 PM

requires one loop ; a method to count off files ; then move the one that hits the magic number ; reset count ; repeat

about 21 lines of code, depending on how you write your scripts.

results every 360

Code:

userx%slackwhere ⚡ production ⚡> ./countOFF360
360
mv -v /media/data/Music-2/Dead-Kennedys/In-God-We-Trust/Dead Kennedys - In God We Trust - 02 - Moral Majority.mp3 /media/data
360
mv -v /media/data/Music-2/Deep-Purple/MadeInJapan(1998.724349419025)Cd2/2-10 - Space Truckin' (Remastered Album Version).mp3 /media/data
360
mv -v /media/data/Music-2/Earth-Wind-and-Fire/Earth,-Wind- and -Fire--Heritage-1990/10-Daydreamin'.mp3 /media/data
360
mv -v /media/data/Music-2/Stray-Cats/18th-July-2004,-London,-Shepherd's-Bush-Empire---Live-from-Europe/17 - Stray Cats - Please Don't Touch.mp3 /media/data
^C

http://www.linuxquestions.org/questi...53#post5713853

or look at post #3 and get an idea on how to write it with two loops then set it up to do one dir after the other. run it - go get a coffee or whatever come back when its all done, instead of having to re-run the script for each dir separately.
that for loop would work too.

the code inside of the two loops:

the inner loop:
count : check 360 ; move ; reset count - code

The outside Loop:
Takes care of your (more than one) dirs.

you can have it switch to send the files to different directories when you change to the another search dir too, that is setup in the post #3 on how to do that.