LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-20-2017, 12:15 PM   #1
fobistech
LQ Newbie
 
Registered: May 2017
Posts: 4

Rep: Reputation: Disabled
Find and move files based on time intervals


Hi,

I have a situation where there are about 100K-150K files in folders divided by month. These are snapshot images that were captured by a Synology Surveillance Station. Problem is we have way too many files to work with as the system was capturing about every 10 seconds. The images are needed to ultimately create a time lapse sequence over about a 12-15 month period.

So I was trying to see if there might be someone who could help me to see if a command could be used to search these folders and move only files that are in 60 minutes intervals to a designated smaller folder? The captures though are not completely in 24 hours intervals per day. The setting was to capture during daylight hours like between 8am-6pm. Hope this makes sense.

Going forward, we reduced the capture time to avoid this problem but needing to deal with the older captures.
 
Old 05-20-2017, 12:27 PM   #2
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (current), FreeBSD, Win10, It varies
Posts: 9,952

Rep: Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148
man find

Code:
find /PathLookingIn -mtime 'n=T' -name "fileType" -exec mv -t {} /pathTo/ \;
60 minutes intervals = 1 every 60min
Math required.
Have to get a start time on 'the first file in folder' then a patten of the 'two' files to know it is the same sequence then compare it against the 'first file'.

if their is a 60 minute gap between the two.

If yes move, (one or both) if no skip go to next 'batch' in same or different folder?

so I think it is going to take a little more then just one command.

Defiantly need more information on data that is being worked on, and how it is actually being stored, ie. same dir or dir/sub-dir or different dirs etc.

Last edited by BW-userx; 05-20-2017 at 12:53 PM.
 
Old 05-20-2017, 12:35 PM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,614
Blog Entries: 3

Rep: Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860
The find utility can also search by minute, see -mmin for that.

Code:
find /PathLookingIn -mmin -15 -type f -exec mv -t /pathTo/ {} \;
You could build a series of searches for successively later intervals:

Code:
seq 0 30 1440
Edit: fixed mv arguments.

Last edited by Turbocapitalist; 05-20-2017 at 01:04 PM. Reason: swap mv arguments
 
1 members found this post helpful.
Old 05-20-2017, 02:03 PM   #4
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (current), FreeBSD, Win10, It varies
Posts: 9,952

Rep: Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148
hum
Code:
userx%voider ⚡ ~ ⚡> sudo find / -mmin 60 -type f > min60
find: ‘/proc/3534’: No such file or directory
find: ‘/proc/3536/task/3536/fdinfo/6’: No such file or directory
find: ‘/proc/3536/fdinfo/5’: No such file or directory
find: ‘/run/user/1000/gvfs’: Permission denied
userx%voider ⚡ ~ ⚡> cat min60
(file shortened)
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/BD6E7DB29B38AB5181B280348D5EB65D972C10EC
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/73A5001B5681C4F0073D6814F66380313F718B06
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/2F2E87B2F213EFC154DAADDB9EE63DA3D26B7B8A
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/7F325FDF5591A53CC051F4D461CE1E3606C11541
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/D66E5A0EADE4A3769AB7C0F952F70DD1B53EC0A0
userx%voider ⚡ ~ ⚡> status /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
bash: status: command not found
userx%voider ⚡ ~ ⚡> stat /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
  File: /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
  Size: 3520            Blocks: 8          IO Block: 4096   regular file
Device: 807h/2055d      Inode: 3937228     Links: 1
Access: (0600/-rw-------)  Uid: ( 1000/   userx)   Gid: ( 1000/   userx)
Access: 2017-05-20 12:59:13.370586112 -0400
Modify: 2017-05-20 12:59:13.370586112 -0400
Change: 2017-05-20 12:59:13.370586112 -0400
 Birth: -
userx%voider ⚡ ~ ⚡>
file min60 created 1:59:xx PM
file checked is 12:59:xx
that takes from current time back 60 mins.

so my word problem I believe still stands to get files 60 minutes apart then move them somewhere else.
 
Old 05-20-2017, 02:15 PM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,614
Blog Entries: 3

Rep: Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860
Quote:
Originally Posted by BW-userx View Post
that takes from current time back 60 mins.

so my word problem I believe still stands to get files 60 minutes apart then move them somewhere else.
Yes, I see now. My example is relative to the current time, but the time offset does not have to be in round units. Even the offset can be made relative to the current time. But maybe that sets in the possibility of a kind of race condition for things.

I'm thinking that the easier way then to get absolute blocks of time might be with perl and CPAN's File::Find module.
 
Old 05-20-2017, 02:56 PM   #6
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (current), FreeBSD, Win10, It varies
Posts: 9,952

Rep: Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148
Quote:
Originally Posted by Turbocapitalist View Post
Yes, I see now. My example is relative to the current time, but the time offset does not have to be in round units. Even the offset can be made relative to the current time. But maybe that sets in the possibility of a kind of race condition for things.

I'm thinking that the easier way then to get absolute blocks of time might be with perl and CPAN's File::Find module.
yes but still a basis for comparison needs to be set due to a whole bunch of files already created mixed within a bunch more that will not have the same intervals.

He has one bunch taken in 10 second intervals for over a years worth, then changed to every 60 minutes over a 24hr period non stop.

if they are all held within the same directory. does he want to go back to the very first photo taken then everyone after that every 60 minutes later (intervals) then save all of them and get rid of the rest?

or just pull out the ones that he started taking every 60 minutes and save them somewhere else, and do whatever to the others prior to the changing of the time photos are taken?

he needs to chime in again to see.
 
Old 05-20-2017, 04:11 PM   #7
fobistech
LQ Newbie
 
Registered: May 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thanks for the helpful info. I am trying to come up to speed on doing stuff on the command line. I can do most rudimentary stuff but in this case as the situation is a bit complicated, I realize I need help.

1. I had about 7 months of these images all sitting in one folder.
2. Through the find command using the mtime parameter, I managed to move then out into folders by month/30 days. There are 6 folders from June thru to Dec.
3. However, there are still way too many files in those folders.
4. Since this is for time lapse use, we only need snapshots on an hourly basis.
5. The snapshots were set to run from like 7am in the morning to 6pm. I could give you specific start and ending times if that would help.
6. So I don't mind doing a command or multi-command on each months folder to move out images that are about 60 minutes apart more or less from the beginning of the snapshot sequence.

Please let me know if I you need any further details from me.

Thank you.

John
 
Old 05-20-2017, 11:41 PM   #8
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,614
Blog Entries: 3

Rep: Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860Reputation: 2860
You also have the -newerXY option in find which can be used to find an absolute date + time if used with "m" and "t" . Here files from 2017-05-20 are grouped by their hour of last modification:

Code:
d='2017-05-20'; 
for h in $(seq 0 1 23); 
do 
       echo -e "\n $h\n"; 
       find /tmp/X/ -type f \
           -newermt "$d $h:00:00" \
           \! -newermt "$d $h:59:59" \
           -print; 
done;
Is that how your files are grouped? Can the times to search for be predicted automatically?

See about -newerXY

Code:
man find
 
Old 05-21-2017, 07:05 AM   #9
fobistech
LQ Newbie
 
Registered: May 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
Ok I need to get the drive back and will work on some of these examples and post back with my results.
 
Old 05-22-2017, 06:49 PM   #10
fobistech
LQ Newbie
 
Registered: May 2017
Posts: 4

Original Poster
Rep: Reputation: Disabled
So I have folders set aside by month. Inside those folders are a ton of images which were taken in 10 second intervals. I want to scan those folders and extract images that are spaced 60 minutes in time and put them in a different folder. The starting time and/or reference is not critical. These are going to be used in a time-lapse sequence over a 12-16 month period so the data by day/month is not critical in terms of those relative times. Hope that makes more sense.

There are not that many folders so if I can find a command that will work for a single month, I can then apply that command with slightly different variables to the others.
 
Old 05-22-2017, 07:27 PM   #11
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,710

Rep: Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548
Just extract every 360'th image - ignore the time.
KISS.
 
Old 05-22-2017, 07:36 PM   #12
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (current), FreeBSD, Win10, It varies
Posts: 9,952

Rep: Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148Reputation: 2148
requires one loop ; a method to count off files ; then move the one that hits the magic number ; reset count ; repeat

about 21 lines of code, depending on how you write your scripts.

results every 360
Code:
userx%slackwhere ⚡ production ⚡> ./countOFF360
360
mv -v /media/data/Music-2/Dead-Kennedys/In-God-We-Trust/Dead Kennedys - In God We Trust - 02 - Moral Majority.mp3 /media/data
360
mv -v /media/data/Music-2/Deep-Purple/MadeInJapan(1998.724349419025)Cd2/2-10 - Space Truckin' (Remastered Album Version).mp3 /media/data
360
mv -v /media/data/Music-2/Earth-Wind-and-Fire/Earth,-Wind- and -Fire--Heritage-1990/10-Daydreamin'.mp3 /media/data
360
mv -v /media/data/Music-2/Stray-Cats/18th-July-2004,-London,-Shepherd's-Bush-Empire---Live-from-Europe/17 - Stray Cats - Please Don't Touch.mp3 /media/data
^C
http://www.linuxquestions.org/questi...53#post5713853

or look at post #3 and get an idea on how to write it with two loops then set it up to do one dir after the other. run it - go get a coffee or whatever come back when its all done, instead of having to re-run the script for each dir separately.
that for loop would work too.

the code inside of the two loops:

the inner loop:
count : check 360 ; move ; reset count - code


The outside Loop:
Takes care of your (more than one) dirs.

you can have it switch to send the files to different directories when you change to the another search dir too, that is setup in the post #3 on how to do that.

Last edited by BW-userx; 05-22-2017 at 08:49 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
find and move files AndrewNZ Linux - Newbie 4 06-23-2014 08:29 PM
Create Directory based on file creation date and move files Xanthari Linux - Newbie 4 01-28-2014 01:37 PM
move files based on date graphicsmanx1 Programming 7 12-18-2012 01:49 AM
Splitting audio files at specific time intervals quickfix Linux - Software 2 08-26-2005 07:18 AM
Find files based on time. FinalFantasy Linux - Software 4 05-21-2004 07:25 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 03:47 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration