Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
05-20-2017, 12:15 PM
|
#1
|
LQ Newbie
Registered: May 2017
Posts: 4
Rep: 
|
Find and move files based on time intervals
Hi,
I have a situation where there are about 100K-150K files in folders divided by month. These are snapshot images that were captured by a Synology Surveillance Station. Problem is we have way too many files to work with as the system was capturing about every 10 seconds. The images are needed to ultimately create a time lapse sequence over about a 12-15 month period.
So I was trying to see if there might be someone who could help me to see if a command could be used to search these folders and move only files that are in 60 minutes intervals to a designated smaller folder? The captures though are not completely in 24 hours intervals per day. The setting was to capture during daylight hours like between 8am-6pm. Hope this makes sense.
Going forward, we reduced the capture time to avoid this problem but needing to deal with the older captures.
|
|
|
05-20-2017, 12:27 PM
|
#2
|
LQ Guru
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (15 current), Slack15, Ubuntu studio, MX Linux, FreeBSD 13.1, WIn10
Posts: 10,342
|
man find
Code:
find /PathLookingIn -mtime 'n=T' -name "fileType" -exec mv -t {} /pathTo/ \;
60 minutes intervals = 1 every 60min
Math required.
Have to get a start time on 'the first file in folder' then a patten of the 'two' files to know it is the same sequence then compare it against the 'first file'.
if their is a 60 minute gap between the two.
If yes move, (one or both) if no skip go to next 'batch' in same or different folder?
so I think it is going to take a little more then just one command.
Defiantly need more information on data that is being worked on, and how it is actually being stored, ie. same dir or dir/sub-dir or different dirs etc.
Last edited by BW-userx; 05-20-2017 at 12:53 PM.
|
|
|
05-20-2017, 12:35 PM
|
#3
|
LQ Guru
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,756
|
The find utility can also search by minute, see -mmin for that.
Code:
find /PathLookingIn -mmin -15 -type f -exec mv -t /pathTo/ {} \;
You could build a series of searches for successively later intervals:
Edit: fixed mv arguments.
Last edited by kakistocrat; 05-20-2017 at 01:04 PM.
Reason: swap mv arguments
|
|
1 members found this post helpful.
|
05-20-2017, 02:03 PM
|
#4
|
LQ Guru
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (15 current), Slack15, Ubuntu studio, MX Linux, FreeBSD 13.1, WIn10
Posts: 10,342
|
hum
Code:
userx%voider ⚡ ~ ⚡> sudo find / -mmin 60 -type f > min60
find: ‘/proc/3534’: No such file or directory
find: ‘/proc/3536/task/3536/fdinfo/6’: No such file or directory
find: ‘/proc/3536/fdinfo/5’: No such file or directory
find: ‘/run/user/1000/gvfs’: Permission denied
userx%voider ⚡ ~ ⚡> cat min60
(file shortened)
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/BD6E7DB29B38AB5181B280348D5EB65D972C10EC
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/73A5001B5681C4F0073D6814F66380313F718B06
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/2F2E87B2F213EFC154DAADDB9EE63DA3D26B7B8A
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/7F325FDF5591A53CC051F4D461CE1E3606C11541
/home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/D66E5A0EADE4A3769AB7C0F952F70DD1B53EC0A0
userx%voider ⚡ ~ ⚡> status /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
bash: status: command not found
userx%voider ⚡ ~ ⚡> stat /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
File: /home/userx/.cache/mozilla/firefox/pjy015wl.default/cache2/entries/0C8C9B8711979568BAF0A264ACDDF6548E5879CA
Size: 3520 Blocks: 8 IO Block: 4096 regular file
Device: 807h/2055d Inode: 3937228 Links: 1
Access: (0600/-rw-------) Uid: ( 1000/ userx) Gid: ( 1000/ userx)
Access: 2017-05-20 12:59:13.370586112 -0400
Modify: 2017-05-20 12:59:13.370586112 -0400
Change: 2017-05-20 12:59:13.370586112 -0400
Birth: -
userx%voider ⚡ ~ ⚡>
file min60 created 1:59:xx PM
file checked is 12:59:xx
that takes from current time back 60 mins.
so my word problem I believe still stands to get files 60 minutes apart then move them somewhere else.
|
|
|
05-20-2017, 02:15 PM
|
#5
|
LQ Guru
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,756
|
Quote:
Originally Posted by BW-userx
that takes from current time back 60 mins.
so my word problem I believe still stands to get files 60 minutes apart then move them somewhere else.
|
Yes, I see now. My example is relative to the current time, but the time offset does not have to be in round units. Even the offset can be made relative to the current time. But maybe that sets in the possibility of a kind of race condition for things.
I'm thinking that the easier way then to get absolute blocks of time might be with perl and CPAN's File::Find module.
|
|
|
05-20-2017, 02:56 PM
|
#6
|
LQ Guru
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (15 current), Slack15, Ubuntu studio, MX Linux, FreeBSD 13.1, WIn10
Posts: 10,342
|
Quote:
Originally Posted by Turbocapitalist
Yes, I see now. My example is relative to the current time, but the time offset does not have to be in round units. Even the offset can be made relative to the current time. But maybe that sets in the possibility of a kind of race condition for things.
I'm thinking that the easier way then to get absolute blocks of time might be with perl and CPAN's File::Find module.
|
yes but still a basis for comparison needs to be set due to a whole bunch of files already created mixed within a bunch more that will not have the same intervals.
He has one bunch taken in 10 second intervals for over a years worth, then changed to every 60 minutes over a 24hr period non stop.
if they are all held within the same directory. does he want to go back to the very first photo taken then everyone after that every 60 minutes later (intervals) then save all of them and get rid of the rest?
or just pull out the ones that he started taking every 60 minutes and save them somewhere else, and do whatever to the others prior to the changing of the time photos are taken?
he needs to chime in again to see.
|
|
|
05-20-2017, 04:11 PM
|
#7
|
LQ Newbie
Registered: May 2017
Posts: 4
Original Poster
Rep: 
|
Thanks for the helpful info. I am trying to come up to speed on doing stuff on the command line. I can do most rudimentary stuff but in this case as the situation is a bit complicated, I realize I need help.
1. I had about 7 months of these images all sitting in one folder.
2. Through the find command using the mtime parameter, I managed to move then out into folders by month/30 days. There are 6 folders from June thru to Dec.
3. However, there are still way too many files in those folders.
4. Since this is for time lapse use, we only need snapshots on an hourly basis.
5. The snapshots were set to run from like 7am in the morning to 6pm. I could give you specific start and ending times if that would help.
6. So I don't mind doing a command or multi-command on each months folder to move out images that are about 60 minutes apart more or less from the beginning of the snapshot sequence.
Please let me know if I you need any further details from me.
Thank you.
John
|
|
|
05-20-2017, 11:41 PM
|
#8
|
LQ Guru
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,756
|
You also have the -newerXY option in find which can be used to find an absolute date + time if used with "m" and "t" . Here files from 2017-05-20 are grouped by their hour of last modification:
Code:
d='2017-05-20';
for h in $(seq 0 1 23);
do
echo -e "\n $h\n";
find /tmp/X/ -type f \
-newermt "$d $h:00:00" \
\! -newermt "$d $h:59:59" \
-print;
done;
Is that how your files are grouped? Can the times to search for be predicted automatically?
See about -newerXY
|
|
|
05-21-2017, 07:05 AM
|
#9
|
LQ Newbie
Registered: May 2017
Posts: 4
Original Poster
Rep: 
|
Ok I need to get the drive back and will work on some of these examples and post back with my results.
|
|
|
05-22-2017, 06:49 PM
|
#10
|
LQ Newbie
Registered: May 2017
Posts: 4
Original Poster
Rep: 
|
So I have folders set aside by month. Inside those folders are a ton of images which were taken in 10 second intervals. I want to scan those folders and extract images that are spaced 60 minutes in time and put them in a different folder. The starting time and/or reference is not critical. These are going to be used in a time-lapse sequence over a 12-16 month period so the data by day/month is not critical in terms of those relative times. Hope that makes more sense.
There are not that many folders so if I can find a command that will work for a single month, I can then apply that command with slightly different variables to the others.
|
|
|
05-22-2017, 07:27 PM
|
#11
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,414
|
Just extract every 360'th image - ignore the time.
KISS.
|
|
|
05-22-2017, 07:36 PM
|
#12
|
LQ Guru
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (15 current), Slack15, Ubuntu studio, MX Linux, FreeBSD 13.1, WIn10
Posts: 10,342
|
requires one loop ; a method to count off files ; then move the one that hits the magic number ; reset count ; repeat
about 21 lines of code, depending on how you write your scripts.
results every 360
Code:
userx%slackwhere ⚡ production ⚡> ./countOFF360
360
mv -v /media/data/Music-2/Dead-Kennedys/In-God-We-Trust/Dead Kennedys - In God We Trust - 02 - Moral Majority.mp3 /media/data
360
mv -v /media/data/Music-2/Deep-Purple/MadeInJapan(1998.724349419025)Cd2/2-10 - Space Truckin' (Remastered Album Version).mp3 /media/data
360
mv -v /media/data/Music-2/Earth-Wind-and-Fire/Earth,-Wind- and -Fire--Heritage-1990/10-Daydreamin'.mp3 /media/data
360
mv -v /media/data/Music-2/Stray-Cats/18th-July-2004,-London,-Shepherd's-Bush-Empire---Live-from-Europe/17 - Stray Cats - Please Don't Touch.mp3 /media/data
^C
http://www.linuxquestions.org/questi...53#post5713853
or look at post #3 and get an idea on how to write it with two loops then set it up to do one dir after the other. run it - go get a coffee or whatever come back when its all done, instead of having to re-run the script for each dir separately.
that for loop would work too.
the code inside of the two loops:
the inner loop:
count : check 360 ; move ; reset count - code
The outside Loop:
Takes care of your (more than one) dirs.
you can have it switch to send the files to different directories when you change to the another search dir too, that is setup in the post #3 on how to do that.
Last edited by BW-userx; 05-22-2017 at 08:49 PM.
|
|
|
All times are GMT -5. The time now is 01:42 PM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|