LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-29-2016, 12:20 AM   #1
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Rep: Reputation: 8
How to obtain info that file is still open for ftp


I use the server (Synology NAS) on which are copied by ftp, video clips from recorders of CCTV system. When file is copied, small script convert it from DAV format to AVI, using the statement below:

Code:
find ./ -type f -name "*.dav" -exec ffmpeg -y -i {} -vcodec copy -movflags +faststart {}.avi \;
The problem is that there is a lot of clips transmitted constantly from CCTV recorders to Synology. That is many clips is transmitted from recorders to Synology at the same time. So the statement above will try to convert even clips where transmission is not over yet. How can I exclude such non complete clips from conversion? I can not use "inotify" feature because there is not such facility in Synology Linux. I think that it is necessary to know that particular file is still open for writing and exclude it for conwersion. Any idea What to do?
 
Old 09-29-2016, 03:03 AM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,864
Blog Entries: 3

Rep: Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052
The way I can think of would be to make a very short script to a) take a filename from "find" as a parameter, b) find the full path of the file, and then c) check the output of "lsof" for that file including the full path using "grep" The script would return 0 or 1 depending on what is found, but inverted so that if the file is found it fails. It would be called by -exec just before the existing -exec there in "find".

Last edited by Turbocapitalist; 09-29-2016 at 03:04 AM.
 
Old 09-29-2016, 03:08 AM   #3
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,905

Rep: Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615Reputation: 2615
I'd write a script based on a loop like eg
Code:
while true
do 
    find ./ -type f -name "*.dav" -exec ls -l {} \; >vfile.lis

    sleep 30  # or whatever

    # Now loop process vfile.lis, checking current file size. 
    # if a given filesize is the same as 30 secs ago, its finished d/loading, so process it

# After processing all 'completed' files in vfile.lis, rinse & repeat
done
(Overlapped by TC; lsof would work as well )

Last edited by chrism01; 09-29-2016 at 03:10 AM.
 
Old 09-29-2016, 03:43 AM   #4
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Quote:
Originally Posted by chrism01 View Post
I'd write a script based on a loop like eg
.............

(Overlapped by TC; lsof would work as well )
I try to do as You advice several days ago. But amount of video clips from 12 CCTV recorders (each 4-8 cameras connected) is so high that practically all the time a video clips are transmitted. Actually, a few at the same time. Thus, the conversion process does not have a chance (using the script) to start.
Unfortunately there is no lsof command in Synology original software (very strange). The only possibility is to use flock command. If it resolve a problem of course.
 
Old 09-29-2016, 04:05 AM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,864
Blog Entries: 3

Rep: Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052
Quote:
Originally Posted by mackowiakp View Post
The only possibility is to use flock command. If it resolve a problem of course.
Can you use a separate FTP session to grab each video clip? If so then "flock" might work because you could wrap the session in "flock" and check on it via "find". Again, you'd need a very short wrapper script to call from an extra -exec
 
Old 09-29-2016, 04:17 AM   #6
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
No I cant use separate FTP session to grab each video clip
 
Old 09-29-2016, 04:48 AM   #7
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,864
Blog Entries: 3

Rep: Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052
If it's not too nosy to ask, can you go into more detail about how the videos are getting to your machine? I'd like to think there is still a way to identify which ones are new and when they have finished downloading.
 
Old 09-29-2016, 05:02 AM   #8
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Each CCTV recorder, transmit via FTP video clips to dedicated subdir on Synology. Recorders during transmission procedure, creates separate directory for each camera connected to one. So I have separeted dirs for each recorder and inside such dir, FTP creates separated subdir for each camera. Thats all
 
Old 09-29-2016, 05:17 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,989

Rep: Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736
find .... -exec <Script>
the script first should rename the file, therefore the next find will not catch that.
 
Old 09-29-2016, 05:21 AM   #10
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,864
Blog Entries: 3

Rep: Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052Reputation: 3052
Maybe "fuser" is available? it will return 0 if the file is open and 1 if it is not. I don't have Synology to look at.

Looking around, there are add-ons via "ipkg" where you should be able to get "lsof"
 
Old 09-29-2016, 05:28 AM   #11
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Hmmm. I dont understand. What about if the file will be in transmission state and the script renamed it?
 
Old 09-29-2016, 05:34 AM   #12
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,989

Rep: Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736Reputation: 5736
if it was already opened (still writing it) - (you can check it with fuser for example) you can leave it as is.
It it was not opened at all rename it and process.
 
Old 09-29-2016, 05:35 AM   #13
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512Reputation: 1512
Does the synology file server support modification time records? If it does, you MIGHT be able to determine the files that have not been updated in xxx time (where xxx is larger than the longest update interval). Thus files whose update time older than the current time - xxx would be subject to renaming/conversion.
 
Old 09-29-2016, 05:37 AM   #14
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Quote:
Originally Posted by Turbocapitalist View Post
Maybe "fuser" is available? it will return 0 if the file is open and 1 if it is not. I don't have Synology to look at.

Looking around, there are add-ons via "ipkg" where you should be able to get "lsof"
ipkg is a part of optware that means part of "crack" of the system by adding more features. I can not install it because according to directives, the system must be original because only in this case, Synology paid support wi help me in any problems.

fuser is absent too. The only potentially useful command existing in system is lslocks.
 
Old 09-29-2016, 05:40 AM   #15
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Quote:
Originally Posted by pan64 View Post
if it was already opened (still writing it) - (you can check it with fuser for example) you can leave it as is.
It it was not opened at all rename it and process.
There is no fuser command
 
  


Reply

Tags
ftp, linux, shell script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Where to obtain info if /proc/bus/usb is deprecated ! intmail01 Linux - Hardware 5 11-11-2014 06:56 AM
Security Cameras and obtain info tools metallica1973 Linux - Hardware 1 06-02-2010 08:56 PM
Obtain file system info with df AQG Solaris / OpenSolaris 1 08-02-2007 08:18 PM
Is there a command to obtain uname info remotely sohos Linux - Newbie 2 01-12-2006 08:58 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:12 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration