LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 02-11-2019, 05:22 AM   #1
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 6, SH4, Debian
Posts: 137

Rep: Reputation: Disabled
incron usage with create/close options together


I have one server for collecting measurement data from approximately 300 units. All data come in form of CSV files every 10 min per unit. Each file has 30-40 kB size.
And there is another server (in different location) which RSYNC (over SSH) CSV files from the first server to second one.
It is necessary to convert text form of CSV file to another text format acceptable by InfluxDB database API. So I create small program in C to make such conversion and place the content of such file in database. It works correctly.
But I have to know (on the second one server) that new file come. Because of that I use incron facility which is extension of inotify-tools package. So the action is activated if new file in pointed dir was open/modify/create etc. But transmission (RSYNC) takes several sec so from my point of view file is new when it was created as new, transmit and close. Only opening the file should not activate any action. Modification - yes. But any action should be taken if file was create or modify and than close.
How incron config file should looks like?
Any idea? Please notice that 300 measuring units could initiated hundreds of conversion processes. It is because of sometimes communication is broken-down with particular unit for several hours and after that such unit can transmit dozens of files.
Any idea how to configure incron?
 
Old 02-11-2019, 05:41 AM   #2
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 3,663
Blog Entries: 3

Rep: Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644
Have you checked "man 5 incrontab" for the correct syntax? I expect you would want something like this:

Code:
/path/to/some/directory/ IN_CLOSE_WRITE /usr/local/bin/converter $#
However, I have not tested that fully with rsync and I suspect there will be problems because temporary files get openened and closed by rsync. So maybe incron is not appropriate here.

Are only new files being created or are the old files also being updated?

My suspicion is that the script calling rsync will also have to call the conversion program passing it the name of the updated file(s).
 
Old 02-11-2019, 06:04 AM   #3
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 6, SH4, Debian
Posts: 137

Original Poster
Rep: Reputation: Disabled
Yes, the last (the newest) file could be modify each RSYNC. Of course there are created temp rsync files. But maybe there is possibility to place it in another dir (like RAM-disk)? The server has 64 GB RAM and all temporary files I moved to RAM-disk, to safe lifespan of SSD disk. Including file conversion tmp files. It is because after conversion from CSV to InfluxDB format and placing it in DB, such files are not necessary. Server has no X interface turn-on so amount of RAM memory I have is as much as I want.
 
Old 02-11-2019, 07:57 AM   #4
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 6, SH4, Debian
Posts: 137

Original Poster
Rep: Reputation: Disabled
Yes RSYNC tmp files is possible to create anywhere:
Code:
-T, --temp-dir=DIR create temporary files in directory DIR
But what about running conversion procedure running after create and close or after modify and close not meanwhile ? Is it possible from incron?
 
Old 02-11-2019, 08:44 AM   #5
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 3,663
Blog Entries: 3

Rep: Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644
Maybe you could have the temporary files outside of the watched directory and then have a lock file to prevent triggering the incron job until rsync has exited.
 
Old 02-11-2019, 09:49 AM   #6
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 3,663
Blog Entries: 3

Rep: Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644Reputation: 1644
You could also use --remote-option to make a log file on the remote host using --log-file and then watch that file for when it closes and then parse it.

Code:
rsync ... --remote-option="--log-file=/some/path/rs.log" ...
Then the script called by incron would have to extract the changed files. The paths would be relative to the paths used in rsync.

Code:
awk '$4~/^>/{print $5}' /some/path/rs.log
 
  


Reply

Tags
close, create, cron, inotifywait


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Command associated with Incron not getting executed Bharath_ Linux - Newbie 3 05-15-2013 12:44 PM
LXer: Incron – a cron based on File System events LXer Syndicated Linux News 0 02-28-2011 01:50 AM
incron experiences - what do you think? billymayday Linux - Software 4 11-20-2008 11:17 AM
LXer: Triggering Commands On File/Directory Changes With Incron LXer Syndicated Linux News 0 09-02-2008 02:10 PM
LXer: Scheduling jobs based on filesystem activity with incron LXer Syndicated Linux News 0 08-21-2008 02:20 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 10:20 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration