LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)
-   -   incron usage with create/close options together (https://www.linuxquestions.org/questions/linux-server-73/incron-usage-with-create-close-options-together-4175648059/)

mackowiakp 02-11-2019 04:22 AM

incron usage with create/close options together
 
I have one server for collecting measurement data from approximately 300 units. All data come in form of CSV files every 10 min per unit. Each file has 30-40 kB size.
And there is another server (in different location) which RSYNC (over SSH) CSV files from the first server to second one.
It is necessary to convert text form of CSV file to another text format acceptable by InfluxDB database API. So I create small program in C to make such conversion and place the content of such file in database. It works correctly.
But I have to know (on the second one server) that new file come. Because of that I use incron facility which is extension of inotify-tools package. So the action is activated if new file in pointed dir was open/modify/create etc. But transmission (RSYNC) takes several sec so from my point of view file is new when it was created as new, transmit and close. Only opening the file should not activate any action. Modification - yes. But any action should be taken if file was create or modify and than close.
How incron config file should looks like?
Any idea? Please notice that 300 measuring units could initiated hundreds of conversion processes. It is because of sometimes communication is broken-down with particular unit for several hours and after that such unit can transmit dozens of files.
Any idea how to configure incron?

Turbocapitalist 02-11-2019 04:41 AM

Have you checked "man 5 incrontab" for the correct syntax? I expect you would want something like this:

Code:

/path/to/some/directory/ IN_CLOSE_WRITE /usr/local/bin/converter $#
However, I have not tested that fully with rsync and I suspect there will be problems because temporary files get openened and closed by rsync. So maybe incron is not appropriate here.

Are only new files being created or are the old files also being updated?

My suspicion is that the script calling rsync will also have to call the conversion program passing it the name of the updated file(s).

mackowiakp 02-11-2019 05:04 AM

Yes, the last (the newest) file could be modify each RSYNC. Of course there are created temp rsync files. But maybe there is possibility to place it in another dir (like RAM-disk)? The server has 64 GB RAM and all temporary files I moved to RAM-disk, to safe lifespan of SSD disk. Including file conversion tmp files. It is because after conversion from CSV to InfluxDB format and placing it in DB, such files are not necessary. Server has no X interface turn-on so amount of RAM memory I have is as much as I want.

mackowiakp 02-11-2019 06:57 AM

Yes RSYNC tmp files is possible to create anywhere:
Code:

-T, --temp-dir=DIR create temporary files in directory DIR
But what about running conversion procedure running after create and close or after modify and close not meanwhile ? Is it possible from incron?

Turbocapitalist 02-11-2019 07:44 AM

Maybe you could have the temporary files outside of the watched directory and then have a lock file to prevent triggering the incron job until rsync has exited.

Turbocapitalist 02-11-2019 08:49 AM

You could also use --remote-option to make a log file on the remote host using --log-file and then watch that file for when it closes and then parse it.

Code:

rsync ... --remote-option="--log-file=/some/path/rs.log" ...
Then the script called by incron would have to extract the changed files. The paths would be relative to the paths used in rsync.

Code:

awk '$4~/^>/{print $5}' /some/path/rs.log


All times are GMT -5. The time now is 11:55 PM.