incron usage with create/close options together
I have one server for collecting measurement data from approximately 300 units. All data come in form of CSV files every 10 min per unit. Each file has 30-40 kB size.
And there is another server (in different location) which RSYNC (over SSH) CSV files from the first server to second one. It is necessary to convert text form of CSV file to another text format acceptable by InfluxDB database API. So I create small program in C to make such conversion and place the content of such file in database. It works correctly. But I have to know (on the second one server) that new file come. Because of that I use incron facility which is extension of inotify-tools package. So the action is activated if new file in pointed dir was open/modify/create etc. But transmission (RSYNC) takes several sec so from my point of view file is new when it was created as new, transmit and close. Only opening the file should not activate any action. Modification - yes. But any action should be taken if file was create or modify and than close. How incron config file should looks like? Any idea? Please notice that 300 measuring units could initiated hundreds of conversion processes. It is because of sometimes communication is broken-down with particular unit for several hours and after that such unit can transmit dozens of files. Any idea how to configure incron? |
Have you checked "man 5 incrontab" for the correct syntax? I expect you would want something like this:
Code:
/path/to/some/directory/ IN_CLOSE_WRITE /usr/local/bin/converter $# Are only new files being created or are the old files also being updated? My suspicion is that the script calling rsync will also have to call the conversion program passing it the name of the updated file(s). |
Yes, the last (the newest) file could be modify each RSYNC. Of course there are created temp rsync files. But maybe there is possibility to place it in another dir (like RAM-disk)? The server has 64 GB RAM and all temporary files I moved to RAM-disk, to safe lifespan of SSD disk. Including file conversion tmp files. It is because after conversion from CSV to InfluxDB format and placing it in DB, such files are not necessary. Server has no X interface turn-on so amount of RAM memory I have is as much as I want.
|
Yes RSYNC tmp files is possible to create anywhere:
Code:
-T, --temp-dir=DIR create temporary files in directory DIR |
Maybe you could have the temporary files outside of the watched directory and then have a lock file to prevent triggering the incron job until rsync has exited.
|
You could also use --remote-option to make a log file on the remote host using --log-file and then watch that file for when it closes and then parse it.
Code:
rsync ... --remote-option="--log-file=/some/path/rs.log" ... Code:
awk '$4~/^>/{print $5}' /some/path/rs.log |
All times are GMT -5. The time now is 11:55 PM. |