I have an application generating output at a high rate. Milliseconds. The output is appended to a file in /dev/shm so it's fast enough and won't tax the SSD.

Now, I need to capture that output with another application which I will write in a scripting language. It basically "slurps" the output file at very short (milliseconds) intervals.

I've given up on this idea before, I want to try again. I remember there were a number of pitfalls, mainly around the problem of "skipping beats."

The last time, I tried it with mkfifo but I didn't have a good experience. It was awkward because the capture wouldn't really begin until I opened the receiving end of the fifo with tail and I wish I could avoid that. I think the flow also got interrupted after some time as if it was exhausted or broken or something.

In the Windows 9x era, that would be done with COM or DDE I guess. What about Linux? What other approaches can you suggest? Networking? Sockets? Something else?

Can’t you have one program write to standard output, and one program read from standard input, and pipe one into the other?

Yes, that would be the obvious solution. But it's a Windows application running on Wine. It can write "logs" to disk but it can't send data to standard output. It's not stdout-aware.

Rewrite the first program so that it does both at once with the output.
Show what the first program is doing. The source code for it. Or the docs for it.

You don't understand. It's a third-party app provided by a client. I didn't write it and I don't have the source. It just writes to disk, it's all I have. Best I can do is make it write to a RAM disk. In fact, it writes to a symlink that points to /dev/shm. It thinks it's a regular directory.

What are you going to do when the RAM-disk becomes full? Have you got some log-rotating mechanism?

1 members found this post helpful.

so the best way is still to create a fifo, and ask that app to write into it. You only need to open it before, like tail -f fifo or similar and you can redirect its output anywhere you want. Or capture it with your script (pipe it to any other app).
/dev/shm is not a regular directory, in /dev there is nothing "regular", but anyway you can use that if you wish.

I'm glad you asked. The log doesn't append, it overwrites. So the log file is always one line only. So you made me remember that there was this problem of the file being not found by the capturing program every once in a while. Easy problem to handle from the capturing program's standpoint, but I vaguely remember that it was a problem for the fifo idea.

hm. you cannot delete anything from the fifo, if you sent something into it. So it will accept all the lines one by one. Probably the app will reopen it every time, but it does not really matter. As far as I see.

I'm running tests. I find fifo very unreliable.

Terminal A:
$ mkfifo myfifo.fifo
Terminal B:
$ less -f myfifo.fifo
(empty)

Terminal A:
$ echo test1 > myfifo.fifo
Terminal B:
test1

Terminal A:
$ echo test2 > myfifo.fifo
Terminal B:
test1
test2

Terminal A:
$ echo test3 > myfifo.fifo
Terminal B:
test1
test2
test3

First problem: I don't want to retrieve a list. I don't want it piling up in any way. Why is it piling up?

Another test:

$ watch less -f myfifo.fifo
That doesn't even work. It won't capture.

Back to the other test. A simple repeat:

Terminal A:
$ mkfifo myfifo.fifo
Terminal B:
$ less -f myfifo.fifo
(empty)

Terminal A:
$ echo test1 > myfifo.fifo
Terminal B:
test1

Terminal A:
$ echo test2 > myfifo.fifo
Terminal B:
test1

Terminal A:
$ echo test3 > myfifo.fifo
Terminal B:
test1
test3

Terminal A:
$ echo test2 > myfifo.fifo
Terminal B:
test1
test3
test2

Why did the first "test2" miss this time around? Why did I have to repeat it? It didn't miss the first time but it did now. Why does that happen? That cannot happen. If it does, fifos are unfit for the purpose.

I don't really understand, timing can be important. If you want to read the same fifo both in A and B you need to use tee or some other tool to duplicate the output. Otherwise it may not really do what you expect.
Don't forget they are buffered (too).

But it's not both reading. Terminal A writes (echo >) and Terminal B reads.

The timing is manual, I mean, I have two terminal windows open, I "write" in one then switch to the other one manually to see what happened then switch back to write again and so on. It's nothing really fast.

Hmm, I get quite different results from you.
In this case, I get
and I have to quit less before I can see any further data.

This works fine for me. I tried in terminal A, and in terminal B, and I saw "test1" then "test2" and "test3" each in turn.

Maybe some behaviour of less has changed? I'm running on a somewhat old system. Anyway, I don't think it makes much sense to use less on a fifo. Probably cat is better for testing.

1 members found this post helpful.

You are right, cat has more consistent behavior.

But anyway, I found that the Windows application cannot write to a named pipe. I'm not sure what happens but it just won't work. Nothing is written, I can't read the other end of the pipe at all. It has to be a regular text file.

It is not cat, but buffering. But it is not important any more, if your app cannot use a pipe. I would check if that was a permission issue and/or if your app can use stdout or some other way. You cannot easily collect that output, if that file is continuously overwritten (you will never know if you missed something or if the file is overwritten during the read or ??)