c++ queues and kernel buffers

Hi all,

I don't know if this is the right forum to ask this question but here it is!

The context is a c++ application implementing a classic producer/consumer pattern using two threads and the std::queue. The implementation comes from one of many thread-safe examples available on the web. The queue has no control on maximum size, that is the producer can push as many elements as it wants.

The producer receives real-time sample buffers from an optical network interface (something like 50/100M samples per second) and pushes them in the queue. Buffers are 40k bytes size.

The consumer extracts buffers from the queue and writes them on SSD disk.

Running the application, "top" shows that the kernel buffers start to eat the available memory (16GB) until there's about 0.5GB available. The situation then remains stable, with the producer pushing millions of samples in the queue and the consumer slowly writing on the disk.

After all samples (for 5 minutes of streaming something like 120GB at 100M samples per second) are received, the consumer has yet to write to disk much more GB than the ones available in memory.

"top" also shows that no disk swapping is performed.

When the consumer finishes its job several minutes after, the file contaning all samples has correct size.

The question is: how can this happen? I thought that after allocating all memory for kernel buffers, the OS began to swap or the kernel killed the application because of out of memory. Where is this huge queue memorized while the consumer slowly writes its elements to file? I can't explain this...maybe some kernel expert can help me understand.

Thank you very much!

Regards,
Elvio

Hi Elvio, I assume that you C++ program is running in user space, so it wouldn't consume kernel memory other than what is used to buffer i/o. The kernel buffers writes to disk when the disk can't keep up. Do you have a swap file or partition? You won't see swapping without that, but if the swap device is the same SSD as you're writing the file to, swapping won't be any faster than writing to the output file itself. Hope this helps.

I think the most likely explanation is that the ssd mostly keeps up with the data, but needs the small writes to be grouped together into bigger ones.

While we wait for a kernel expert ...

What you describe is classic page cache usage. That is how it is designed to work. The usage is "elastic" in that it will occupy otherwise unused RAM, and will give it up if some other task requires it.
This also explains the delayed writes, but was designed for spinning disk. For SSD, depending on the model, and what bus it is connected to, can maintain multiple concurrent writes with the correct I/O scheduler.
Note all the "it depends".

If it's working ok, I'd say leave it alone unless you can tolerate making things worse as a learning exercise. Maybe have a read of this for some background.

Thank you all for your answers and explanations, it is clear that I need more background on the subject.
In the mean time the kernel is just a black box doing magic!

PS the swap file is on another SSD.

Regards,
Elvio

Basically, I think you need to dream up some kind of effective throttle. For example, some variant of std::queue] that does have "flow controls."

(Of course, when I say, "dream up," I really mean that you should methodically search through (e.g.) "GitHub" and "SourceForge" to find existing work that already does this.)

The writer may find that the queue sometimes refuses to accept another entry. This may mean that a sample might be dropped.

Another possibility is to design the system so that there are multiple consumers who are grabbing-off samples and writing them to different devices, to be sorted-out later.

You are stating you have 16 GB RAM and 120 GB data. That won't fit, for*sure.
I don't think top is the right tool to analyze it. You may completely switch off swap to see what will happen.

Hi all,

actually the RAM is 32Gb, not 16Gb, maybe this could partially explain the kernel doing magic.
The files do contain the right stuff (even the 120Gb ones) therefore the system is indeed doing right.

As said before a mix of page caching, fast SSD and maybe other factors do the trick.
Thanks again for your explanations.

Regards,
Elvio

PS Didn't try to disable the swap, if I have the chance I will try.