Quote:
Originally Posted by sarmadys
I have developed the software myself. It was previously developed using Java but now it is being converted to C++. I am free to do whatever I want.
|
OK.
I actually have a system in place that does exactly what you are trying to do, though I'm only dealing with 32K buffers. My system is multi-threaded and multi-process.
One process (written in C) has threads that collect data, process data, and report data. The process data and report data threads, together, do what you want to do. Process data works with N buffers, where N is a configuration item and I normally have it set to about 52 or 53. This means that process data works with the current data and the 51 or 52 previous sets of data (I'm collecting real-time statistics along the way and using it for my maximum likelihood estimator routine).
Report data has the job of shipping the contents of buffers once the processing on them is complete (and never before it is complete). Report data sends the buffers to another process (written in C++) that visualizes the data (which is what you want to do). The visualize process may be located someplace else across the internet, or it may be on the same machine...doesn't matter.
The process data and report data threads, being threads, both access the buffer data from within the same process. The report data thread uses UDP sockets to ship the data to the visualization process.
My development and testing setup has the process data and report data threads running on an old 1 GHz Celeron processor with 768 Megs of RAM. The visualization routine runs on a much bigger and faster machine elsewhere on the LAN, and the LAN is a 100 Mb/sec environment.
With this setup, and my 32K buffers, I can update my visualization display about 40 times/sec, using no data compression when I send the data across the network. The celeron processor is at about 15% utilization during this process, and the network is running at nearly full speed.
This system is doing real-time data collection and I honestly can't tell you how fast it COULD run transferring data because the whole thing winds up waiting on the front end, which is an L-band tuner followed by an A/D and a DSP that does an FFT on the incoming signal. It takes between 10 and 15 milliseconds for each sample to come in, and that sets a hard upper limit on how fast my system will run which is why I'm happy to use an old celeron to do the processing job.
With faster multicore hardware and (if needed) a faster LAN, I'll bet you could meet your throughput requirements easily enough.
