Thank you for responding. In answer to your questions, the app is being run in a controlled lab "test environment". The environment consists of three machines (two PCs, one embedded STB) in a dedicated and closed LAN. The two PCs are used to generate two output A/V streams each for a total of four different streams. The STB can choose to display 1 of the four available streams simulating the eventual deployment by allowing a customer to switch channels to different sources.
All of the data is being sent/received using UDP/IP, and the company that I work for is responsible for adding FEC protection to the actual data packets so packet loss can be handled without needing retransmission.
What happens when vmstat reports CPU idle percentage dropping to very low rates (8 or 9 percent idle) is that the application is simply running out of CPU horsepower to the extent that the FEC decoding takes up too much processing and frames are dropped because the application can't pull them from the IP stacks quick enough. This is observable by watching the video output on a TV, you can see artifacting and frame jitter occurring.
The workload itself does not change, the two transmitting PCs simply loop continuously replaying (encoding and outputting) the two A/V streams, and the STB just plays what it is told to play. At this point I really don't see how the environment can affect what is occurring.
BTW: We have also done "real world" testing using a partial deployment and the behavior was first discovered there. We are running the tests in our own labs in order to attempt to identify the root cause under controlled conditions.
I also completely agree with you that we may not be tweaking the right knobs. At this point we are just attempting to theorize possible problems and test those theories by practical observations. We are not sure which "knobs" actually are the right ones to tweak.
Again, thank you very much for responding. Any insights or suggestions you can offer are greatly appreciated!