Kernel Memory Management
Hi all,
I haven't posted here before, but I will now be frequenting the board a lot, as I am somewhat inexperienced in Linux and am constantly finding that this board is a great resource with a ton of great knowledge.
I recently took a job as a Software Engineer, however, a large portion of my job entails a lot of systems administration work.
With that in mind,the company that I work for currently has a number of different software titles that rely heavily on memory. These applications are fairly graphic intensive and do a lot of 3D image manipulation, thus they are constantly writing large data sets into physical memory, and even more so, accessing the data in memory.
We are running Red Hat 8 on an intel x86 architecture with 1 GB of DDR 400 RAM, configured to support dual channel data transmission, and an intel D865 board.
There is an application that I have been testing extensively in order to maximize performance. As more image data is loaded into this application, the process grows larger and larger in memory. The strange deal here, is when the process reaches ~256 MB in size, the performance of the application takes an exponential decline. To be more specific, when a request is made to manipulate some image data, the performance goes from approximately 1-2 seconds of rendering time down to more like 4-8 seconds of rendering time.
There appears to be somewhat of a threshold, thats why I mention the ~256 MB mark. When the process is 240 MB, rendering time is fine, approximately 1-2 seconds. Add one more data set to the mix, to make the process more like ~265 MB in size and we see this huge performance decrease.
I am almost positive this is not a hardware issue, I have tested different boards, CPUs, different RAM sticks, etc... Another strange addition to the problem, is that when the application is run on a system with 2GB of memory, this exponential decline in performance goes away completely. Now, I know what you're all thinking, and that is that the Kernel is swapping this data out to disk. Here is the catch, there is no swapping going on at all. I have tested this extensively as well, and the Kernel does not start swapping until an inordinate amount of image data is loaded. At which point, rendering time is much worse at 12-15 seconds.
My final conclusion is that this is an issue of the way the Kernel is managing memory. I believe this is the case because I recompiled the Kernel such that Highmem was disabled (/proc/meminfo then shows ~130 MB less in RAM). When disabling Highmem, this exponential decrease in performance goes away, and there is a nice linear decline in performance as more image data is progressively loaded into this application, which is what I should expect to see.
My thoughts on this are that when disabling Highmem, we are taking away ZONE_HIGHMEM (896MB and above) completley and forcing the Kernel to exclusively use ZONE_NORMAL (16-896MB) as its sapce for memory allocation and since on the x86 platform, the Kernel sort of folds ZONE_HIGHMEM onto ZONE_NORMAL to effectively create 1 zone for memory (that is how I understand it).
Now that all of that is out of the way, my question is this: Does anyone know if there are some more tunable parameters that I can play with to adjust the way the Kernel is paging or managing memory? I am confident that somewhere, there is some sort of configuration file or something that I can change to optimize performance. Modifying the source is my preferred last resort, as the source code for this is pretty nasty looking. Maybe there is some way to increase the efficiency of the TLB?
I could just leave Highmem disabled, but then I lose all of my upper-memory (~130 MB worth). Any help would be greatly appreciated.
Also, I am running the 2.4.21 Kernel.
Thanks a lot,
Meat
|