I got 16GB ram in my shiny new box, but since the application Im running only supports 32bit os I have to run PAE to access all of it (porting it to x86_64 will probably take forever)
After a while it seams that Im running out of lowmem. Ive tried various kernels/kernelconfigs from Fedora7 default to 2.6.23.
It starts fine with 500mb free lowmem but after some hours it has worked itself down to 9mb free, then it varies between 7mb and 9mb for the remainder of the day until i oom-killer starts killing all my processes. As for highmem I dont use more than a few hundred megs, the rest becomes disk cache.
Any experience, any remedies, any tricks I might want to try?

Linux is very tricky in how it handles memories. Linux will cache processes in memory, even after you unload them. It will then free it up when it has to use memory. So don't be alarmed when you run free -m and notice that your physical memory is low.

To really make sure, is your swap being used? If the answer is no, then you are perfectly fine.

We have the same behavior on our RHEL servers. They all look like the entire memory is used. The swap usage is the key here. No concerns unless some swapping occurs.

It looks like I need to clearify.
The server crashes with the oom killer killing all processes one by one, while I still have heaps of memory left (most allocated as cache but those are irrelevant) what I miss is lowmem (free -l shows it)

I found a hint that it should be possible to run a x86_64 kernel on a i386 fedora install. Any details on this? (google is unhelpful)