Hi all,

I am an engineer of an application that solves a computationally expensive problem, and can do so in a highly parallel fashion. A user can basically specify that it should run P processes to solve the problem, and off it goes. Each process is likely to essentially tie up one core and use several hundred megabytes of physical memory. If too many processes are started, the system can thrash and performance suffers.

The trouble is, we really want to automatically determine how many processes to run instead of making users guess P. We should react to dynamically changing conditions on the machine, creating and destroying processes as needed. The resource needs of the application should grow to the capacity of the machine, without negatively impinging on other applications running on the system.

Furthermore, we would like to have a reasonable degree of fairness. If a user runs multiple instances of our application, or other applications with similarly elastic resource requirements, resources should be split roughly evenly (assuming all applications are striving to be socially-minded in this fashion). For example, the first application to spin up should not get all the resources.

We have solved this problem in a simplistic manner by inspecting available memory and the number of logical cores on the system. However, I think there is a lot of room for improvement, and I'm wondering how well we can really do without OS assistance. How much of the file system cache should we assume is "up for grabs?" It seems to me that the OS is in the best position to judge whether there is too much memory pressure for another process. I suppose we could reactively exit if our application starts incurring too many page faults, but that seems awfully reactive and I can imagine loopholes.

There is a more detailed write-up of what we ended up doing here:
http://www.grammatech.com/blog/pleas...physical-pages

Since writing that post I have discovered cgroups: http://en.wikipedia.org/wiki/Cgroups
They don't really fit the bill, if I understand the article correctly.

Mac OSX has a feature that helps a little:
http://developer.apple.com/library/i...P40009251-SW13
Basically, a process can declare itself expendable, meaning the OS can kill it at will. Not sure how well this works in practice.

Bear in mind that we have little control over the computers the application gets deployed or run on, so one must assume a healthy ecosystem of competing resource demands outside our control.

With the prevalence of cloud computing and highly parallel applications, other developers must have run into this sort of issue. What has and has not worked? How can we improve on the status quo? How would you go about solving this issue? Discuss.

Long ago in a Linux-verse that actually cared about the possible usefulness of class-based resource management, ckrm started to evolve. I installed some of the early code releases.

It was put to the sword by the proponents of "containers". cpusets, later to become cgroups developed from the remnants. The original idea had many potential excellent solutions to (user controlled) overall system management. cgroups, while useful lacks the dynamic controls you (and I) desire methinks. Even this late in the game.

You could partition your servers into virutal compute nodes and let the users run without restriction within the resources allocated to them. The nodes could be of various sizes (small, med. large) and the user could login or be allocated a node based upon their perceived need or size of compute task.

This is largely the thinking around many cloud compute node options as well, you'd just be in direct control and could route/gate/control the use of the nodes through some internal workflow/queue (perhaps paper-based). Or just say "one login per VM" and people can just take what's available.

As to how many processes should be launched within the VM, I suppose you could use some simple logic like counting the number of (virtual) cores exposed within the VM and respond with sparking up an appropriate number of compute threads for your app.

Disclaimer: The above advice may be total crap.

thedaver: A good idea for many scenarios. Unfortunately, our software runs on machines we don't control or administer. Most users aren't looking to do any configuration -- we're looking for something that could be done in a fully automated fashion without superuser privileges or asking users to do more work.

syg00: Thanks for the info.

Then think about a slider-bar or tunable user-facing parameter. "How badly should I crush your computing resources? 1) friendly, 2) rough, 3) no room for others"

Then use information about cores/RAM or other resources to make a determination of how friendly you want to be relative to other applications.

Frankly I'm not sure I visualize why something with such elastic computing elegance would be relegated to running in a situation where resource contention with "other applications" was this big a consideration. Perhaps that's just an aside. Haven't read your links to understand the circumstances, maybe my fault there.