LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 06-11-2013, 07:18 PM   #1
dvitek
LQ Newbie
 
Registered: Jun 2013
Posts: 2

Rep: Reputation: Disabled
throttling load to available capacity


Hi all,

I am an engineer of an application that solves a computationally expensive problem, and can do so in a highly parallel fashion. A user can basically specify that it should run P processes to solve the problem, and off it goes. Each process is likely to essentially tie up one core and use several hundred megabytes of physical memory. If too many processes are started, the system can thrash and performance suffers.

The trouble is, we really want to automatically determine how many processes to run instead of making users guess P. We should react to dynamically changing conditions on the machine, creating and destroying processes as needed. The resource needs of the application should grow to the capacity of the machine, without negatively impinging on other applications running on the system.

Furthermore, we would like to have a reasonable degree of fairness. If a user runs multiple instances of our application, or other applications with similarly elastic resource requirements, resources should be split roughly evenly (assuming all applications are striving to be socially-minded in this fashion). For example, the first application to spin up should not get all the resources.

We have solved this problem in a simplistic manner by inspecting available memory and the number of logical cores on the system. However, I think there is a lot of room for improvement, and I'm wondering how well we can really do without OS assistance. How much of the file system cache should we assume is "up for grabs?" It seems to me that the OS is in the best position to judge whether there is too much memory pressure for another process. I suppose we could reactively exit if our application starts incurring too many page faults, but that seems awfully reactive and I can imagine loopholes.

There is a more detailed write-up of what we ended up doing here:
http://www.grammatech.com/blog/pleas...physical-pages

Since writing that post I have discovered cgroups: http://en.wikipedia.org/wiki/Cgroups
They don't really fit the bill, if I understand the article correctly.

Mac OSX has a feature that helps a little:
http://developer.apple.com/library/i...P40009251-SW13
Basically, a process can declare itself expendable, meaning the OS can kill it at will. Not sure how well this works in practice.

Bear in mind that we have little control over the computers the application gets deployed or run on, so one must assume a healthy ecosystem of competing resource demands outside our control.

With the prevalence of cloud computing and highly parallel applications, other developers must have run into this sort of issue. What has and has not worked? How can we improve on the status quo? How would you go about solving this issue? Discuss.
 
Old 06-12-2013, 04:48 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,431

Rep: Reputation: 1055Reputation: 1055Reputation: 1055Reputation: 1055Reputation: 1055Reputation: 1055Reputation: 1055Reputation: 1055
Long ago in a Linux-verse that actually cared about the possible usefulness of class-based resource management, ckrm started to evolve. I installed some of the early code releases.

It was put to the sword by the proponents of "containers". cpusets, later to become cgroups developed from the remnants. The original idea had many potential excellent solutions to (user controlled) overall system management. cgroups, while useful lacks the dynamic controls you (and I) desire methinks. Even this late in the game.
 
Old 06-12-2013, 08:04 AM   #3
thedaver
Member
 
Registered: Jan 2010
Posts: 65

Rep: Reputation: 21
You could partition your servers into virutal compute nodes and let the users run without restriction within the resources allocated to them. The nodes could be of various sizes (small, med. large) and the user could login or be allocated a node based upon their perceived need or size of compute task.

This is largely the thinking around many cloud compute node options as well, you'd just be in direct control and could route/gate/control the use of the nodes through some internal workflow/queue (perhaps paper-based). Or just say "one login per VM" and people can just take what's available.

As to how many processes should be launched within the VM, I suppose you could use some simple logic like counting the number of (virtual) cores exposed within the VM and respond with sparking up an appropriate number of compute threads for your app.

Disclaimer: The above advice may be total crap.
 
Old 06-12-2013, 10:29 AM   #4
dvitek
LQ Newbie
 
Registered: Jun 2013
Posts: 2

Original Poster
Rep: Reputation: Disabled
thedaver: A good idea for many scenarios. Unfortunately, our software runs on machines we don't control or administer. Most users aren't looking to do any configuration -- we're looking for something that could be done in a fully automated fashion without superuser privileges or asking users to do more work.

syg00: Thanks for the info.
 
Old 06-12-2013, 11:44 AM   #5
thedaver
Member
 
Registered: Jan 2010
Posts: 65

Rep: Reputation: 21
Then think about a slider-bar or tunable user-facing parameter. "How badly should I crush your computing resources? 1) friendly, 2) rough, 3) no room for others"

Then use information about cores/RAM or other resources to make a determination of how friendly you want to be relative to other applications.

Frankly I'm not sure I visualize why something with such elastic computing elegance would be relegated to running in a situation where resource contention with "other applications" was this big a consideration. Perhaps that's just an aside. Haven't read your links to understand the circumstances, maybe my fault there.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
about throttling Gyanendra Dhakal Linux - Newbie 2 05-24-2013 11:20 AM
Interrupt Throttling patsylin Linux - Software 0 06-07-2010 09:59 AM
throttling fakie_flip Linux - Server 1 12-01-2007 10:50 AM
CPU throttling lcaley Linux - Hardware 14 01-06-2007 06:14 AM
Bandwidth Throttling bg_27 *BSD 3 07-22-2003 09:09 PM


All times are GMT -5. The time now is 07:44 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration