LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-16-2009, 07:53 AM   #1
mecha_tr
LQ Newbie
 
Registered: Mar 2009
Posts: 1

Rep: Reputation: 0
How to reduce used memory and dedicate processor to a single process


Hello,

I am a new user with a few questions (for the time being though).

First of all, let me try to explain what we will do with Linux:

- We have a CFD (computational fluid dynamics) tool which requires really high computing power in order to calculate basic state equations of a fluid dynamics problem. We are going to use Linux for this (actually we already use it but we need some improvements).

- The reliability of the result and the accuracy of the calculations depend on the total amount of RAM the system has.

- The speed of the calculations is highly dependent on the speed of the processor.

Now, as the hardware we have:

Processor: AMD Phenom II X4 @ 3.0 GHz
Memory: 4x2 GiB (the maximum amount the motherboard allows)

As the Sotfware:

GNU/Linux Ubuntu 64 bit Version 8.10 for AMD64

Our Issue:

1- We want to reduce the total amount of memory the system uses for other applications (it uses 337 MiB for the time being and every single byte is important for us).

2- We want to dedicate at least, if not all, 75% of the cpu resources to this CFD program to speed up the process. The cpu has 4 cores and 75% means that we want at least 3 cores to be dedicated to this program. For the time being, Linux uses only 1 processor in 100% capacity (which is equivalent to 25% overall).

The first issue is relatively simple for most intermediate or advanced Linux user yet the second issue is, I guess, rather complicated and needs the opinion of a developer.

Any opinion is wellcome and thanks to any helper in advance.

Bora
 
Old 03-16-2009, 08:21 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 14,839

Rep: Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822Reputation: 1822
Quote:
Originally Posted by mecha_tr View Post
For the time being, Linux uses only 1 processor in 100% capacity (which is equivalent to 25% overall).
Sounds like your program is single threaded - if that is the case, reserving 3 cores for it will make no difference; it will only use the one.
If you have the opportunity to make it multi-threaded, you can use cgroups to create a container for that process with 3 CPUs/cores - all other work can be isolated to the other core. I use cgroups for benchmarks that I expect to "burn" the system, and I can reserve a CPU so I can monitor what's happening and not be "locked out".
See ../Documentation/cpusets.txt (the old name for this) and ../Documentation/cgroups/cgroups.txt
 
Old 03-17-2009, 04:09 AM   #3
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,053

Rep: Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881
Quote:
Originally Posted by mecha_tr View Post
1- We want to reduce the total amount of memory the system uses for other applications (it uses 337 MiB for the time being and every single byte is important for us).
If by ubuntu, you mean something with a Gnome user interface, then getting rid of Gnome in favour of a lighter GUI (or no Graphical UI) would be a good move in reducing memory usage. It would also be a good idea to go through the services that are running and check whether any of those can be left un-run.

However, note that 'memory usage, but not particularly active' shouldn't be hurting you much. Assuming that you have swap, it should be swapped if there is pressure on memory usage and it should largely stay there.

What evidence do you have that the memory is an issue? Commonly people look at memory in use and conlude that they have a problem when linux is using its 'haven't anything better to do with this ram' algorithm and would swap, if helpful.

Quote:
The reliability of the result and the accuracy of the calculations depend on the total amount of RAM the system has.
I assume that you must mean that you can use a finer mesh (and still fit it into ram rather than using swap), if you have more ram. Otherwise, I can't see how the amount of RAM affects the accuracy and reliability.


Quote:
The first issue is relatively simple for most intermediate or advanced Linux user yet the second issue is, I guess, rather complicated and needs the opinion of a developer.
This (using several cores on a multi-cored cpu with a single-threaded app) is the exact opposite of trivial currently. There are ways of making progress, but don't think its as simple as 'oh, we've just got to throw the use multithreaded switch on the compiler' and it will happen, unless the app has already been specifically architected in this way.

In the interim, note that this may mean that there is not a dramatic slowdown in running several simulations simultaneously on the same processor (assuming that you have the memory, of course, which it sounds as if you don't currently, and also assuming that you aren't I/O limited) as might be advantageous if you were doing gentic algorithms and/or Monte Carlo / Tagucchi for optimisation over many runs.
 
Old 03-17-2009, 03:46 PM   #4
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1285Reputation: 1285Reputation: 1285Reputation: 1285Reputation: 1285Reputation: 1285Reputation: 1285Reputation: 1285Reputation: 1285
To me it looks like there may be a misunderstanding or several.

Post the output of free and you'll get something like:
Code:
bash-3.1$ free -m
             total       used       free     shared    buffers     cached
Mem:          1994        576       1417          0          0        309
-/+ buffers/cache:        266       1727
Swap:         1945          0       1945
The amount of true free RAM here is 1727 with 266 used. Is that true for you ? The rest for me is being used for disk cache which will be freed when a program needs it. To use less RAM, use a lighter window manager and file manager and programs, or just forget about using a window manager and use the command line.

syg00 is right about the multithreading, if your program is not multithreaded then it will always only be able to use 1 core or processor. You may need to rewrite the program.

If you have not already done so, compile a kernel specifically for the processor, choose 'No Forced Preemption (Server)', Disable 'Preempt The Big Kernel Lock', 'Timer frequency = 100 Hz', which will give you maximum processing power and lowest latency. You may have already done this.

Last edited by H_TeXMeX_H; 03-17-2009 at 03:47 PM.
 
Old 03-17-2009, 03:53 PM   #5
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Squeeze 2.6.32.9 SMP AMD64
Posts: 3,275

Rep: Reputation: 126Reputation: 126
Here's still another take. Given your application, computational fluid dynamics, I'm surprised it's not already multi-thread capable. If it is, and if there is a way to tell it to use more threads, I think you will be surprised at just how well Linux's CPU usage works without your getting involved.
 
Old 03-17-2009, 04:35 PM   #6
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Quote:
Originally Posted by mecha_tr View Post
We have a CFD (computational fluid dynamics) tool
Your main problem is that the program is single threaded.

That is a hard problem to redesign as multi-threaded. Is the work of changing the application to multi-threaded being done by the engineer who wrote the original code? Or by a more skilled engineer with access to all the original source code, with enough ability to understand and improve on the original design?

If "no" to both of those, then forget multi-threaded. The application will remain single threaded.

Quote:
The reliability of the result and the accuracy of the calculations depend on the total amount of RAM the system has.
I think you are mistaken. If you are not mistaken, the tool includes some explicit check for available memory and adjusts its algorithm based on the results. If you have the source code, find that explicit check and tell us what method the program uses to measure available memory. There are a number of possibilities and depending on the method used, so correction is probably appropriate (or at least worth understanding).

Quote:
Processor: AMD Phenom II X4 @ 3.0 GHz
Memory: 4x2 GiB (the maximum amount the motherboard allows)
If better memory utilization would really help, you are considering spending a lot of effort for a tiny improvement. The cost of a better motherboard and ram would be lower for more benefit.

Quote:
GNU/Linux Ubuntu 64 bit Version 8.10 for AMD64
How was the application program compiled?

Quote:
1- We want to reduce the total amount of memory the system uses for other applications (it uses 337 MiB for the time being and every single byte is important for us).
I think you are mistaken. I don't believe the last 337MiB matters to you on an 8GiB system.

Quote:
Linux uses only 1 processor in 100% capacity (which is equivalent to 25% overall).
So it isn't swapping at all now.

Quote:
The first issue is relatively simple for most intermediate or advanced Linux user
Hardly.

Quote:
yet the second issue is, I guess, rather complicated and needs the opinion of a developer.
Borderline impossible.

If you don't have the source code to the application, you're not going to improve anything by tweaking your Linux system.

If you have the source code, I would start by using Oprofile to see where the code spends its time. Other profilers give you more information but they distort the information, often quite a lot. Plus they are harder to use (not that Oprofile is all that easy).

You may have a lot of room for improvement just by improving the code in the hot spots. You might also think of a way to locally multithread just in some hot spots.

As long as your application has one to three active threads on a four core system, you don't need anything complicated to make sure it gets the maximum share of CPU time it can use. Just give it a higher priority than anything else using significant CPU time. If you manage to multi-thread to more than three useful threads, then you can revisit the then complicated question of how to give it most of the CPU power without locking up the system so you can't even look around.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how can two process can be scheduled in two different processor in multi processor ? bishalpoudyal Programming 4 08-31-2006 03:22 PM
share cpu time and reduce process load sailu_mvn Linux - Software 3 03-09-2006 07:20 PM
Addressing 36-bits of memory from single process using Xeon processor clustopher Slackware 1 02-24-2005 11:37 AM
How to get it known what processor and memory resources using by the concrete process ukrainet Linux - Newbie 3 12-20-2004 07:41 AM
Reduce memory to install? Put back after? What? pvdl Linux - Hardware 5 10-15-2004 01:46 AM


All times are GMT -5. The time now is 10:31 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration