LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Desktop
User Name
Password
Linux - Desktop This forum is for the discussion of all Linux Software used in a desktop context.

Notices


Reply
  Search this Thread
Old 04-26-2015, 06:37 PM   #1
Lsatenstein
Member
 
Registered: Jul 2005
Location: Montreal Canada
Distribution: Fedora 31and Tumbleweed) Gnome versions
Posts: 311
Blog Entries: 1

Rep: Reputation: 59
How does Linux handle programs and execution memory?


I come from the IBM VMCMS and IBM Mainframe days (MVT MFT, TSO) etc.

When we wrote programs, particularly for vmcms, the compiler always made one segment for data, and multiple ones for constants and code.

When a program was initiated, the first thing that happened was to load the data, and then for the constants and code, determine if that segment was already loaded by another copy of the same program, or by other programs that may make use of the same segments / constant data section.

So, if there was 5 copies of a program like nautilus, for example, there would only be 5 data sections in memory and 1 copy each of the costant and code segments.

That functionality of course meant that memory consumption was optimized. In the mainframe environment the author could determine, if that a code segment in memory was deemed private, shareable by all or by group.

How does that work in Linux with Intel / Amd architecture?

If program X was comprised of 100k data, 100k constant, and 300k code segments, what would be in memory if, while program X was executing, a second program X was loaded by another user(process id)?

I would expect 1x100k data segment added to what is already in memory.

Is that what happens?
__________________
 
Old 04-27-2015, 06:33 AM   #2
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Number one is I'm not versed well enough to answer your question fully and describe in detail the sequencing and manners in which memory is allocated for Linux programs when they run. However I'll offer some references as well as my opinions for what it's worth.

There are processes and there are threads. My impression is that any given process owns all its memory and does not share that memory for re-use or duplication purposes such as you describe where the memory model re-uses constants as a method to efficiently manage memory.

Processes are an entire program's copy of every resource it requires to run.

Even if you fork() to create a child process, the child starts as a complete copy of the parent process with the minor exceptions that the identify of the process itself and the knowledge of parent/child is different between the parent and the child - being that the parent knows it has a child, it also knows the higher parent which created it and meanwhile the child knows who it's parent is. Otherwise the programs are identical in resources, however they are copies and therefore occupy unique memory spaces.

Threads have typically been called "lightweight processes" where they are copies of the process resources, but things like open files are direct re-uses. In other words, things which are mostly public are defined as shared resources between threads. There are very specific, but also understandable descriptions of the exactness of all of this, and I'd recommend you just do some general web searching to see those distinctions for yourself.

Below are some links for processes, threads, and thread vs process comparisons. I think the process description is very inclusive and will tell you a lot about how memory is managed. However I also think the thread vs process summary I happened to find is helpful but there are probably better descriptions which can relate the exact memory details of what is shared versus what is private per thread ID.

I think the bottom line (or my bottom line) here is that my opinion is that the Operating System does not do similar things to what you're speaking about. Instead it makes unique resource allocations per process and the use of threads was less about memory efficiency versus the need to have interprocess communications. In that you will notice that the many forms of IPC (also a good read for Linux) include things like sockets, shared memory, threads, pipes, and a few other techniques.

http://www.tldp.org/LDP/tlk/kernel/processes.html
http://www.tldp.org/FAQ/Threads-FAQ/index.html
http://www.thegeekstuff.com/2013/11/...s-and-threads/
 
Old 04-27-2015, 07:17 AM   #3
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Modern OS's, including Linux, have "anonymous" and non anonymous memory. Non anonymous memory is "mapped" from files and brought into ram by demand paging. Anonymous memory is created in ram and might be paged out to the swap space.

Non anonymous memory is typically shareable, meaning any pages brought into memory by demand paging the file can be shared among any processes mapping the same file.

In terms of the original question, most code and constants will be mapped and shareable. Most writeable data in most processes will by anonymous.

Quote:
Originally Posted by Lsatenstein View Post
If program X was comprised of 100k data, 100k constant, and 300k code segments, what would be in memory if, while program X was executing, a second program X was loaded by another user(process id)?
In 4Kb chunks, whichever parts of that 100k of constants have been referenced by one of the two processes would be in ram once (shared if both processes have accessed it). The same is mostly true of the 300k of code. But some 4kb pages of some code need load time fixups, which I think aren't shared even when identical, and if the code is in .so files the fixups typically aren't identical across processes sharing the code.

The 100k of data is created as "demand zero" (meaning it doesn't really exist in ram) or in some cases "copy on write" mappings when the program starts. Each 4Kb page of "demand zero" or "copy on write" memory that is written to is allocated as a real page of ram the first time it is written to.

Quote:
Originally Posted by rtmistler View Post
My impression is that any given process owns all its memory and does not share that memory for re-use or duplication purposes such as you describe where the memory model re-uses constants as a method to efficiently manage memory.
Each process has its own mapping, which acts like it (the mapping tables themselves) are not shared. But even parts of the mapping tables can be shared when the OS can make shared mapping tables act as if they are not shared. Similarly, "demand zero" and "copy on write" mappings are shared in a way that lets the OS make it appear to the process that they are not shared (including converting to actually not shared when necessary to preserve the illusion that it never had been shared.

But there are also parts of the address space that are explicitly shared across processes.

Quote:
my opinion is that the Operating System does not do similar things to what you're speaking about.
Apologies if I'm landing on the wrong side of the tradeoff between avoiding misinforming the OP and avoiding being nasty to others who answer, but ...

These are much more matters of fact than opinion. I might have some of the subtle details incorrect in what I said earlier, but you have the basic facts much more fundamentally wrong. What Linux does is similar to (though more complicated than) what the OP described.

In modern systems, the amount of anonymous data has grown more than the amount of code and constants, and typical amounts of physical ram has grown even more than that. So all that complicated code and constant sharing is becoming a minor efficiency feature, rather than a fundamental part of what makes a Linux system usable. But it is all still in there.

Last edited by johnsfine; 04-27-2015 at 07:41 AM.
 
Old 04-27-2015, 05:54 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,131

Rep: Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121
I might quibble with the last paragraph.
KSM and z{ram,swap,cache} show that this is still an active area of interest and development in the kernel. All could be viewed as extensions to what the OP asked, but have an influence on which memory pages are ultimately shared.
Large pages throws another spanner in the works as well.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
errors in execution of wireless programs naren123.k Linux - Software 1 02-23-2011 05:03 AM
How to control execution of programs in a shell script. Chrisantha1 Programming 5 04-28-2010 03:29 AM
Need to build a high-end number cruncher - how much memory can Linux handle? MamaWombat Linux - Hardware 3 06-07-2006 08:16 PM
how does linux handle memory allocation? nodger Programming 4 04-17-2004 10:10 PM
Linux memory allocation to programs mlaudu Linux - Software 1 03-29-2004 04:25 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Desktop

All times are GMT -5. The time now is 10:05 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration