Virtual Memory Clarifications

tripialos · 11-29-2016, 06:19 AM

I have been reading about Virtual Memory but there are a few things i don`t understand which i will try to demonstrate with an example.

I have created the bellow small c program which allocates 50Mb of memory using malloc() and then using a loop to "touch" every page in order to make it residence.

Code:

int main (){

    // Map 50M to RAM
    unsigned char *p = malloc(52428800);
    sleep(5);
      
    // Touch every page 
    for (int i = 0; i< 52428800; i+=4096)
        p[i] = 1;    
    sleep(100000);
}

Indeed the program seems to work, meaning that after the first 5 second sleep the resident memory on the "top command" output starts to fill up and eventually allocating all Virtual Memory to RAM.

Code:

 PID %MEM    VIRT    RES   CODE    DATA    SHR                                             
32486  0.6   55396  52360      4   51528   1104

I noticed the page faults of the program and there are only minor ones:

Code:

ps -ef -o min_flt,maj_flt 32486 

 MINFL  MAJFL
 12879      0

Shouldn't there be major page faults? As far as i understood, when i use malloc(), a virtual address space of 50Mb is created. Prior writing on each virtual page, the actual residence size is very small, but after is equal to the virtual memory requested.

When i "touch" the pages (in order to make it residence), the each page, is moved from disk to DRAM, right? Why there aren't any Major Page Faults then?

Also, when you malloc() for 50m and you noticed at the residence size, there are only a few Kbytes, where are the rest pages? are they on the disk?

arre · 11-29-2016, 02:38 PM

Hi,

Cool that you're trying to figure out something as interesting as virtual memory in linux.
Anyway, don't have much time to post now, but the first link I hit in google already pretty well explains it:

https://www.quora.com/What-is-the-di...fault-in-Linux

I think the clue here is that when you malloc stuff, there kernel promises to deliver it, but does not "create" the memory anywhere. There is nothing stored yet in the allocated memory, so it could just as well not be anywhere. Once you start writing to it, linux -has- to start keeping your data somewhere. It maps the virtual memory space of the relevant page onto something in physical ram, and writes to it. I could be wrong here, but this could be coined as miner page faults, since the first access to the memory is not immediate : a mapping still has to be made. (but this is fast, since no disk I/O is involved, so minor).

Only when you would go about and write to so many pages that your ram would get pretty full, linux would start deciding its a good idea to move some of the ram to the swap file (or disk). That frees up space for the current memory activity, but.. if another application then would try to read/write to its page, it might not be a fast operation (since the disk has to be read, copied into ram, and then the operation can be done). This is expensive, and afaik this is your hard fault.

Hope this helps a bit
Arnout

syg00 · 11-29-2016, 05:29 PM

Pretty much. You will drive yourself nuts trying to understand memory management too deeply ...

You are using anonymous memory, so nothing exists "on disk" until you save it somewhere. Minor page faults can also be taken on data that has been swapped out if the page hasn't been re-used by another process.
And pages can be compressed and de-duplicated, so all those pages can be represented by one "real" page ...

This gets real complicated real quick.

sundialsvcs · 11-29-2016, 06:51 PM

No page-ins are occurring because nothing has been paged-out. You're simply populating the page and segment tables associated with your virtual memory space. When, and if, it becomes necessary to "steal" pages from you, or if this memory space becomes inactive and gets swapped out, then space will be allocated in backing storage and the data will be written out.

AwesomeMachine · 11-29-2016, 11:17 PM

Linux appears to use a lot of memory, but it uses all it can as a buffer. If actual processes require more memory, the kernel just overwrites ram it was using to buffer. There's more intelligence to how the kernel evaluates which parts of the buffer space to eliminate, but you get the basic idea.

tripialos · 11-30-2016, 04:31 AM

Quote:

Originally Posted by arre

Hi,

Cool that you're trying to figure out something as interesting as virtual memory in linux.
Anyway, don't have much time to post now, but the first link I hit in google already pretty well explains it:

https://www.quora.com/What-is-the-di...fault-in-Linux

I think the clue here is that when you malloc stuff, there kernel promises to deliver it, but does not "create" the memory anywhere. There is nothing stored yet in the allocated memory, so it could just as well not be anywhere. Once you start writing to it, linux -has- to start keeping your data somewhere. It maps the virtual memory space of the relevant page onto something in physical ram, and writes to it. I could be wrong here, but this could be coined as miner page faults, since the first access to the memory is not immediate : a mapping still has to be made. (but this is fast, since no disk I/O is involved, so minor).

Only when you would go about and write to so many pages that your ram would get pretty full, linux would start deciding its a good idea to move some of the ram to the swap file (or disk). That frees up space for the current memory activity, but.. if another application then would try to read/write to its page, it might not be a fast operation (since the disk has to be read, copied into ram, and then the operation can be done). This is expensive, and afaik this is your hard fault.

Hope this helps a bit
Arnout

Thanks for your reply. It does help a lot. So the case with the small program i created is some sort of COW (copy on write)? The address space is created but it is not yet mapped since there isn't anything to be done. When i start writing this is when actually the Virtual Pages are mapped to Physical ones. Is this correct?

Quote:

Originally Posted by sundialsvcs

No page-ins are occurring because nothing has been paged-out. You're simply populating the page and segment tables associated with your virtual memory space. When, and if, it becomes necessary to "steal" pages from you, or if this memory space becomes inactive and gets swapped out, then space will be allocated in backing storage and the data will be written out.

Are the Major Page Faults and Swapping the same "thing"? I am trying to make a distinction between those two terms. Swapping is the process for moving pages from RAM to disk and vice-versa right? Swapping most usually occurs when RAM is full and space is needed for a process. Now when a Major Page fault occurs, a page from disk is moved to the RAM and vice versa . So we are basically talking about the same thing right? On the other side, a major page fault can occur without the memory been full. For example, a page was not used for a long time so it was paged out on the disk, then after a while iy was paged in again causing a major page fault despite the fact memory was not full.

Is a major page fault the event, and swapping is the technique to deal with it?
When your system is swapping that means it suffers from page faults?

Dam I am so confused!!