LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-08-2016, 09:45 AM   #46
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513

You should realize that when a program accesses other portions of memory, the first one may be swapped out/dropped from physical memory. The algorithm is USUALLY least-recently-used (LRU) as that tends to provide the most optimum
use of physical memory.

You can always use the "top" utility to show a snapshot of the state of the system. "VIRT" column shows how much virtual memory is being used by the process, the "RES" column shows how much is actually in memory... but you have no control of which physical pages are used for what.

It is very rare (to never) that the entire virtual memory range resides in physical memory. The only time I can think of when that can happen is when the program is less than 4K, and uses less than 4K for data (which requires no runtime libraries...). Paging is used to exchange currently idle memory pages to use for pages containing active accesses. This does mean the virtual memory map gets changed to do that - but that is what paging is for. It makes a limited resource (RAM) appear to be more unlimited.

As shown in the reference I gave, the memory is NOT in any particular order - pages of physical memory get re-used/remapped/dropped as system activity continues.

https://www.tutorialspoint.com/opera...management.htm

Last edited by jpollard; 10-08-2016 at 09:51 AM.
 
Old 10-08-2016, 12:20 PM   #47
Thirdeyematt
Member
 
Registered: Aug 2016
Posts: 31

Original Poster
Rep: Reputation: Disabled
So the program that fills up the virtual memory space in Windows with one image file also has the power to open and view each image file. So you have the power to select how many to open, and then you can load them all into the virtual memory space instead of actually opening them... The files are put into virtual memory before it actually opens the image files... Could it be that the program is temporarily storing the files to open them up when asked? This way those files will all stay in the virtual memory as long as I want them to.
 
Old 10-08-2016, 02:19 PM   #48
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by Thirdeyematt View Post
So the program that fills up the virtual memory space in Windows with one image file also has the power to open and view each image file. So you have the power to select how many to open, and then you can load them all into the virtual memory space instead of actually opening them... The files are put into virtual memory before it actually opens the image files... Could it be that the program is temporarily storing the files to open them up when asked? This way those files will all stay in the virtual memory as long as I want them to.
Nope. there isn't enough memory to "load them all". assuming a 1280x1024 image that would be 3932160 bytes (1280x1024*3) - about 3.73MB each. 200,000 copies would make that 74000MB (or 74GB). I'm quite sure you don't have that much.

Paging is a technique to copy data into/out of memory as needed. The files are simply mapped to your virtual address. The only thing required is to adjust the list of virtual pages. Since the file is memory mapped, all that happens is that duplication of entries in the process memory map, resulting in physically only needing 3.7MB of actual memory (when used) and storage for the memory map (one entry for each 4K page, about 16 bytes - actual number bytes used may be larger).

When a given page is referenced, if that page happens to be in memory, then everything continues.
If a given page is NOT in memory, then a page fault occurs. The kernel then examines the memory map to identify which page needs to be copied to memory; looks to see what page in physical memory is available, and if none, identifies pages that are not modified flags any memory maps that refer to it marking it "not available", saves the page on disk - then reads the desired data (specified by the memory map) into the free page, modifies the original process memory map to indicate in memory, and which page it is, and then resumes/restarts the instruction.

Memory maps allow processes to be MUCH smaller than would otherwise be needed.

You really need to learn how memory is used. You can start with the references I've already provided.

Last edited by jpollard; 10-08-2016 at 02:22 PM.
 
Old 10-09-2016, 09:21 AM   #49
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,274

Rep: Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696
Quote:
the program that fills up the virtual memory space in Windows with one image file also has the power to open and view each image file.
Just one comment: it is not really correct. The file (image) you opened will be put into memory. If you want to see it (to be displayed) it must be in the main memory, because the viewer program you use need it. If you do not want to see it (actually you opened another image) the file may be put into virtual memory, but it will be copied back to the main memory if you want to use that.

The handling of memory and files are different on windows and linux, but in general the runnning/living program and data must be available in the main memory, the virtual memory cannot be accessed directly.
 
Old 10-09-2016, 10:09 AM   #50
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Just a comment on the comment :-)

Quote:
Originally Posted by pan64 View Post
Just one comment: it is not really correct. The file (image) you opened will be put into memory. If you want to see it (to be displayed) it must be in the main memory, because the viewer program you use need it. If you do not want to see it (actually you opened another image) the file may be put into virtual memory, but it will be copied back to the main memory if you want to use that.
Even the image doesn't have to be in memory for the display - as it may have its own memory to contain the data being displayed. The computer only needs the piece it is passing to the video card. This happens more often with large images. I use to process satellite images - which would be 4K by 12K covering from about latitude 72 degrees north to about latitude 15 degrees north. The computer only needed one raster line in memory at a time. It was only the video card that had the entire image. If I was rescaling the image data then I might have 3 or 5 raster lines (or parts of raster lines...).

Other issues are that image files tend to be compressed to save space - thus the "image file", even in computer memory, is not the same (video files are severely compressed). Jpeg files have various levels of compression applied.

Only raw image files have no applied compression, usually each pixel is represented by three one byte unsigned integers (one for red, green and blue). Other image files may use YUV (the Y is the brigtness, the UV determine the color) or HIS representation (Hue, Intensity and Saturation). Each format has its own advantage/disadvantages.
The RGB raw form is easiest and fastest for conversion to a display. I think the YUV is easier for color printing. HIS is good for various other processing (a greyscale image can be examined strictly from the intensity). It all depends on the image file format.

Quote:
The handling of memory and files are different on windows and linux, but in general the runnning/living program and data must be available in the main memory, the virtual memory cannot be accessed directly.
In main memory for the instance an instruction that is manipulating the data, or it is being passed to another peripheral. Otherwise it doesn't need to be there.
 
Old 10-09-2016, 05:01 PM   #51
Thirdeyematt
Member
 
Registered: Aug 2016
Posts: 31

Original Poster
Rep: Reputation: Disabled
So are you saying that I will not be able to fill up the virtual memory with one file? I know it can be done because I already have a program to do this... I'm sorry if I'm not understanding I only know basic entry-level programming
 
Old 10-09-2016, 05:46 PM   #52
JeremyBoden
Senior Member
 
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,950

Rep: Reputation: 513Reputation: 513Reputation: 513Reputation: 513Reputation: 513Reputation: 513
You don't understand.
Time to give up.
 
Old 10-09-2016, 08:12 PM   #53
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by Thirdeyematt View Post
So are you saying that I will not be able to fill up the virtual memory with one file? I know it can be done because I already have a program to do this... I'm sorry if I'm not understanding I only know basic entry-level programming
THINK ABOUT IT.

A virtual address on most x86 machines is 64 bits.

That is 18,446,744,073,709,551,616 bytes. You will NEVER be able to "fill it".

I've given you references for how virtual addressing works. READ THEM.
 
Old 10-10-2016, 01:46 AM   #54
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,274

Rep: Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696Reputation: 7696
Quote:
I will not be able to fill up the virtual memory with one file?
How did you deduce it? We just trying to explain how it works. It is not about programming, but how virtual memory is used at all.
Quote:
I know it can be done because I already have a program to do this
So you are ready now?
 
Old 10-11-2016, 07:11 PM   #55
Thirdeyematt
Member
 
Registered: Aug 2016
Posts: 31

Original Poster
Rep: Reputation: Disabled
Yes it did work... mount -o remount,size=9T /dev/shm ... for i in{1..200000} ; do cp Love.png /dev/shm/Love-$i.png ; done That worked thank you!!... free -t shows that all the swap is used up...

But that was for my main 10tb hard drive that runs linux. When I go to mount another tmpfs system on another external hard drive, I can create a tmpfs file system there that has a size of 4tb. However, when I load files into it, it uses the main hard drive's swap space.. Is there any way I can avoid using my main hard drive's swap space and start using the swap space of the external hard drive since the main hard drive's swap space is just about used up???

Last edited by Thirdeyematt; 10-11-2016 at 07:49 PM.
 
Old 10-11-2016, 08:39 PM   #56
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by Thirdeyematt View Post
Yes it did work... mount -o remount,size=9T /dev/shm ... for i in{1..200000} ; do cp Love.png /dev/shm/Love-$i.png ; done That worked thank you!!... free -t shows that all the swap is used up...

But that was for my main 10tb hard drive that runs linux. When I go to mount another tmpfs system on another external hard drive, I can create a tmpfs file system there that has a size of 4tb. However, when I load files into it, it uses the main hard drive's swap space.. Is there any way I can avoid using my main hard drive's swap space and start using the swap space of the external hard drive since the main hard drive's swap space is just about used up???
Nope. tmpfs uses swap. You have used up whatever memory you have. After the first is used up it will use the rest. The only way to add more memory is to add more swap.

And you still won't use up your virtual memory. The theoretical maximum is 18.1 exabytes...

You can't afford that much.

Now the actual limit is much smaller 2^48 bits of address for Intel virtual paging - 256TB. And I'm pretty sure you can't afford that much disk space either.

Last edited by jpollard; 10-11-2016 at 08:44 PM.
 
Old 10-11-2016, 08:52 PM   #57
Thirdeyematt
Member
 
Registered: Aug 2016
Posts: 31

Original Poster
Rep: Reputation: Disabled
Is there a way to extend the operating system past the first hard drive and have it take up multiple hard drives, all in the same operating system?

*Edit:
-Spanned Volume....


I will try this and report back to see if it will increase my swap space I think it should..

Last edited by Thirdeyematt; 10-11-2016 at 08:55 PM.
 
Old 10-11-2016, 08:56 PM   #58
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by Thirdeyematt View Post
Is there a way to extend the operating system past the first hard drive and have it take up multiple hard drives, all in the same operating system?
Of course. Initialize another disk as a swap partition, then add it in the fstab as more swap, or use the swapon utility to add during normal operations.

At one time the limit was 128 swap entries, but that may have been lifted.

It really doesn't matter. The only difference between having a file in swap and a file in another filesystem is just how you address it. Both are still on disk. Granted, it is much faster to delete the stuff in a tmpfs filesystem. All you have to do is reboot.

Last edited by jpollard; 10-11-2016 at 08:58 PM.
 
Old 10-11-2016, 09:12 PM   #59
Thirdeyematt
Member
 
Registered: Aug 2016
Posts: 31

Original Poster
Rep: Reputation: Disabled
Also, your talking 4tb external passport (no power supply needed, small) hd for $130, 13 port usb hub $24... power supply is 850w so the usb 2.0 should be able to fit at least 20 hopefully.. Your talking $1500 for close to 50tb that's not a fortune..

The digital image files that can be amplified is a real thing that could be purchased online. I can't really make anyone believe in them but all I can say is that they are much stronger than brainwave entrainment, which uses sounds to change the way you think.

Thank you everyone this is now solved.
 
Old 10-11-2016, 09:24 PM   #60
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by Thirdeyematt View Post
Also, your talking 4tb external passport (no power supply needed, small) hd for $130, 13 port usb hub $24... power supply is 850w so the usb 2.0 should be able to fit at least 20 hopefully.. Your talking $1500 for close to 50tb that's not a fortune..

The digital image files that can be amplified is a real thing that could be purchased online. I can't really make anyone believe in them but all I can say is that they are much stronger than brainwave entrainment, which uses sounds to change the way you think.

Thank you everyone this is now solved.
Unless you have some magic to convert an image file to something displayable and have a display device, there is no communications between the computer and you.

doesn't happen.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] swap file, virtual memory Art McClure Slackware 2 04-08-2012 11:56 AM
Where to put Bash scripts written to run at various times to examine Alsa sound probs xologist Linux - Software 3 08-13-2008 11:09 AM
Difference between resident memory,shared memory and virtual memory in system monitor mathimca05 Linux - Newbie 1 11-11-2007 05:05 AM
Information for virtual memory file system peiwai Linux - Software 1 05-01-2006 01:09 PM
RH 8.0 Mapping Virtual Memory to get access to VMIC Reflective Memory PCI card. Merlin53 Linux - Hardware 0 05-05-2003 01:50 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:14 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration