Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
|
|
10-08-2016, 09:45 AM
|
#46
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
You should realize that when a program accesses other portions of memory, the first one may be swapped out/dropped from physical memory. The algorithm is USUALLY least-recently-used (LRU) as that tends to provide the most optimum
use of physical memory.
You can always use the "top" utility to show a snapshot of the state of the system. "VIRT" column shows how much virtual memory is being used by the process, the "RES" column shows how much is actually in memory... but you have no control of which physical pages are used for what.
It is very rare (to never) that the entire virtual memory range resides in physical memory. The only time I can think of when that can happen is when the program is less than 4K, and uses less than 4K for data (which requires no runtime libraries...). Paging is used to exchange currently idle memory pages to use for pages containing active accesses. This does mean the virtual memory map gets changed to do that - but that is what paging is for. It makes a limited resource (RAM) appear to be more unlimited.
As shown in the reference I gave, the memory is NOT in any particular order - pages of physical memory get re-used/remapped/dropped as system activity continues.
https://www.tutorialspoint.com/opera...management.htm
Last edited by jpollard; 10-08-2016 at 09:51 AM.
|
|
|
10-08-2016, 12:20 PM
|
#47
|
Member
Registered: Aug 2016
Posts: 31
Original Poster
Rep:
|
So the program that fills up the virtual memory space in Windows with one image file also has the power to open and view each image file. So you have the power to select how many to open, and then you can load them all into the virtual memory space instead of actually opening them... The files are put into virtual memory before it actually opens the image files... Could it be that the program is temporarily storing the files to open them up when asked? This way those files will all stay in the virtual memory as long as I want them to.
|
|
|
10-08-2016, 02:19 PM
|
#48
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
Quote:
Originally Posted by Thirdeyematt
So the program that fills up the virtual memory space in Windows with one image file also has the power to open and view each image file. So you have the power to select how many to open, and then you can load them all into the virtual memory space instead of actually opening them... The files are put into virtual memory before it actually opens the image files... Could it be that the program is temporarily storing the files to open them up when asked? This way those files will all stay in the virtual memory as long as I want them to.
|
Nope. there isn't enough memory to "load them all". assuming a 1280x1024 image that would be 3932160 bytes (1280x1024*3) - about 3.73MB each. 200,000 copies would make that 74000MB (or 74GB). I'm quite sure you don't have that much.
Paging is a technique to copy data into/out of memory as needed. The files are simply mapped to your virtual address. The only thing required is to adjust the list of virtual pages. Since the file is memory mapped, all that happens is that duplication of entries in the process memory map, resulting in physically only needing 3.7MB of actual memory (when used) and storage for the memory map (one entry for each 4K page, about 16 bytes - actual number bytes used may be larger).
When a given page is referenced, if that page happens to be in memory, then everything continues.
If a given page is NOT in memory, then a page fault occurs. The kernel then examines the memory map to identify which page needs to be copied to memory; looks to see what page in physical memory is available, and if none, identifies pages that are not modified flags any memory maps that refer to it marking it "not available", saves the page on disk - then reads the desired data (specified by the memory map) into the free page, modifies the original process memory map to indicate in memory, and which page it is, and then resumes/restarts the instruction.
Memory maps allow processes to be MUCH smaller than would otherwise be needed.
You really need to learn how memory is used. You can start with the references I've already provided.
Last edited by jpollard; 10-08-2016 at 02:22 PM.
|
|
|
10-09-2016, 09:21 AM
|
#49
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,274
|
Quote:
the program that fills up the virtual memory space in Windows with one image file also has the power to open and view each image file.
|
Just one comment: it is not really correct. The file (image) you opened will be put into memory. If you want to see it (to be displayed) it must be in the main memory, because the viewer program you use need it. If you do not want to see it (actually you opened another image) the file may be put into virtual memory, but it will be copied back to the main memory if you want to use that.
The handling of memory and files are different on windows and linux, but in general the runnning/living program and data must be available in the main memory, the virtual memory cannot be accessed directly.
|
|
|
10-09-2016, 10:09 AM
|
#50
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
Just a comment on the comment :-)
Quote:
Originally Posted by pan64
Just one comment: it is not really correct. The file (image) you opened will be put into memory. If you want to see it (to be displayed) it must be in the main memory, because the viewer program you use need it. If you do not want to see it (actually you opened another image) the file may be put into virtual memory, but it will be copied back to the main memory if you want to use that.
|
Even the image doesn't have to be in memory for the display - as it may have its own memory to contain the data being displayed. The computer only needs the piece it is passing to the video card. This happens more often with large images. I use to process satellite images - which would be 4K by 12K covering from about latitude 72 degrees north to about latitude 15 degrees north. The computer only needed one raster line in memory at a time. It was only the video card that had the entire image. If I was rescaling the image data then I might have 3 or 5 raster lines (or parts of raster lines...).
Other issues are that image files tend to be compressed to save space - thus the "image file", even in computer memory, is not the same (video files are severely compressed). Jpeg files have various levels of compression applied.
Only raw image files have no applied compression, usually each pixel is represented by three one byte unsigned integers (one for red, green and blue). Other image files may use YUV (the Y is the brigtness, the UV determine the color) or HIS representation (Hue, Intensity and Saturation). Each format has its own advantage/disadvantages.
The RGB raw form is easiest and fastest for conversion to a display. I think the YUV is easier for color printing. HIS is good for various other processing (a greyscale image can be examined strictly from the intensity). It all depends on the image file format.
Quote:
The handling of memory and files are different on windows and linux, but in general the runnning/living program and data must be available in the main memory, the virtual memory cannot be accessed directly.
|
In main memory for the instance an instruction that is manipulating the data, or it is being passed to another peripheral. Otherwise it doesn't need to be there.
|
|
|
10-09-2016, 05:01 PM
|
#51
|
Member
Registered: Aug 2016
Posts: 31
Original Poster
Rep:
|
So are you saying that I will not be able to fill up the virtual memory with one file? I know it can be done because I already have a program to do this... I'm sorry if I'm not understanding I only know basic entry-level programming
|
|
|
10-09-2016, 05:46 PM
|
#52
|
Senior Member
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,950
|
You don't understand.
Time to give up.
|
|
|
10-09-2016, 08:12 PM
|
#53
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
Quote:
Originally Posted by Thirdeyematt
So are you saying that I will not be able to fill up the virtual memory with one file? I know it can be done because I already have a program to do this... I'm sorry if I'm not understanding I only know basic entry-level programming
|
THINK ABOUT IT.
A virtual address on most x86 machines is 64 bits.
That is 18,446,744,073,709,551,616 bytes. You will NEVER be able to "fill it".
I've given you references for how virtual addressing works. READ THEM.
|
|
|
10-10-2016, 01:46 AM
|
#54
|
LQ Addict
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 23,274
|
Quote:
I will not be able to fill up the virtual memory with one file?
|
How did you deduce it? We just trying to explain how it works. It is not about programming, but how virtual memory is used at all.
Quote:
I know it can be done because I already have a program to do this
|
So you are ready now?
|
|
|
10-11-2016, 07:11 PM
|
#55
|
Member
Registered: Aug 2016
Posts: 31
Original Poster
Rep:
|
Yes it did work... mount -o remount,size=9T /dev/shm ... for i in{1..200000} ; do cp Love.png /dev/shm/Love-$i.png ; done That worked thank you!!... free -t shows that all the swap is used up...
But that was for my main 10tb hard drive that runs linux. When I go to mount another tmpfs system on another external hard drive, I can create a tmpfs file system there that has a size of 4tb. However, when I load files into it, it uses the main hard drive's swap space.. Is there any way I can avoid using my main hard drive's swap space and start using the swap space of the external hard drive since the main hard drive's swap space is just about used up???
Last edited by Thirdeyematt; 10-11-2016 at 07:49 PM.
|
|
|
10-11-2016, 08:39 PM
|
#56
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
Quote:
Originally Posted by Thirdeyematt
Yes it did work... mount -o remount,size=9T /dev/shm ... for i in{1..200000} ; do cp Love.png /dev/shm/Love-$i.png ; done That worked thank you!!... free -t shows that all the swap is used up...
But that was for my main 10tb hard drive that runs linux. When I go to mount another tmpfs system on another external hard drive, I can create a tmpfs file system there that has a size of 4tb. However, when I load files into it, it uses the main hard drive's swap space.. Is there any way I can avoid using my main hard drive's swap space and start using the swap space of the external hard drive since the main hard drive's swap space is just about used up???
|
Nope. tmpfs uses swap. You have used up whatever memory you have. After the first is used up it will use the rest. The only way to add more memory is to add more swap.
And you still won't use up your virtual memory. The theoretical maximum is 18.1 exabytes...
You can't afford that much.
Now the actual limit is much smaller 2^48 bits of address for Intel virtual paging - 256TB. And I'm pretty sure you can't afford that much disk space either.
Last edited by jpollard; 10-11-2016 at 08:44 PM.
|
|
|
10-11-2016, 08:52 PM
|
#57
|
Member
Registered: Aug 2016
Posts: 31
Original Poster
Rep:
|
Is there a way to extend the operating system past the first hard drive and have it take up multiple hard drives, all in the same operating system?
*Edit:
-Spanned Volume....
I will try this and report back to see if it will increase my swap space I think it should..
Last edited by Thirdeyematt; 10-11-2016 at 08:55 PM.
|
|
|
10-11-2016, 08:56 PM
|
#58
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
Quote:
Originally Posted by Thirdeyematt
Is there a way to extend the operating system past the first hard drive and have it take up multiple hard drives, all in the same operating system?
|
Of course. Initialize another disk as a swap partition, then add it in the fstab as more swap, or use the swapon utility to add during normal operations.
At one time the limit was 128 swap entries, but that may have been lifted.
It really doesn't matter. The only difference between having a file in swap and a file in another filesystem is just how you address it. Both are still on disk. Granted, it is much faster to delete the stuff in a tmpfs filesystem. All you have to do is reboot.
Last edited by jpollard; 10-11-2016 at 08:58 PM.
|
|
|
10-11-2016, 09:12 PM
|
#59
|
Member
Registered: Aug 2016
Posts: 31
Original Poster
Rep:
|
Also, your talking 4tb external passport (no power supply needed, small) hd for $130, 13 port usb hub $24... power supply is 850w so the usb 2.0 should be able to fit at least 20 hopefully.. Your talking $1500 for close to 50tb that's not a fortune..
The digital image files that can be amplified is a real thing that could be purchased online. I can't really make anyone believe in them but all I can say is that they are much stronger than brainwave entrainment, which uses sounds to change the way you think.
Thank you everyone this is now solved.
|
|
|
10-11-2016, 09:24 PM
|
#60
|
Senior Member
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,908
|
Quote:
Originally Posted by Thirdeyematt
Also, your talking 4tb external passport (no power supply needed, small) hd for $130, 13 port usb hub $24... power supply is 850w so the usb 2.0 should be able to fit at least 20 hopefully.. Your talking $1500 for close to 50tb that's not a fortune..
The digital image files that can be amplified is a real thing that could be purchased online. I can't really make anyone believe in them but all I can say is that they are much stronger than brainwave entrainment, which uses sounds to change the way you think.
Thank you everyone this is now solved.
|
Unless you have some magic to convert an image file to something displayable and have a display device, there is no communications between the computer and you.
doesn't happen.
|
|
|
All times are GMT -5. The time now is 10:14 PM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|