LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   How to use virtual memory to load large data? (https://www.linuxquestions.org/questions/programming-9/how-to-use-virtual-memory-to-load-large-data-492532/)

wudt 10-15-2006 03:30 AM

How to use virtual memory to load large data?
 
Hi:
I have a problem about large data accessed!
I have a large data ,about 600MB,need to visual process,but my memory only 512NB,so I want to virtual memory load this data ,then in program ,I can access it like memory ,I think I have to use swap space.
I am a new guy about it,so I need some help ,who can give me some example(C++) or useful document or links to guide me to implement it !
thanks advanced!

jlliagre 10-15-2006 04:41 AM

Whatever the language you are using, it is almost certainly already using virtual memory. Only the kernel is dealing with physical memory.

Tischbein 10-16-2006 11:28 AM

Yes. That said, in Knuth's idealised processor the machine language includes hints such as "I'm going to be asking for this memory soon!" and "I'm asking for this memory to be placed in said register but I won't be using it for a long time yet". I wouldn't be surprised in the least if there were optimising commands of that form available at the C programming level and above. Here you might well want to ask for the huge data file to be put straight into swap.

But that is just a performance optimising issue. As long as the swap is there and is active all should work fine, just as Jillagre pointed out. (If you want to make a big swap partition or file, investigate fdisk, mkswap, swapon and swapoff).

paulsm4 10-16-2006 12:21 PM

If your data happened to be a disk file, you might also consider memory mapping ("mmap()" on Linux, or "CreateFileMapping()/MapViewOfFile()" on Windows). Here's a link with more info:

http://en.wikipedia.org/wiki/Mmap

PS:
As jlliagre and the other posters said, you don't have to do "anything special" to enable swapping. As far as programming: swapping is "free".

But as far as *performance*, swapping is very, very expensive. Manipulating 600++MB on a system with 512MB RAM will cause swapping. Swapping won't only make your program run slower - it will drag down the entire system.

You should definitely consider running this program on a system with more RAM.

IMHO .. PSM

bigearsbilly 10-18-2006 09:16 AM

Language?

if you are using a modern language (like
python or perl say), then it will deal with all that for you.

I routinely slurp huge files and let someone else worry about the VM.

jlliagre 10-18-2006 12:00 PM

Quote:

Originally Posted by bigearsbilly
Language?

if you are using a modern language (like
python or perl say), then it will deal with all that for you.

Old languages, even the older or assembly still use virtual memory under Unix, unless you are programming the kernel or a module.


All times are GMT -5. The time now is 02:18 PM.