LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-15-2006, 03:30 AM   #1
wudt
LQ Newbie
 
Registered: Mar 2005
Posts: 1

Rep: Reputation: 0
How to use virtual memory to load large data?


Hi:
I have a problem about large data accessed!
I have a large data ,about 600MB,need to visual process,but my memory only 512NB,so I want to virtual memory load this data ,then in program ,I can access it like memory ,I think I have to use swap space.
I am a new guy about it,so I need some help ,who can give me some example(C++) or useful document or links to guide me to implement it !
thanks advanced!
 
Old 10-15-2006, 04:41 AM   #2
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.4, Oracle Linux, Mint, Tribblix, Ubuntu/WSL
Posts: 9,763

Rep: Reputation: 461Reputation: 461Reputation: 461Reputation: 461Reputation: 461
Whatever the language you are using, it is almost certainly already using virtual memory. Only the kernel is dealing with physical memory.
 
Old 10-16-2006, 11:28 AM   #3
Tischbein
Member
 
Registered: Oct 2006
Distribution: debian
Posts: 124

Rep: Reputation: 15
Yes. That said, in Knuth's idealised processor the machine language includes hints such as "I'm going to be asking for this memory soon!" and "I'm asking for this memory to be placed in said register but I won't be using it for a long time yet". I wouldn't be surprised in the least if there were optimising commands of that form available at the C programming level and above. Here you might well want to ask for the huge data file to be put straight into swap.

But that is just a performance optimising issue. As long as the swap is there and is active all should work fine, just as Jillagre pointed out. (If you want to make a big swap partition or file, investigate fdisk, mkswap, swapon and swapoff).

Last edited by Tischbein; 10-16-2006 at 11:30 AM.
 
Old 10-16-2006, 12:21 PM   #4
paulsm4
LQ Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
If your data happened to be a disk file, you might also consider memory mapping ("mmap()" on Linux, or "CreateFileMapping()/MapViewOfFile()" on Windows). Here's a link with more info:

http://en.wikipedia.org/wiki/Mmap

PS:
As jlliagre and the other posters said, you don't have to do "anything special" to enable swapping. As far as programming: swapping is "free".

But as far as *performance*, swapping is very, very expensive. Manipulating 600++MB on a system with 512MB RAM will cause swapping. Swapping won't only make your program run slower - it will drag down the entire system.

You should definitely consider running this program on a system with more RAM.

IMHO .. PSM

Last edited by paulsm4; 10-16-2006 at 12:25 PM.
 
Old 10-18-2006, 09:16 AM   #5
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Debian, Mint, Puppy, Raspbian
Posts: 3,450

Rep: Reputation: 210Reputation: 210Reputation: 210
Language?

if you are using a modern language (like
python or perl say), then it will deal with all that for you.

I routinely slurp huge files and let someone else worry about the VM.
 
Old 10-18-2006, 12:00 PM   #6
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.4, Oracle Linux, Mint, Tribblix, Ubuntu/WSL
Posts: 9,763

Rep: Reputation: 461Reputation: 461Reputation: 461Reputation: 461Reputation: 461
Quote:
Originally Posted by bigearsbilly
Language?

if you are using a modern language (like
python or perl say), then it will deal with all that for you.
Old languages, even the older or assembly still use virtual memory under Unix, unless you are programming the kernel or a module.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
need help on processing large data files eph Programming 3 03-11-2004 04:56 AM
Large Memory Problem Ravenmoonheart Linux - Newbie 7 07-09-2003 02:06 PM
RH 8.0 Mapping Virtual Memory to get access to VMIC Reflective Memory PCI card. Merlin53 Linux - Hardware 0 05-05-2003 12:50 PM
trouble installing suse linux [There is not enough memory to load all data] rishineedsajob Linux - Newbie 1 03-23-2003 08:18 PM
Large data files on CD dema Linux - Newbie 1 01-26-2002 10:30 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:24 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration