[SOLVED] File I/O and System Memory

Ronayn · 12-17-2008, 06:16 AM

Hello,

I have designed an application which reads in X-sized chunks from a large file (1G) and writes each of the chunks out to its own file.

I have noticed that while the program is running, the system memory (shown by free) gets clobbered. In fact, even when the application closes, the system memory remains at a very low level.

I have noticed this "problem" before when doing file I/O. Is this just how Linux works? Please note that I am not doing any dynamic memory allocation in the code (other than to create the initial buffer, which I reuse and delete at the end of the application).

I am currently running this app on RedHat 9, kernel 2.4.20-6smp.

Hko · 12-17-2008, 07:13 AM

Yes, this sounds very normal. Linux keeps file data it has read in memory. Chances are that it needs to be read again soon, and when it is still in memory, it will be way faster.

So Linux uses almost all memory for IO-caching. When more memory is requested by programs, it will be made available to the programs, and less room will be used for caching and buffering of IO.

The program "free" gives information about how much memory is free and how much is used for caching and buffering.

Code:

bash$ free -m
             total       used       free     shared    buffers     cached
Mem:          3289       3160        128          0        111       2399
-/+ buffers/cache:        649       2639
Swap:         9538          0       9538

The -m option makes it show number in megabytes.
The number in blue is the one you'd want to watch to see it memory is being eaten by some program. It indicates free memory, counting cache and buffers as "free".

chrism01 · 12-18-2008, 08:16 PM

Just FYI, RedHat 9, kernel 2.4.20-6smp is very old (about 10 versions behind the replacement RH Fedora 10) and hasn't been updated/supported (inc security holes) for yrs. Try to upgrade if possible.

bigearsbilly · 12-19-2008, 03:48 AM

yes this is interesting.

i wrote a perl program to test buffer sizes once,
a buffer writer piped into a reader.

I noticed when I print'd without a \n the system memory just filled up.
(due to line buffering)

interestingly on linux most times it eventually froze up the system
needing a power off.
solaris and BSD kicked the process off.

it didn't happen using C stdio strangely.

buffer_write

Code:

#!/usr/bin/perl  -s

$\ = undef;

$main::size ||= 100;

$count;

$| = 1 if $pipe;

sub do_write {

    $buf = chr int(rand(25)+65);
    $message = $buf x $size;
    $message .= "\n" if $line;;
    printf "$message";
    $count += length $message;
    warn "$count bytes\n";
    
}

while ($loop || <>) {

    do_write;
}

Code:

#!/usr/bin/perl -n

print length($_), "\n";
print;

Ronayn · 12-23-2008, 06:37 AM

Thanks to all for the replies

And yes, I want to update my OS -- however, I still have projects that require support and are running under RH9.