LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-12-2004, 04:31 AM   #1
alshishtawy
LQ Newbie
 
Registered: May 2004
Posts: 4

Rep: Reputation: 0
Question memory full when reading large files with fgetc()


i'm trying to read a large text file (about 600MB) using fgetc(), i'm processing the file character by character and not storing anything in memory or use any buffers.

but it seams that the OS cache the file while i'm reading it and the memory keep growing until it is full and every thing becomes very slow

so is there a way to set a maximum size for buffers/cache used for caching files

the code is something like this:

#include <stdio.h>

int main()
{
char ch;
int i=0;
FILE *myf = fopen("1train","r");
while(!feof(myf))
{
ch = fgetc(myf);
}
}
 
Old 05-12-2004, 08:50 AM   #2
itsme86
Senior Member
 
Registered: Jan 2004
Location: Oregon, USA
Distribution: Slackware
Posts: 1,246

Rep: Reputation: 59
I don't think buffers/cache is your problem. The kernel will automatically free memory used for that as memory is needed for applications. Maybe fopen() allocates memory for the entire file. You might want to try the lower-level file routines (open(), et al.)

Code:
char ch;
int fd = open("1train", O_RDONLY);

while(read(fd, &ch, 1) > 0)
{
  // perform operations on ch
}

close(fd);
I don't know if that will help, but it will speed your program up anyway

Last edited by itsme86; 05-12-2004 at 08:51 AM.
 
Old 05-12-2004, 09:14 AM   #3
kooch
Member
 
Registered: Mar 2004
Location: Upstate NY
Distribution: Slackware/YDL
Posts: 77

Rep: Reputation: 15
Have you confirmed that the memory image is actually growing?

If it's not then the slowdown may be caused by the huge number of I/O ops being performed. I've had this experience but I usually only notice it when I'm doing large amounts of console output as well.(for a machine simulator and a dinero clone)
 
Old 05-12-2004, 09:17 AM   #4
jim mcnamara
Member
 
Registered: May 2002
Posts: 964

Rep: Reputation: 36
Something does not sound right here.

The standard I/O routines are probably not the cause. I routinely work with 2GB+ files, no problems. Input file buffers get flushed and refreshed.
They normally do not expand forever, unless someone changed something in the kernel.

What are you doing with the data inside your code, after you read in the character?

Are you sure your process is tyhe problem?
 
Old 05-12-2004, 11:07 AM   #5
alshishtawy
LQ Newbie
 
Registered: May 2004
Posts: 4

Original Poster
Rep: Reputation: 0
no open() and read did not solve the problem

this is what i get while running the program

$ free
total used free shared buffers cached
Mem: 497964 493592 4372 0 876 424824
-/+ buffers/cache: 67892 430072
Swap: 1012084 496 1011588

as u can see the memory is filled with cached data and all the system became very slow :S

sure there is no problem with the standard I/O routines. i'm wondering if it is possible to set a max for buffers/cache ?? i'm using Fedora Core 1

thanx
 
Old 05-12-2004, 02:32 PM   #6
aluser
Member
 
Registered: Mar 2004
Location: Massachusetts
Distribution: Debian
Posts: 557

Rep: Reputation: 43
your swap usage is under a meg, so I'm not sure why this should be slowing you down. It is normal for the kernel to cache as much of the disk as possible -- I would guess that the large amount cached simply means that no process is demanding use of ram.

Here are my stats at the moment:
Code:
15:28 aluser@alf:~$ free
             total       used       free     shared    buffers     cached
Mem:       1033648     995192      38456          0     184296     269888
-/+ buffers/cache:     541008     492640
Swap:      2008084       1008    2007076
As you can see, my system fills up most of its ram, too.
 
Old 05-13-2004, 02:33 AM   #7
alshishtawy
LQ Newbie
 
Registered: May 2004
Posts: 4

Original Poster
Rep: Reputation: 0
i don't know what is happening!! i have 512MB ram & the program is running very good with files less than my memory but when i run it with 600MB test file the processing and all the system become very slow!! what is really annoying me is that the program runs very well on MS Windows installed on the same machine !?

i think this slow down is happening because when the kernel caches the file and physical memory is full, then the swap is used!

so i need either to limit the memory size the kernel uses for caching the file or some c code that i can use in my program to tell the kernel to free the allocated cache for the file

thanx to every one
 
Old 05-13-2004, 06:03 AM   #8
Hko
Senior Member
 
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: Debian
Posts: 2,536

Rep: Reputation: 111Reputation: 111
Very strange.
I tried your program, literally copied and pasted and have no problems.
Using: Debian sarge/testing, (vanilla) kernel 2.6.6, glibc6 v2.3.2ds1, gcc 3.3.3.
 
Old 05-13-2004, 06:30 AM   #9
Ma3oiS
LQ Newbie
 
Registered: May 2004
Posts: 12

Rep: Reputation: 0
Hi,

I think you should try read much more data at once and then process each character as you want. This will surely speed up your program (and if not then there's something wrong with your system settings).

Code:
   FILE *myf=fopen("1train","r");
   enum { BUFLEN = 200000 };
   char buf[BUFLEN], ch;
   int i, nread;

   while ((nread = fread(buf, 1, BUFLEN, myf)))
      for (i=0; i<nread; i++) {
         ch = buf[i];
         // your stuff here
      }
 
Old 05-30-2004, 07:45 AM   #10
alshishtawy
LQ Newbie
 
Registered: May 2004
Posts: 4

Original Poster
Rep: Reputation: 0
Hi everyone,

i faced the above problem when running my code on Fedora Core 1 !

when testing the same code on the same machine but using Mandrake 9.1 or Windows XP the program ran without problems!!!!

is this a bug in memory management in FC1 or miss configuration??
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
I/O error when copying large file from Memory Stick airman99 Linux - General 3 08-22-2005 09:26 AM
memory full sailu_mvn Linux - Software 3 08-22-2005 08:06 AM
Partition shows 99% full even after deleting large files other1 Linux - Newbie 4 04-19-2004 11:18 AM
Memory is Full orenr Linux - Software 3 11-14-2003 10:36 AM
Large Memory Problem Ravenmoonheart Linux - Newbie 7 07-09-2003 02:06 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 08:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration