ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
i'm trying to read a large text file (about 600MB) using fgetc(), i'm processing the file character by character and not storing anything in memory or use any buffers.
but it seams that the OS cache the file while i'm reading it and the memory keep growing until it is full and every thing becomes very slow
so is there a way to set a maximum size for buffers/cache used for caching files
the code is something like this:
#include <stdio.h>
int main()
{
char ch;
int i=0;
FILE *myf = fopen("1train","r");
while(!feof(myf))
{
ch = fgetc(myf);
}
}
I don't think buffers/cache is your problem. The kernel will automatically free memory used for that as memory is needed for applications. Maybe fopen() allocates memory for the entire file. You might want to try the lower-level file routines (open(), et al.)
Code:
char ch;
int fd = open("1train", O_RDONLY);
while(read(fd, &ch, 1) > 0)
{
// perform operations on ch
}
close(fd);
I don't know if that will help, but it will speed your program up anyway
Have you confirmed that the memory image is actually growing?
If it's not then the slowdown may be caused by the huge number of I/O ops being performed. I've had this experience but I usually only notice it when I'm doing large amounts of console output as well.(for a machine simulator and a dinero clone)
The standard I/O routines are probably not the cause. I routinely work with 2GB+ files, no problems. Input file buffers get flushed and refreshed.
They normally do not expand forever, unless someone changed something in the kernel.
What are you doing with the data inside your code, after you read in the character?
your swap usage is under a meg, so I'm not sure why this should be slowing you down. It is normal for the kernel to cache as much of the disk as possible -- I would guess that the large amount cached simply means that no process is demanding use of ram.
i don't know what is happening!! i have 512MB ram & the program is running very good with files less than my memory but when i run it with 600MB test file the processing and all the system become very slow!! what is really annoying me is that the program runs very well on MS Windows installed on the same machine !?
i think this slow down is happening because when the kernel caches the file and physical memory is full, then the swap is used!
so i need either to limit the memory size the kernel uses for caching the file or some c code that i can use in my program to tell the kernel to free the allocated cache for the file
Very strange.
I tried your program, literally copied and pasted and have no problems.
Using: Debian sarge/testing, (vanilla) kernel 2.6.6, glibc6 v2.3.2ds1, gcc 3.3.3.
I think you should try read much more data at once and then process each character as you want. This will surely speed up your program (and if not then there's something wrong with your system settings).
Code:
FILE *myf=fopen("1train","r");
enum { BUFLEN = 200000 };
char buf[BUFLEN], ch;
int i, nread;
while ((nread = fread(buf, 1, BUFLEN, myf)))
for (i=0; i<nread; i++) {
ch = buf[i];
// your stuff here
}
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.