How to read buffer in C?

obladioblada · 06-27-2007, 11:24 AM

Hi there,

I have a question about how to read buffer after I read a file into buffer.

The file format is like this:

0 100.00
description of the case
comment line
1 2 3.00 "id1" 4.0
2 2 3.00 "id2" 5.0
.
.
.
30 2 4.45 "id30" 6.0

The second and the third line are some description and comments. I have a bunch of this kind of files and the description and comments are different. The format of the fourth line to the last line is fixed "%d%d%f%c%f". How to read this buffer?

The reason I do this is that I want to read file into a buffer, then MPI_Bcast the buffer to all processors, then read buffer in each processor. I am a rookie, any suggestion?

Thanks a lot

nc3b · 06-27-2007, 01:24 PM

Three times fgets and then fscanf until feof.. doesn't this work? I'm not sure if I understood your question very well..

obladioblada · 06-27-2007, 03:57 PM

I want to read buffer.

Read file into buffer first, then read buffer.

Thanks.

PatrickNew · 06-27-2007, 11:27 PM

I think what you're looking for is the mmap() system call. That is, you want to take the whole file and read it into a chunk of memory? Try "man 2 mmap" to get more information.

obladioblada · 06-28-2007, 01:44 PM

Quote:

Originally Posted by PatrickNew

I think what you're looking for is the mmap() system call. That is, you want to take the whole file and read it into a chunk of memory? Try "man 2 mmap" to get more information.

Thanks. but my question is how to read buffer, instead of put something into buffer.

Maybe I can read the ASCII input file and obtain the arrays I need. Then write these arrays to a buffer, broadcast the buffer and read it. Do you have any idea about how to write different arrays (like struct) into a buffer, and then read it?

knobby67 · 06-28-2007, 02:13 PM

Your not really making yourself clear.

If you can read the file and load into a buffer how can you not then read the buffer in the same way you wrote to it?

What sort of buffer are you using? An array of structures a linked list or any of half a dozen sorts?

If you want help show us what the file says, and how you load it into a buffer.

obladioblada · 06-28-2007, 05:01 PM

Quote:

Originally Posted by knobby67

Your not really making yourself clear.

If you can read the file and load into a buffer how can you not then read the buffer in the same way you wrote to it?

What sort of buffer are you using? An array of structures a linked list or any of half a dozen sorts?

If you want help show us what the file says, and how you load it into a buffer.

Let me make myself clear.

The beginning of the file input is:

0 10.0
Aug. 20 2006
some descriptions
1, 3, 0.0, 1.000, 2.000, 2.200, 22.01000,0.00,'id 1',13.0000, 1
2, 3, 0.1, 1.000, 2.000, 1.200, 22.02000,0.00,'id 2',13.0000, 1
3, 3, 0.2, 1.000, 2.000, 3.200, 22.04000,0.00,'id 3',13.0000, 1
4, 3, 0.3, 1.000, 2.000, 4.500, 22.05000,0.00,'id 4',13.0000, 1
5, 3, 0.4, 1.000, 2.000, 2.600, 22.06000,0.00,'id 5',13.0000, 1

My code:
first case. I want to load all file into a buffer, and then read the buffer. But I do not know how to read the buffer.

FILE *file;
char *buffer;
unsigned long fileLen ;

file = open(name,"r");
fseek(file,0,SEEK_END);
fileLen=ftell(file);
fseek(file,0,SEEK_SET);

buffer=(char *) malloc (fileLen + 1);
if (!buffer)
{ printf("Memore error!"!\n);
return;
}

fread(buffer,fileLen,1,file);
fclose(file);

// Now I want to read the buffer like read a file, how to do it?
// something like sscanf(buffer...)??

Second case, I want to read all the arrays I need from the file. And then write these arrays into a buffer, broadcast it, and read the arrays from the buffer in other processors.

if ((fp = fopen(fname,"r")) == NULL)
{ printf("can not open input file\n"); exit(0);
}
fscanf(fp, "%d%f", &i, &ii);
fgets(tmpbuffer,80,fp); //char tmpbuffer[80];
fgets(tmpbuffer,80,fp);
fgets(tmpbuffer,80,fp);
for (i=0;i<n5;i++)
{
fscanf(fp, "%d,%d,%lf,%lf,%lf,%lf,%lf,%lf,%6c,%lf,%s,", &b1,&b2,&b3,&b4,&b5,&b6,&b7,&b8,b9,&b10,&b11);
a1[i]=b1; //int
b1[i]=b2; //int
c1[i]=b3; //double
d1[i]=b4; //double
e1[i]=b5; //double
sprintf(f1[i],"%s",b9); //char
}

Now I want to save these arrays into a buffer, and read the buffer? How to do it?

Thanks,

paulsm4 · 06-28-2007, 05:02 PM

Hi, obladioblada -

By "buffer", it sounds like you mean an MPI ("Message Passing Interface") buffer. So the real questions are:

1) Can I read structured data into an MPI buffer, so that I can send it with a single MPI_Bcast command?

2) If so, how can I parse the data from my text file into this buffer?

I've frankly never used MPI, so I don't know the answer to the first question. The second question, parsing the data, should be trivial. Just pick a language you feel comfortable with (or feel comfortable learning), and go for it.

Here's a tutorial on MPI (assuming you might not already know all this stuff):
http://www-unix.mcs.anl.gov/mpi/tuto...ropp/talk.html

Languages that might be worthwhile to parse your data file with include:
a) Perl
b) Python
c) C/C++
d) Fortran
etc etc

'Hope that helps .. PSM

PS:
A buffer in the 'C' sense (like in your example code above) is just a "bag of bits" - it has no inherent structure. An MPI buffer, on the other hand, is typically an "array of integers" or
an "array of reals".

Perhaps the easiest solution is just to rephrase the problem like this:

a) Parse my text file into local arrays
b) Broadcast the arrays to other processors (via MPI buffers)
... and ...
c) Read broadcasts from other processors into my local arrays
d) Write the arrays to a new text file

exvor · 06-28-2007, 08:02 PM

You can read it just like you would any other string that you have. You could use sscanf or you can read each char one by one.

On a side note this gives me some new ideas on how to manage my own program that just uses text for data storage rather then a complicated linked list. Tho im not sure which program is more memory conservative.

Code:

#include<stdio.h> 
#include<stdlib.h> 

int main()
{ 
   char *buffer = NULL; 

   unsigned int filelen = 0; 
   FILE *testfile = NULL; 



   testfile = fopen("datab.txt","r"); 


   fseek(testfile,0,SEEK_END); 

   filelen = ftell(testfile); 

   rewind(testfile); 


   printf("\nLength of file is %i charecters\n",filelen); 

   buffer = malloc(filelen +1); 
 
   fread(buffer,filelen,1,testfile); 


   fclose(testfile); 


   printf("\nThe first charecter in the buffer is %c\n",buffer[0]); 

   free(buffer); 


   return 0; 
}

Of course this program has no error checking at all.

exvor · 06-28-2007, 11:23 PM

Tho remember that this would be ok for reading but not for adding anything as the buffer is only the size of the file and would overrun.

bigearsbilly · 06-29-2007, 02:47 AM

why do you want to do this?
there's no point as the C lib reads files into a buffer anyway.
I've experimented with this myself the only advantage is
added complexity

exvor · 06-29-2007, 05:06 PM

Yeah after looking at that again it is quite useless.

paulsm4 · 06-29-2007, 05:23 PM

As I understand it, obladioblada is trying to use an MPI buffer, not a "buffer" in the ordinary, everyday sense.

He posted some sample code just about the same time as I sent him a response; I'm not sure exactly what he meant - but I'm pretty sure he wasn't looking for something like "mmap()" (which lets you treat a disk file as though it were a block of memory) or "sscanf ()" (which lets you do formatted reads from a string in memory, instead of a text string read from disk).

He definitely wants to parse the text file (we're all agreed upon that). He probably wants to read the data from disk into arrays (it looks like MPI - like FORTRAN - is oriented toward arrays of simple types). And, if he's not conversant with 'C', he might want to consider writing this tool in something simpler like Python, Perl .. or even FORTRAN.

IMHO .. PSM

knobby67 · 06-30-2007, 06:37 AM

If I understand you, (sorry if I don't or other people have answered)
Why don't you create a linked list of structures for your numbers, the list is linked to the date field Aug. 20 2006 in your example. Or you could even have all the date description part as a seperate structure or the struct first element in the list. Then read in the data values eg

1, 3, 0.0, 1.000, 2.000, 2.200, 22.01000,0.00,'id 1',13.0000, 1

so int, int ,float/double etc structure elements.

and store these values in each element of the list. Then if you need to add a new set of values traverse the list and add where needed. When you have finished you can save the structures to a file. You do this sort of thing in games progreamming.

obladioblada · 07-01-2007, 05:46 PM

paulsm4 is right.

I want to read structured data into an MPI buffer, so that I can send it with a single MPI_Bcast command? And then, at the receiver processor, I want to read the buffer.

I was trying to use following codes to write buffer:
int j;
j = 0;
for (i=0;i<n;i++)
{
j+= sprintf(fbuffer+j, "%d %f ",int_a[i], float_b[i]);
}

but I can NOT read it by using following code:

j = 0;
for (i=0;i<n;i++)
{
j+=sscanf(fbuffer+j, "%d %f ",&a,, &f1);
printf("%d %f ",a,f1);
}

And it will worse if there is char.