[SOLVED] fgets stops processing

schneidz · 10-22-2010, 04:12 PM

hi, i am trying to read in a file 1 line at a time and for some reason it stops printing out at about line 62,000.

i am doing this:

Code:

 while(fgets(c0, 1085, fstream0) != NULL)

but after about 62,0000 lines it stops printing. no seg-fault, no core dump. it just stops printing to the terminal then returns me to the command line after a couple of minutes.

as a hack i am doing split -l 50000 on the input and calling my program 5 times.

is there some limitation on fgets that i am not understanding ?

rupertwh · 10-22-2010, 06:15 PM

There is no limit to fgets().
Does it end by EOF or an error?
Is the buffer really large enough? (Why the literal 1085?) You are not guaranteed a segfault just because memory is overwritten.

eSelix · 10-22-2010, 06:25 PM

Show more code. The memory buffer reservation, opening stream, printing data (for best everything in the loop).

schneidz · 10-23-2010, 09:58 AM

Quote:

Originally Posted by rupertwh

There is no limit to fgets().
Does it end by EOF or an error?
Is the buffer really large enough? (Why the literal 1085?) You are not guaranteed a segfault just because memory is overwritten.

i guess this may be part of my problem:

Code:

 awk '{print length()}' file-1.in | sort | uniq -c
99086 1077
  26 1078
69189 1079
 awk '{print length()}' file-2.in | sort | uniq -c
57515 1077
41597 1078
69189 1079

but the 1085 buffer seems large enough for each line.
heres some more code:

Code:

#include "stdio.h"
#include <string.h>


main(int argc, char *argv[])
{
 FILE *fstream0, *fstream2;
 char c0[1085], c2[1085];
 fstream0 = fopen(argv[1], "r");
 fstream2 = fopen(argv[2], "r");
i = 0;
 while(fgets(c0, 1085, fstream0) != NULL)
 {
  fgets(c2, 1085, fstream2);
i++; printf(" i = %d\n", i);
 if(strcmp(strndup(c0+0,39),strndup(c2+0,39)) != 0)
  printf("%s%s-%s:FLD-1\n", strndup(c0+39,9), strndup(c0+57,2), strndup(c0+129,4));
 if(strcmp(strndup(c0+39,9),strndup(c2+39,9)) != 0)
  printf("%s%s-%s:FLD-2\n", strndup(c0+39,9), strndup(c0+57,2), strndup(c0+129,4));
 if(strcmp(strndup(c0+57,2),strndup(c2+57,2)) != 0)
  printf("%s%s-%s:FLD-3\n", strndup(c0+39,9), strndup(c0+57,2), strndup(c0+129,4));
...
  fflush(fstream0);
  fflush(fstream2);
 }
 fclose(fstream0);
 fclose(fstream2);
}

the last few lines end like this:

Code:

 i = 59510
12345678900-0004:FLD-1
12345678900-0004:FLD-29
 i = 59511
abcdefghi00-0005:FLD-1
-:FLD-78
 i = 59512
 i = 59513
 i = 59514
 i = 59515
...
 i = 168301

rupertwh · 10-23-2010, 10:20 AM

You don't check the return value on the read from stream2
What are the calls to fflush() supposed to accomplish? -> man fflush. Possibly the cause of your immediate problem.
You don't ever check return values on the strdup()s.
You never seem to free() the memory allocated by each and every strdup()
Why all that strduping at all? Are you aware that each call contains an implicit malloc()?

schneidz · 10-23-2010, 11:59 AM

^ thx, i should call free() 2 times for each if then 3 more times if true.
not sure where to inject those free()'s since i didnt save those pointers (i figured the code would be more compact this way).

edit:

fyi: i tried this on a few fields (the ones that yielded the most hits):

Code:

 char *s1, *s2, *s3;
...
 if(strcmp(strndup(c0+0,39),strndup(c2+0,39)) != 0){
s1=strndup(c0+39,9); s2=strndup(c0+57,2); s3=strndup(c0+129,4);
  printf("%s%s-%s:FLD-1\n", s1, s2, s3); free(s1); free(s2); free(s3);

but it stops processing at the same line (i = 59511). i was expecting to run out of ram around 75,000.

schneidz · 10-23-2010, 09:36 PM

hi, heres the modification that works:

Code:

 char c0[1085], c2[1085], *s1, *s2, *s3, *s4, *s5;
...
int i = 0;
 while(fgets(c0, 1085, fstream0) != NULL)
 {i++; printf("i = %d\n", i);
  fgets(c2, 1085, fstream2);

 if(strcmp(s4=strndup(c0+0,39),s5=strndup(c2+0,39)) != 0)
  printf("%s%s-%s:FLD-1\n", s1=strndup(c0+39,9) ,s2=strndup(c0+57,2) ,s3=strndup(c0+129,4));
  free(s1); free(s2); free(s3); free(s4); free(s5);

 if(strcmp(s4=strndup(c0+39,9),s5=strndup(c2+39,9)) != 0)
  printf("%s%s-%s:FLD-2\n", s1=strndup(c0+39,9) ,s2=strndup(c0+57,2) ,s3=strndup(c0+129,4));
  free(s1); free(s2); free(s3); free(s4); free(s5);
...
 if(strcmp(s4=strndup(c0+1077,1),s5=strndup(c2+1077,1)) != 0)
  printf("%s%s-%s:FLD-89\n", s1=strndup(c0+39,9) ,s2=strndup(c0+57,2) ,s3=strndup(c0+129,4));
  free(s1); free(s2); free(s3); free(s4); free(s5);
 free(c0); free(c2);
 }
 fclose(fstream0);
 fclose(fstream2);
}

yields this result:

Code:

 tail test.out
i = 168297
aaaaaaaaa00-0002:FLD-64
i = 168298
bbbbbbbbb00-0001:FLD-64
i = 168299
ccccccccc00-0001:FLD-64
i = 168300
ddddddddd00-0001:FLD-64
ddddddddd00-0001:FLD-78
i = 168301
eeeeeeeee00-0001:FLD-64

eSelix · 10-24-2010, 11:59 AM

You still have errors. It maybe work correctly now, but someday 'blow up'. Follow rupertwh notices from #5 post (points 1. and 3.). Also you made 'free()' outside of 'if()' - you possibly freed unallocated memory. Split your 'one-line' code to more lines, for example:

Code:

if(strcmp(s4=strndup(c0+0,39),s5=strndup(c2+0,39)) != 0)
  printf("%s%s-%s:FLD-1\n", s1=strndup(c0+39,9) ,s2=strndup(c0+57,2) ,s3=strndup(c0+129,4));
  free(s1); free(s2); free(s3); free(s4); free(s5);

change to something like this:

Code:

s4=strndup(c0+0,39);
s5=strndup(c2+0,39);

if(strcmp(s4,s5)) != 0)
  {
  s1=strndup(c0+39,9);
  s2=strndup(c0+57,2);
  s3=strndup(c0+129,4);

  printf("%s%s-%s:FLD-1\n", s1, s2, s3);
  free(s1); free(s2); free(s3);
  }
free(s4); 
free(s5);

By the way, after clearing, now I think strndup is not necessary. You can check:

Code:

if(strncmp(c0+0,c2+0,39)) != 0)
  {
  printf("%.9s%.2s-%.4s:FLD-1\n", c0+39, c0+57, c0+129);
  }

Essentially it is nothing bad with writing code in 'one-line' style, but it is more readable and easier to see errors in multiple lines. Just out of curiosity - are you perl programmer?

schneidz · 10-24-2010, 12:41 PM

^ damn it, i didnt think strcmp had a strncmp counterpart. (the man pages on this aix server are broken).
i am done with this assignment for now so i will revisit this next release.
note to self: sed s/strndup/strncmp/g compare.c

no i dont do perl more of a bash/ c guy (but now i am understanding sed-awk a little bit too).

thx eselix, rupertwh: i give you 2-points.

i also didnt know you can byte-offset an array inside a printf. i'm glad i took this program, i'm making mistakes but learning alot.