why are strsep and strtok() to be avoided?

slzckboy · 01-22-2006, 08:25 AM

even their own man pages say not to use them?

Anyone know of a similar replacement for these functions,

some library of some sort ??

Pls don't make me write my own function lol

PTrenholme · 01-22-2006, 10:37 AM

Well, you did read the man pages, didn't you? As it states, the problem is that those two functions work by assuming (as is, of course, correct for C) that the first input argument is a pointer to the address of a character array, and they work by modifying the contents of the character array and the address of the start of the array.

Specifically, consider this code:

Code:

#include <stdio.h>
#include <string.h>
int main()
{
  char *x, *y;
  int i = 0;
  x =(char *) malloc((size_t)128);
  strcpy(x,"This is a test;Testing 1, 2, 3;Done");
  printf ("x=%x\ni=%d: %s\n", &x, i, x);
  while (y = strsep(&x, ";")) {
    printf("y=\"%s\"\n", y);
    printf("x=%x\ni=%d: %s\n", &x, ++i, x);
  }
  return 0;
}

which produces this:

Code:

x=bfa6a244
i=0: This is a test;Testing 1, 2, 3;Done
y="This is a test"
x=bfa6a244
i=1: Testing 1, 2, 3;Done
y="Testing 1, 2, 3"
x=bfa6a244
i=2: Done
y="Done"
x=bfa6a244
i=3: (null)

Notice that &x does not change, whilst x[] does change.

The "problem" is that your input string will be modified as a "side effect" of using the function. In general, this kind of action produces code that is very hard to either test or debug. Among other thing, unless you've used a fixed size malloc call (as I did in the example) you may end up with memory leak problems.

In essence, those two functions "convert" the input string from a char[] array to a char[][] array, and return the next value of the "iterator" into the array of strings.

If you have a choice, consider using a C++ string object with well defined behavior.

paulsm4 · 01-22-2006, 02:57 PM

As PTrenholme pointed out, the problem with these two functions is that they're "non-reentrant": they assume that state will be preserved between invocations - an assumption that's decidedly NOT true in a multithreaded application.

Nevertheless, "strtok()" can be an extremely handy function. I would be *cautious* about using it - but I wouldn't necessarily *avoid* using. Especially if it happens to be to be the best solution for your particular code.

When any of my methods or functions use "strtok()", I always make sure:
a) consecutive calls to the string in question are always single-threaded
b) the code is clearly commented "non-reentrant: uses strtok()!"

IMHO .. PSM

slzckboy · 01-25-2006, 10:01 AM

That makes sense.Thnks gents.

I did read the MAN page,but I think I just needed it explained in another way.

I think for my purpose I will just malloc a copy of the buffer to be parsed and then free the resource after I have extracted the information I need as intimated above.

Would someone mind explaining why there is a potential for memory leaks though ?

PTrenholme · 01-25-2006, 03:41 PM

Quote:

Originally Posted by slzckboy

[snip]
Would someone mind explaining why there is a potential for memory leaks though ?

That was an offhand comment prompted from my experience with older C (and, in fact, B) programmers, who often neglected the call to free(). In fact, in the example code, I neglected to do so, myself.

slzckboy · 01-25-2006, 04:15 PM

oh.. no worries.

well thnks again for your explanation.

MrMichaelWill · 04-27-2009, 11:33 AM

Quote:

Originally Posted by slzckboy

even their own man pages say not to use them?

Anyone know of a similar replacement for these functions,

some library of some sort ??

Pls don't make me write my own function lol

Unfortunately the manpage was updated by an amateur. The original author of the functions would certainly disagree :-)

The important information in summary (which you already partially got) is:
1. the strtok will modify the original data, which means that you have to make a copy of it before you work on it unless you don't care
2. the strtok function is not thread-safe, which means it relies on some global (within your program) status data, which works fine for your application unless you use multiple threads in the same process (pthreads). Then you want to use strtok_r instead, which should work fine.

The person leaving the discouraging and confusing comment in the manpage failed to point out an alternative.

I am using strtok_r successfully.

Michael