c++ basics: strtok confusion

jay73 · 11-07-2008, 02:26 PM

Could anyone explain what is wrong with the code below? The thing is that it fails to print any of the statements that follow the while loop but it does not generate any warnings or messages and the program appears to exit fine. The problem does not manifest itself when I replace the part in bold with the alternative code.
Now, obviously, since I am tokenizing a string that does not contain any instances of the specified delimiter, there will be only one token. So if I run the original code, the while loop as it stands will try to print an empty pointer. I can see how this is no good but I am surprised that it does not result in a segmentation fault or any other indication of a logic error. From the output of the program, one would not be able to tell that something is wrong.

Quote:

const char * ar1 = "hello";
const int arraysize = 5;
char ar2[arraysize+1];
strcpy(ar2, ar1);
cout <<"Size of array 2: " << sizeof ar2 << endl;
cout << ar2 << endl;

cout << "Comparison of string1 and string2: " << strcmp(ar1, ar2) << endl;
char * appended = strncat(ar2, ar1, 5);
appended[arraysize+5] = '\0';
cout << appended <<endl;
char * cPtr=appended;
int count=0;
for (; *cPtr!='\0'; cPtr++)
count++;
cout << "Size of appended: " << count <<endl;

cout << "Tokenized: " << endl;
char *tok = 0;
tok = strtok(appended, " ");
cout << tok << endl;

while (tok!=NULL){

tok = strtok(NULL, " ");
cout << "before printing" << endl; // this line printsbut all that follows disappears
cout << tok << endl; //does not point to anything
cout << "after printing" << endl;
}

int x = NULL;
cout << x << endl;

alternative code:

Quote:

tok = strtok(appended, " ");

while (tok!=NULL){
cout << tok << endl;
tok = strtok(NULL, " ");

}

dmail · 11-07-2008, 02:39 PM

Your code should not run to there, or at least it is not defined what it is doing at that point.

You should get a seg fault at the following location as you are indexing out of bounds

Code:

char * appended = strncat(ar2, ar1, 5);
appended[arraysize+5] = '\0';

You seem to be confused on what strncat does, see your reference manual, man, online docs etc such as http://www.cplusplus.com/reference/c...g/strncat.html
May I ask why you are not using std::string?

jay73 · 11-07-2008, 05:04 PM

Quote:

You should get a seg fault at the following location as you are indexing out of bounds

Are you sure? As I understand it, strncat appends n units from the second argument to the string specified by the first argument, overwriting its terminating null, and then it returns the first argument. I assign that result to a new pointer to a char so there aren't any bounds - obviously, a pointer isn't quite the same as an array; so the only thing that delimits the string is its terminating null - which is missing since I appended only the first five characters of ar1, dropping its terminating null. That is why I add a '\0' in the next statement. No problem as far I can see. It is really the tokenizing that does not run as expected. Here is the whole thing, by the way:

Code:

# include <iostream>
using std::cout;
using std::endl;

# include <cstring>
using std::string;
using std::strcpy;

int main(){
      
    const char * ar1 = "hello";
    const int arraysize = 5;
    char ar2[arraysize+1];
    strcpy(ar2, ar1);
    cout <<"Size of array 2: " << sizeof ar2 << endl;
    cout << ar2 << endl;
    
    cout << "Comparison of string1 and string2: " << strcmp(ar1, ar2) << endl;
    char * appended = strncat(ar2, ar1, 5);
    appended[arraysize+5] = '\0';
    cout << appended <<endl;
    char * cPtr=appended;
    int count=0;
    for (; *cPtr!='\0'; cPtr++)
        count++;
    cout << "Size of appended: " << count <<endl;
       
    cout << "Tokenizing: " << endl;
    char *tok = 0;    
    tok = strtok(appended, " ");    
    cout << tok << endl;          
    
    while (tok!=NULL){
        
        tok = strtok(NULL, " ");
        cout << "before printing" << endl;
        cout << tok << endl;          
        cout << "after printing" << endl;
    }
    
    int x = NULL;
    cout << x << endl;
       
    return 0;

}

dmail · 11-07-2008, 05:58 PM

Quote:

Are you sure? As I understand it, strncat appends n units from the second argument to the string specified by the first argument, overwriting its terminating null, and then it returns the first argument. I assign that result to a new pointer to a char so there aren't any bounds - obviously, a pointer isn't quite the same as an array; so the only thing that delimits the string is its terminating null - which is missing since I appended only the first five characters of ar1, dropping its terminating null. That is why I add a '\0' in the next statement.

Yes I am quite sure lets take a look at the source

Code:

const char * ar1 = "hello";
const int arraysize = 5;
char ar2[arraysize+1];
strcpy(ar2, ar1);
cout <<"Size of array 2: " << sizeof ar2 << endl;
cout << ar2 << endl;

cout << "Comparison of string1 and string2: " << strcmp(ar1, ar2) << endl;
/*the next line is why I feel you are confused with strncat the pointer returned is ar2 and it does not allocate more memory,
quote from page I linked to "Pointer to the destination array, which should contain a C string, and be large enough to contain
the concatenated resulting string, including the additional 
null-character." ok you copy the pointer arg2 to appended*/
char * appended = strncat(ar2, ar1, 5);
/*now here appended is an alias for ar2 which has a size of 6 (arraysize+1)
yet you try and index "10" this is out of bounds of the stack memory and it should be blowing some whistles here, 
"toot toot, abandon ship" lol*/
appended[arraysize+5] = '\0';

Now lets look at the STL verison of doing this

Code:

std:string ar1 ("hello");
std::string ar2 = ar1;
ar2 += ar1;
std::string::size_type s = ar2.find_first_of(" ");
...

jay73 · 11-07-2008, 07:18 PM

Hmm, I'm learning from a book that has most of the string operations in its very last chapter so it will be a few more weeks before I get round to that.

Thanks for your suggestions. I'll try them out when I get home.