LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   Parse string tokens and pass remaining as parameter (https://www.linuxquestions.org/questions/programming-9/parse-string-tokens-and-pass-remaining-as-parameter-4175468498/)

yaplej 07-04-2013 01:37 PM
Parse string tokens and pass remaining as parameter
 
I am working on a command line tool for an open source project. When you type a command it parses it into tokens using strtok_r(). This is working great except some commands have additional parameters that I want to pass to the command function.

The problem is that the buffer used by strtok_r() does not work because it removes the delimiter char " " and seems to replace them with "\0" to terminate the token. So when I pass that buffer I can only parse the first token of the original command and don't even see the parameters.

So I need to pass the remaining un-used tokens as a string to the next function. My thought right now is to make another copy of the original string and maybe remove a sub-string from the beginning of it that is the command being executed. I can store the full command string excluding parameters in the command structure easy enough to be used for this.

Is there another way I should consider for this though?

JohnGraham 07-04-2013 04:58 PM
How are you using strtok_r()? You should be able to get at all parts of the string - but you need to do it in the right way. For example, suppose you first call it like:

Code:
char *input_string;
const char *delim = " \t"; // Spaces and tabs as delimiters.
char *saveptr;

// Get the first token.
char *token = strtok_r(input_string, delim, &saveptr);
At this stage, you have your first token. You also shouldn't be touching input_string (and it sounds like you are). When you want to get at subsequent tokens, you do:

Code:
// NULL => use previous string, saveptr unchanged from before.
char *next_token = strtok_r(NULL, delim, &saveptr);
This means you can just pass your delim and saveptr arguments to your command.

For example:

Code:
#include <string.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
  if (argc != 3) {
    printf("Usage: %s STRING DELIMITERS (e.g. %s 'hi there' ' ')\n",
          argv[0], argv[0]);
    return 1;
  }

  char *saveptr;

  // Get first token.
  char *token = strtok_r(argv[1], argv[2], &saveptr);

  // Print all tokens.
  while (token) {
    printf("Token: \"%s\"\n", token);
    token = strtok_r(NULL, argv[2], &saveptr);
  }
}

theNbomr 07-04-2013 05:45 PM
May I suggest using a standard solution to commandline argument processing: getopt ? Doing this will let you use code which is well defined and debugged, behaves for the end-user like virtually all other commandline applications, is flexible enough to accommodate pretty well any requirement you can throw at it, and best of all, it's already done.

--- rod.

yaplej 07-05-2013 10:50 AM
I make a copy of my original string then parsing the copy with strtok_r(). I haven't had any problems accessing the tokens. At some point I have to stop and pass the remaining un-read tokens of the string to another function.

Never thought of passing the copied string and saved pointer so that just might work. It however wont work if I have multiple functions that need to execute using the same parameter. Not sure if I should keep that functionality in or not. I might keep it only for commands that don't accept parameters.

As for using getopt() I like to think my cli is a bit more sophisticated that simply parsing a few static arguments. Its a modular cli with commands registered at run-time. The registered command are parsed. They are stored in a tree with a dynamic help menu and other neat-o goodness. The cli functions more like a shell environment for a RTOS think a router command line. When the user types a command it parses the first token tries to locate a match in the root tree nodes then follows them down until it finds one of them with a valid function pointer. It then executes that function pointer passing it any remaining tokens.

Here is the full code.
https://sourceforge.net/p/opennop/da...commands.c#l37

JohnGraham 07-05-2013 11:03 AM
Quote:
Originally Posted by yaplej (Post 4984756)
Never thought of passing the copied string and saved pointer so that just might work. It however wont work if I have multiple functions that need to execute using the same parameter. Not sure if I should keep that functionality in or not. I might keep it only for commands that don't accept parameters.
Consider building a dynamically allocated array of the sub-strings out of the tokens, and then pass that to the functions.

yaplej 07-06-2013 08:47 AM
Quote:
Originally Posted by JohnGraham (Post 4984764)
Consider building a dynamically allocated array of the sub-strings out of the tokens, and then pass that to the functions.
That should work. The command functions will just have to read the elements from the array and process them a little different than how the command line handler does. That should not be a big deal.

theNbomr 07-09-2013 05:53 PM
Quote:
Originally Posted by yaplej (Post 4984756)
As for using getopt() I like to think my cli is a bit more sophisticated that simply parsing a few static arguments. Its a modular cli with commands registered at run-time. The registered command are parsed. They are stored in a tree with a dynamic help menu and other neat-o goodness. The cli functions more like a shell environment for a RTOS think a router command line. When the user types a command it parses the first token tries to locate a match in the root tree nodes then follows them down until it finds one of them with a valid function pointer. It then executes that function pointer passing it any remaining tokens.
So your parser is not a commandline parser in the sense of taking argc/argv values, but is an interactive aspect of your text-mode application? In some sense, it is perhaps a little interpretive language, then? Perhaps a full-on language grammar with a lexical parser such as one might contrive with lex & yacc (flex & bison, in GNUland) would be appropriate. Certainly a lot easier to specify the parser & grammar without ambiguity. A bit of a learning curve at first, but you'll find it much easier to use once some feature-creep has taken hold in your program.

--- rod.

yaplej 07-12-2013 12:55 PM
Quote:
Originally Posted by theNbomr (Post 4987326)
So your parser is not a commandline parser in the sense of taking argc/argv values, but is an interactive aspect of your text-mode application? In some sense, it is perhaps a little interpretive language, then? Perhaps a full-on language grammar with a lexical parser such as one might contrive with lex & yacc (flex & bison, in GNUland) would be appropriate. Certainly a lot easier to specify the parser & grammar without ambiguity. A bit of a learning curve at first, but you'll find it much easier to use once some feature-creep has taken hold in your program.

--- rod.
Yes its an interactive cli for a daemon. I am not familiar with lex, yacc, flex or even bison (I know of them but less of what they do). I have to admit this a lot but I don't know a clue what I am doing :) I just stumble my way through the code and try stuff until it works. A few people have come along and bash me upside the head hard enough to save me from my own ignorance and point me in another direction.

So far I am having some luck with the dynamic array of char pointers. For each "parameter" token I re-allocate the array and point the next array index at that token. It seems to work but has a few issues right now. After 3-4 tokens its crashing. I will save that for another thread that I am off to make now.


All times are GMT -5. The time now is 02:36 PM.