LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 07-04-2013, 01:37 PM   #1
yaplej
Member
 
Registered: Apr 2009
Distribution: CentOS, Ubuntu, openSuSE
Posts: 148
Blog Entries: 1

Rep: Reputation: 22
Parse string tokens and pass remaining as parameter


I am working on a command line tool for an open source project. When you type a command it parses it into tokens using strtok_r(). This is working great except some commands have additional parameters that I want to pass to the command function.

The problem is that the buffer used by strtok_r() does not work because it removes the delimiter char " " and seems to replace them with "\0" to terminate the token. So when I pass that buffer I can only parse the first token of the original command and don't even see the parameters.

So I need to pass the remaining un-used tokens as a string to the next function. My thought right now is to make another copy of the original string and maybe remove a sub-string from the beginning of it that is the command being executed. I can store the full command string excluding parameters in the command structure easy enough to be used for this.

Is there another way I should consider for this though?
 
Old 07-04-2013, 04:58 PM   #2
JohnGraham
Member
 
Registered: Oct 2009
Posts: 467

Rep: Reputation: 138Reputation: 138
How are you using strtok_r()? You should be able to get at all parts of the string - but you need to do it in the right way. For example, suppose you first call it like:

Code:
char *input_string;
const char *delim = " \t"; // Spaces and tabs as delimiters.
char *saveptr;

// Get the first token.
char *token = strtok_r(input_string, delim, &saveptr);
At this stage, you have your first token. You also shouldn't be touching input_string (and it sounds like you are). When you want to get at subsequent tokens, you do:

Code:
// NULL => use previous string, saveptr unchanged from before.
char *next_token = strtok_r(NULL, delim, &saveptr);
This means you can just pass your delim and saveptr arguments to your command.

For example:

Code:
#include <string.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
  if (argc != 3) {
    printf("Usage: %s STRING DELIMITERS (e.g. %s 'hi there' ' ')\n",
	   argv[0], argv[0]);
    return 1;
  }

  char *saveptr;

  // Get first token.
  char *token = strtok_r(argv[1], argv[2], &saveptr);

  // Print all tokens.
  while (token) {
    printf("Token: \"%s\"\n", token);
    token = strtok_r(NULL, argv[2], &saveptr);
  }
}
 
Old 07-04-2013, 05:45 PM   #3
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
May I suggest using a standard solution to commandline argument processing: getopt ? Doing this will let you use code which is well defined and debugged, behaves for the end-user like virtually all other commandline applications, is flexible enough to accommodate pretty well any requirement you can throw at it, and best of all, it's already done.

--- rod.
 
Old 07-05-2013, 10:50 AM   #4
yaplej
Member
 
Registered: Apr 2009
Distribution: CentOS, Ubuntu, openSuSE
Posts: 148
Blog Entries: 1

Original Poster
Rep: Reputation: 22
I make a copy of my original string then parsing the copy with strtok_r(). I haven't had any problems accessing the tokens. At some point I have to stop and pass the remaining un-read tokens of the string to another function.

Never thought of passing the copied string and saved pointer so that just might work. It however wont work if I have multiple functions that need to execute using the same parameter. Not sure if I should keep that functionality in or not. I might keep it only for commands that don't accept parameters.

As for using getopt() I like to think my cli is a bit more sophisticated that simply parsing a few static arguments. Its a modular cli with commands registered at run-time. The registered command are parsed. They are stored in a tree with a dynamic help menu and other neat-o goodness. The cli functions more like a shell environment for a RTOS think a router command line. When the user types a command it parses the first token tries to locate a match in the root tree nodes then follows them down until it finds one of them with a valid function pointer. It then executes that function pointer passing it any remaining tokens.

Here is the full code.
https://sourceforge.net/p/opennop/da...commands.c#l37
 
Old 07-05-2013, 11:03 AM   #5
JohnGraham
Member
 
Registered: Oct 2009
Posts: 467

Rep: Reputation: 138Reputation: 138
Quote:
Originally Posted by yaplej View Post
Never thought of passing the copied string and saved pointer so that just might work. It however wont work if I have multiple functions that need to execute using the same parameter. Not sure if I should keep that functionality in or not. I might keep it only for commands that don't accept parameters.
Consider building a dynamically allocated array of the sub-strings out of the tokens, and then pass that to the functions.
 
Old 07-06-2013, 08:47 AM   #6
yaplej
Member
 
Registered: Apr 2009
Distribution: CentOS, Ubuntu, openSuSE
Posts: 148
Blog Entries: 1

Original Poster
Rep: Reputation: 22
Quote:
Originally Posted by JohnGraham View Post
Consider building a dynamically allocated array of the sub-strings out of the tokens, and then pass that to the functions.
That should work. The command functions will just have to read the elements from the array and process them a little different than how the command line handler does. That should not be a big deal.
 
Old 07-09-2013, 05:53 PM   #7
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Quote:
Originally Posted by yaplej View Post
As for using getopt() I like to think my cli is a bit more sophisticated that simply parsing a few static arguments. Its a modular cli with commands registered at run-time. The registered command are parsed. They are stored in a tree with a dynamic help menu and other neat-o goodness. The cli functions more like a shell environment for a RTOS think a router command line. When the user types a command it parses the first token tries to locate a match in the root tree nodes then follows them down until it finds one of them with a valid function pointer. It then executes that function pointer passing it any remaining tokens.
So your parser is not a commandline parser in the sense of taking argc/argv values, but is an interactive aspect of your text-mode application? In some sense, it is perhaps a little interpretive language, then? Perhaps a full-on language grammar with a lexical parser such as one might contrive with lex & yacc (flex & bison, in GNUland) would be appropriate. Certainly a lot easier to specify the parser & grammar without ambiguity. A bit of a learning curve at first, but you'll find it much easier to use once some feature-creep has taken hold in your program.

--- rod.

Last edited by theNbomr; 07-09-2013 at 05:55 PM.
 
Old 07-12-2013, 12:55 PM   #8
yaplej
Member
 
Registered: Apr 2009
Distribution: CentOS, Ubuntu, openSuSE
Posts: 148
Blog Entries: 1

Original Poster
Rep: Reputation: 22
Quote:
Originally Posted by theNbomr View Post
So your parser is not a commandline parser in the sense of taking argc/argv values, but is an interactive aspect of your text-mode application? In some sense, it is perhaps a little interpretive language, then? Perhaps a full-on language grammar with a lexical parser such as one might contrive with lex & yacc (flex & bison, in GNUland) would be appropriate. Certainly a lot easier to specify the parser & grammar without ambiguity. A bit of a learning curve at first, but you'll find it much easier to use once some feature-creep has taken hold in your program.

--- rod.
Yes its an interactive cli for a daemon. I am not familiar with lex, yacc, flex or even bison (I know of them but less of what they do). I have to admit this a lot but I don't know a clue what I am doing I just stumble my way through the code and try stuff until it works. A few people have come along and bash me upside the head hard enough to save me from my own ignorance and point me in another direction.

So far I am having some luck with the dynamic array of char pointers. For each "parameter" token I re-allocate the array and point the next array index at that token. It seems to work but has a few issues right now. After 3-4 tokens its crashing. I will save that for another thread that I am off to make now.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
What happened when passing a static string to a std::string& parameter? fantasy1215 Programming 7 11-30-2012 07:34 AM
[SOLVED] How can i pass an entire Array into a function parameter by value? (no pointer pass) esgol Programming 26 07-31-2012 11:34 AM
C++ std::string tokens to char*[] nonis Programming 9 11-14-2008 08:00 AM
C function to split a string in tokens rogx Programming 16 08-06-2008 11:33 AM
bash - getting tokens from string klavuzkarga Linux - Newbie 6 12-14-2007 01:13 PM


All times are GMT -5. The time now is 11:41 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration