For example, my lexer recognizes a function call pattern:

Now that flex recognizes the pattern, but it goes passed the last character in the pattern (i.e. after stored foo(...) inside yytext, the lexer will point to the next character after foo(...))

How can I reset the lexer pointer back to the beginning of the function pattern? i.e. after recognizing foo(..), I want to the lexer to point to the start of foo(..), so I can start tokenizing it.

I need to do this because for each regex pattern, only one token can be returned for each pattern. i.e. after matching foo(...), I can only return either foo or ( or ) with return statement but not all.

I'm pretty sure you can't go backwards. You shouldn't match a function call expression with a lexer, regular languages can't match arbitrarily nested things, so you would have with things like: func((a*(b+c)), x).

Time to look at a parser, since you've started with flex, perhaps bison.

1 members found this post helpful.

Hi,

Thanks for the answer. However, I found the answer a few days ago and intended to post it back here, but I forgot. To make the lexer go backward, so it won't ignore any character, we use yyless(). If my lexer found the string like init(, using yyless(yyleng-1) (yyleng is the length of yytext) will put back one character starting from the right most side. In this case, init( will become init, and ( will be put back into the stream to be tokenized later. Similarly, using yyless(yyleng-2), only ini will remain in yytext, and the character t and ( will be put back into the stream.

In my case, my expression will only try to match for a function-name token and return it. It has to have a way to detect what a function name is. For example, in function init(), init is the function-name. My expression will match any sequence of characters which contain ( in it as a hint of a function, and attempt to take any characters prior to ( as a function name, and return to the lexer.

I will use bison later. But currently, I have to create the lexer first using flex. This is one of the assignment I'm doing for the Compilers course from Coursera: Compilers. Coursera also offers lots of useful courses as well.

EDIT: I fixed my original regular expression (in original post) to detect only init( in init(...) (... is parameters if exist). The patern will recognize function name like foo or _foo_ or _foo_foo_ etc...

1 members found this post helpful.

A quick look at the reference manual for COOL (the language for that course) says
Function call is not a lexical element.

You are right. After struggling with recognizing function call using start condition which complicates later lexing process (currently I am having many start conditions, which makes it really hard to manage), I decided to use another way to recognize the function name in a function. After all, function name is just an object identifier which does not contain not alnum characters (except for \- and \_ ).