TC-1/2-Lexer Code to Write

Be sure to read RE/flex documentation and tutorials, see RE/flex & Bison.

Makefile.am

Include your own testsuite in the tests directory. Some test examples are given in the tests folder (Given Test Cases). Do not hesitate to make use of them.

Include your own test suite in the tests directory, and hook it to make check. See Generating The Test Driver.

---

lib/misc/symbol.* & lib/misc/unique.*

The class misc::symbol keeps a single copy of identifiers, see The lib/misc Directory. Its implementation in lib/misc/symbol.hxx is incomplete. Note that running make check at the root of the project will produce in lib/misc unit testing binaries. Having this unit test lib/misc/test-symbol.cc pass should be a goal by itself. As a matter of fact, unit tests were left to help you: once they pass successfully you may proceed to the rest of the compiler. misc::symbol’s implementation is based on misc::unique, a generic class implementing the Flyweight design pattern. The definition of this class, lib/misc/unique.hxx, is also to be completed.

lib/misc/variant.*

The implementation of the class template misc::variant<T, Ts...> lacks a couple of conversion operators that you have to supply.

---

src/parse/scantiger.ll

Specifications:

The scanner must be able to scan inputs as described in Lexical Specifications and Additional Lexical Specifications.

What is RE/flex:

RE/flex is a C++ library used to generate lexical analyzers that can interact with Bison. Its purpose is to generate a C++ lexer class with a lex method, which can be called to initiate the lexical analysis of the input data and interact with Bison to produce a complete parser.

The file has a special syntax which is explained here.

More about RE/flex & Bison.

What does RE/flex generate?

RE/flex generates two files parsetiger.hh and parsetiger.cc. These files contain the parse::Lexer class which has a lex method that allows the lexer to be launched. Bison will use it to feed the parser with tokens.

How do I write a rule with RE/flex?

The goal of a lexer is to produce a token from an input stream. For that, RE/flex uses a Maximal Munch algorithm and thus proposes to define patterns as regex that will, when matched, create a token.

Here is an example of how to create a token from a regex to create an array token:

%%

"array"       return parser::make_ARRAY(td.location);
"array"       return TOKEN(ARRAY); // Alternatively.

%%

To create the tokens in the parser you have to use the Bison functions defined for each token. They follow this pattern: parser::make_TOKEN(...).

The macro TOKEN() allows to simplify the creation of tokens because Bison’s syntax for creating tokens is verbose.

You can find out how to write a rule on the RE/flex documentation.

To have more readability on the regexes you can define them before using them as done on this example.

RE/flex and sublexers:

To be able to lex the strings you will have to use sublexers, here is the syntax of the sublexers:

...
"\""        {grown_string.clear(); start(SC_STRING);}

/* Handling of the strings. Initial " is eaten. */

<SC_STRING>{
"\"" {
  start(INITIAL);
  return TOKEN_VAL(STRING, grown_string);
}

...

\\x[0-9a-fA-F]{2}  {
  grown_string += strtol(text() + 2, 0, 16);
}

...
}

On RE/flex documentation, sublexers are called states.

Strings will be stored as C++ std::string. See the following code for the basics.

Symbols (i.e. identifiers) must be returned as misc::symbol objects, not strings.

Metavariable:

Be careful, metavariable (Additional Lexical Specifications) keywords should only be scanned when extensions are enabled, i.e. not on direct user input.

Trace the lexing

To keep track of lexing you can enable it using the set_debug(true) method on the lexer or the SCAN environment variable.

Location tracking

The locations are tracked. The class Location to use is produced by Bison: src/parse/location.hh.

To track locations, adjust your scanner, use YY_USER_ACTION and the lex prologue:

...
%%
%{
  // Everything here is run each time `lex` is *invoked*.
%}

/* The rules. */

"if"    return TOKEN(IF);
...
%%