Token Data Structure

Converting lexical items into tokens simplifies later processing by reducing the input to a few standard kinds of tokens: reserved words, identifiers, numbers, operators, strings, delimiters. The tokens serve as the terminal symbols of the parser grammar.

A token will contain:

1. token type (identifier, operator, etc.)

2. data type: a numeric code indicating integer, real, etc.

3. pointers to the symbol table

4. pointers for making trees from tokens

5. value of the token (identifier name, number value, numeric code indicating which operator).

Contents    Page-10    Prev    Next    Page+10    Index