token.txt 08 Aug 07 Conventions for Use of Tokens This document describes the conventions for using the token data structure as in the file token.h . The type TOKEN is a pointer to a token record, which contains the following fields: tokentype The kind of token: OPERATOR, DELIMITER, RESERVED, IDENTIFIERTOK, STRINGTOK, or NUMBERTOK. datatype INTEGER, REAL, STRINGTYPE, or BOOLETYPE This is a small-integer type code that is used when the type is a basic type, e.g. INTEGER. If the type is not a basic type, this field is unused. symtype a SYMBOL pointer for the type of the token. symtype is necessary for every token in an expression tree whose type is not a basic type (as in datatype above). Operator tokens, as well as Identifier tokens, may have a symtype pointer, e.g. the aref operator in (aref p 4) has a symtype that points to the type of the result. symentry symentry points to the symbol table entry of an Identifier; other kinds of tokens do not have a symentry pointer. symentry is needed for Identifiers because the offset of the variable (in the symbol table) is needed to generate the load/store instruction to access the variable. operands a TOKEN pointer to the first operand of an operator. In drawings of expression trees, operands is always a down arrow. link a TOKEN pointer to the next operand at the same level. In drawings of expression trees, link is always a horizontal arrow pointing to the right. In an expression that prints as (+ i j), the pointer from + to i is operands, the pointer from i to j is link, and the link field of j is NULL. + / operands / / I ------> J link The following value fields share storage; only one of them can be valid for any particular token: whichval for OPERATOR, DELIMITER, or RESERVED tokens, this small integer code tells which operator, delimiter, or reserved word it is. intval the value of a NUMBERTOK that is an INTEGER. realval the value of a NUMBERTOK that is REAL. stringval the character string that is the name of an IDENTIFIERTOK or the value of the string constant of a STRINGTOK. See the file symtab.txt for code to look up an identifier in the symbol table and mke the proper connections in its token.