Lexical Analysis

If speed is needed, the Line Handler and Lexer can be coded in assembly language.

The Lexer does the following:

• Reads input characters.

• Groups characters into meaningful units or ``words'', producing data structures called tokens.

• Converts input to internal form, e.g. converts numbers to machine binary form.

• Serves as a front end for and provides input to the Parser.