If speed is needed, the Line Handler and Lexical Analyzer can be coded
in assembly language.
The Lexical Analyzer does the following:
- Reads input characters.
- Groups characters into meaningful units or ``words'', producing
data structures called tokens.
- Converts units to internal form, e.g. converts numbers to
machine binary form.
- Serves as a front end for and provides input to the Parser.