CS 375: Lexical Analyzer using Lex

Due: February 6, 2024.

Redo the lexical analyzer for Pascal, with the same specifications as in the previous assignment, but using the Lex system to generate the syntactic analysis program.

Lex is described in the textbook in Section 3.5 . There is a man description of lex on-line; also see the file commentsinlex.txt for useful hints on handling comments in Lex. Several books on Lex and Yacc are available, although you should not need them for this assignment. http://www.cs.utexas.edu/users/novak/lexpaper.htm is a paper on lex. http://flex.sourceforge.net/ has the manual on Flex (free software version of Lex).

Lex allows the tokens of a language to be described using regular expressions. The Lex compiler compiles these to produce finite automaton tables for an automaton to parse the language. When a complete token has been found, user-specified actions (written in C) are executed to produce the desired output. Lex always produces two outputs:

1. What is it?: the kind of token that was found. This is the return() value, a small integer, e.g. return(NUMBER).
2. The value of the token that was found. This is returned as a side-effect by setting the variable yylval to the value; for our purposes, yylval will always be of type TOKEN.

Lex allows a lexical analyzer to be constructed more easily than by writing one as a program. For this assignment, it is allowable to use C library routines such as sscanf to help with number conversion.

Several files are provided to help you get started. The file lexasu.l is an implementation of a simple Lex scanner similar to that shown in Figure 3.18 in the book. You can try this program with the following commands and data to Unix:

make lexasu	Use lex to compile lexasu.l --> lexasu.c --> lexasu
lexasu	Execute compiled lexasu
if switch then 3.14 else 4	Test data
^D	Control-D for end of file to stop.

The file lex2.l creates tokens similar to the ones in the first assignment; you can try it as follows:

make lex2	Use lex to compile lex2.l --> lex2.c --> lex2
lex2	Execute compiled lex2
if x > y then 3.14 else 4.5	Test data
^D	Control-D for end of file to stop.

You can copy lex2.l to lexan.l (cp lex2.l lexan.l) and use lexan.l as the starting point for your program; lexan.l can be run using make lexer as described in comments in the file.

Testing:

Test your program on the files graph1.pas and scantst.pas . Error checking for numbers out of range is not required for this assignment. How well does the lex program work on the difficult cases of scantst.pas? Is it as robust as your hand-written lexical analyzer?