The lexical analyzer is the first phase of compiler. Its main task is to read the input characters and produce as output a sequence of tokens that the parser uses for syntax analysis
- It is implemented by making lexical analyzer be a subroutine
- Upon receiving a “get next token” command from parser, the lexical analyzer reads the input character until it can identify the next token
- It may also perform secondary task at user interface
- One such task is stripping out from the source program comments and white space in the form of blanks, tabs, and newline characters
- Some lexical analyzer are divided into cascade of two phases, the first called scanning and second is “lexical analysis”.
- The scanner is responsible for doing simple task while lexical analysis does the more complex task
Issues in Lexical Analysis:
There are several reasons for separating the analysis phase of compiling into lexical analysis and parsing:- Simpler design is perhaps the most important consideration. The separation of lexical analysis often allows us to simplify one or other of these phases.
- Compiler efficiency is improved
- Compiler portability is enhanced