Lexical analysis, lexing or tokenization is the process of converting a sequence of characters into a sequence of tokens. Lexical Analyzer reads the source program character by character and returns t tokens of the source program. Lexical Analyzer also called as Scanner, Tokenizer or lever. It reads the stream of characters making up the source program and groups the characters into meaningful sequences called Lexemes.
For each lexeme, it produces as output a token of the form <token_name, attribute_value=””></token_name,> token_name is an abstract symbol that is used during syntax analysis, and the second component attribute_value points to an entry in the symbol table for this token. A token describes a pattern of characters having same meaning in the source program. (such as identifiers, operators, keywords, numbers, delimeters and so on) For Example:
x = a + b * 2;
[(identifier, x), (operator, =), (identifier, a), (operator, +), (identifier, b), (operator, *), (literal, 2), (separator, ;)]Puts information about identifiers into the symbol table not all attributes. Regular expressions are used to describe tokens (lexical constructs). A (Deterministic) Finite State Automation can be used in the implementation of a lexical analyzer.