Token: Sequence of character having a collective meaning is known as token.
Typical tokens are,
1) Identifiers
2) keywords
3) operators
4) special symbols
5) constants
Pattern: The set of rules called pattern associated with a token.
Lexeme: The sequence of character in a source program matched with a pattern for a token is called lexeme.
| Token | Lexeme | Pattern |
| Const | Const | Const |
| If | If | If |
| Relation | <,<=,= ,< >,>=,> | < or <= or = or < > or >= or > |
| Id | Pi, count, n, I | letter followed by letters and digits. |
| Number | 3.14159, 0, 6.02e23 | Any numeric constant |
| Literal | “RNS Institute” | Any character between “ and “ except “ |
Example:
total = sum + 12.5 Tokens are: total (id), = (relation) Sum (id) + (operator) 12.5 (num)
Lexemes are: total, =, sum, +, 12.5