Lex
Table of contents
Overview
Lexing converts input source code into tokenized output. Literals, such as string literals, have their value parsed and form a single token at this stage.
Bracket matching
The lexer handles matching for (), [], and {}. When a bracket lacks a match, it will insert a “recovery” token to produce a match. As a consequence, the lexer’s output should always have matched brackets, even with invalid code.
Where the missing bracket belongs is decided by a search over candidate repairs, using indentation, structural facts about the language, and formatting conventions as evidence. See mismatched bracket recovery.