-
Notifications
You must be signed in to change notification settings - Fork 879
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use speculative parsing for match
statement
#11443
Conversation
244e848
to
22f51f3
Compare
610fbdd
to
af07fd0
Compare
bdd372f
to
56ee76e
Compare
// Match is considered a soft keyword, so we will treat it as an identifier if | ||
// it's followed by an unexpected token. | ||
|
||
match self.classify_match_token() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this effectively an optimization? E.g., could we always use try_parse_match_statement
and fall back when it returns None
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is an optimization because if we're sure that it's a keyword, we can skip speculative parsing. The reason being that speculative parsing has some overhead of creating the checkpoint and possibly throwing away the parsed expression if it's not a keyword.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Makes sense!
64da911
to
642036a
Compare
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
## Summary This PR adds support for parsing `match` statement based on whether `match` is used as a keyword or an identifier. The logic is as follows: 1. Use two token lookahead to classify the kind of `match` token based on the table down below 2. If we can't determine whether it's a keyword or an identifier, we'll use speculative parsing 3. Assume that `match` is a keyword and parse the subject expression 4. Then, based on certain heuristic, determine if our assumption is correct or not For (4), the heuristics are: 1. If the current token is `:`, then it's a keyword 2. If the current token is a newline, then 1. If the following two tokens are `Indent` and `Case`, it's a keyword 2. Otherwise, it's an identifier 3. Otherwise, it's an identifier In the above heuristic, the (2) condition is so that the parser can correctly identify the following as a `match` statement in case of a missing `:`: ```py match foo case _: pass ``` ### Classification table Here, the token is the one following the `match` token. We use two token lookahead for `not in` because using the `in` keyword we can classify it as an identifier or a keyword. Token | Keyword | Identifier | Either | -- | -- | -- | -- | Name | ✅ | | | Int | ✅ | | | Float | ✅ | | | Complex | ✅ | | | String | ✅ | | | FStringStart | ✅ | | | FStringMiddle | - | - | - | FStringEnd | - | - | - | IpyEscapeCommand | - | - | - | Newline | | ✅ | | Indent | - | - | - | Dedent | - | - | - | EndOfFile | | ✅ | | `?` | - | - | - | `!` | | ✅ | | `(` | | | ✅ | `)` | | ✅ | | `[` | | | ✅ | `]` | | ✅ | | `{` | ✅ | | | `}` | | ✅ | | `:` | | ✅ | | `,` | | ✅ | | `;` | | ✅ | | `.` | | ✅ | | `*` | | | ✅ | `+` | | | ✅ | `-` | | | ✅ | Other binary operators | | ✅ | | `~` | ✅ | | | `not` | ✅ | | | `not in` | | ✅ | | Boolean operators | | ✅ | | Comparison operators | | ✅ | | Augmented assign operators | | ✅ | | `...` | ✅ | | | `lambda` | ✅ | | | `await` | ✅ | | | `yield` | ✅ | | | Other keywords | | ✅ | | Soft keywords | ✅ | | | Singletons | ✅ | | |
Summary
This PR adds support for parsing
match
statement based on whethermatch
is used as a keyword or an identifier.The logic is as follows:
match
token based on the table down belowmatch
is a keyword and parse the subject expressionFor (4), the heuristics are:
:
, then it's a keywordIndent
andCase
, it's a keywordIn the above heuristic, the (2) condition is so that the parser can correctly identify the following as a
match
statement in case of a missing:
:Classification table
Here, the token is the one following the
match
token. We use two token lookahead fornot in
because using thein
keyword we can classify it as an identifier or a keyword.?
!
(
)
[
]
{
}
:
,
;
.
*
+
-
~
not
not in
...
lambda
await
yield