-
Notifications
You must be signed in to change notification settings - Fork 879
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make Lexer
lazy
#11244
Make Lexer
lazy
#11244
Conversation
fe0fa75
to
8c77d9d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this a lot! Nice work.
I think we can improve the lexer performance by change how we handle errors in next_token
(see inline comment).
We also need to think about how parse_expression_at
should work. I think we need to pass the offset through to the lexer so that it can offset all token ranges and start lexing after offset
in the source.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It almost makes me sad to say goodby to @charliermarsh's genius implementation for supporting soft keywords with lalrpop.
3ca2ba3
to
2f35044
Compare
2f35044
to
6dad12f
Compare
efc7138
to
47ec6fe
Compare
if start_offset > TextSize::new(0) { | ||
lexer.cursor.skip_bytes(start_offset.to_usize()); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is how the lexer will start lexing from the given offset.
47ec6fe
to
dcab2ef
Compare
I'm going to merge this into another PR to keep going. I'm finding it difficult to refactor with changes separated in different branches. |
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
Summary
This PR updates the
Lexer
to make it lazy in the sense that the tokens are emitted only when it's requested.Lexer
Iterator
implementationSoftkeywordTransformer
LexicalError
in the lexer and return it onfinish
callcurrent
token and provide methods to query itTokenValue
struct which stores the owned value for certain tokens 1lex_*
methods to return the token instead ofResult
Lexer::new
to take astart_offset
parameterToken Source
Iterator
implementationLexer
Parser
bump_value
to bump the given token kind and return the corresponding owned valuebump_any
to bump any token except for end of fileFootnotes
At the end, we'll remove the
Tok
completely ↩