Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #710 +/- ##
==========================================
+ Coverage 97.13% 97.18% +0.05%
==========================================
Files 31 31
Lines 3663 3804 +141
Branches 328 346 +18
==========================================
+ Hits 3558 3697 +139
- Misses 63 65 +2
Partials 42 42 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
living180
force-pushed
the
dynamic_tokenlist_value_v2
branch
4 times, most recently
from
March 30, 2023 10:11
de63e50 to
ff4f391
Compare
|
This is just the fix I need. Ran into some problems with parsing SQL with ~50k ID |
|
any progress here? |
9 tasks
living180
force-pushed
the
dynamic_tokenlist_value_v2
branch
from
January 18, 2025 22:55
ff4f391 to
2da3e63
Compare
Contributor
Author
|
Updated the PR to fix merge conflicts. |
Avoid stripping T.Comment tokens contained within an sql.Comment before stripping the sql.Comment itself. Now an sql.Comment token will be stripped first along with any contained T.Comment tokens.
living180
force-pushed
the
dynamic_tokenlist_value_v2
branch
from
September 15, 2026 11:48
2da3e63 to
116b654
Compare
Rename Token to TokenBase and make it a superclass for TokenList and a new Token class. Move some of the functionality of TokenBase into Token and TokenList. This will make it easier to maintain separate functionality for Token versus TokenList.
living180
force-pushed
the
dynamic_tokenlist_value_v2
branch
from
September 15, 2026 11:53
116b654 to
5239ee0
Compare
The fact that a new value was being computed each time TokenList.group_tokens() was called caused supralinear runtime when token grouping was enabled. Address by making TokenList.value a dynamically-computed property rather than a static attribute.
living180
force-pushed
the
dynamic_tokenlist_value_v2
branch
from
September 15, 2026 12:00
5239ee0 to
14abdd6
Compare
Contributor
Author
|
Updated to fix merge conflicts and make tests pass again. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supersedes #623. The meat of the PR is the same: fix the remaining portion of issue #621 by making
TokenList.valuea dynamically-computed property rather than an attribute. This avoids the quadratic runtime behavior that occurred due to recomputingTokenList.valueeach timeTokenList.group_tokens() was called withextend=True.The previous PR #623 had some rather awkward hacks related to stripping comments, but I found that I could avoid those by simply tweaking the comment stripping process to strip comments from a token list before stripping any sublists, making this PR much simpler.