Repository navigation
Share lexical regions with syntax highlighting - #253
Merged
Merged
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
jserv
reviewed
Oct 7, 2026
Separate comment and string rules let highlighting disagree with indentation, particularly while code was unfinished. Use the shared tokenizer for these regions and repaint the active editor when its grammar becomes ready, retaining existing keyword tables and colors. Cover scanner and parser paths, delayed and failed grammar loads, language switches, and editor text, caret and undo behavior.
moon-jam
force-pushed
the
fix/editor-tokenizer-highlighting
branch
from
October 7, 2026 06:06
05cebdb to
cf5cde8
Compare
Contributor
|
Thank @moon-jam for contributing! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This follows up on #91 by using its shared tokenizer for syntax highlighting. Previously, highlighting maintained separate comment and string rules. For example, an unfinished
/* ...affected indentation but remained uncolored until*/appeared.Highlighting now uses the tokenizer’s comment, string, and regular expression ranges while retaining the existing keyword tables and colors. It reuses the editor’s cached ranges introduced in #238, so bracket matching and highlighting do not tokenize the same buffer twice. When a grammar finishes loading, highlighting refreshes without requiring another edit.
Regression tests cover unfinished comments, multiline strings, template interpolation, regular expressions, bracket markers, escaping, and source preservation. Browser tests also cover delayed and failed grammar downloads, language switching, caret position, and undo/redo. The full offline browser check also passes locally.
Retaining syntax trees for incremental parsing or sharing the same tree between highlighting and indentation could improve performance, but would add complexity. This could be considered in a follow-up if needed.
Refs #91
Summary by cubic
Follows up on #91 by sharing the tokenizer's comment, string, and regexp ranges with syntax highlighting, so highlighting and indentation stay consistent even while a region is unfinished. An unclosed
/* ...now takes color immediately instead of waiting for the closing*/, and when a grammar finishes loading, the active editor repaints without requiring another edit.Highlighting keeps the existing keyword, literal, and number tables and reuses the cached token ranges, so the buffer is tokenized once for both indentation and highlighting.
Refactors
Written for commit cf5cde8. Summary will update on new commits.