feat(extraction): add Odin language support - #1523
Open
ivandrenjanin wants to merge 2 commits into
Open
Conversation
… built from the tag's checked-in parser.c tree-sitter-wasms does not ship an Odin grammar, so the wasm is built here from tree-sitter-grammars/tree-sitter-odin @ v1.3.0 (e8adc73, MIT, Copyright (c) 2023 Amaan Qureshi) with tree-sitter-cli 0.26.11 `build --wasm`, ABI 14. The tag's checked-in parser.c is used as-is rather than regenerated: it predates the ABI-15 generator, and regenerating would diverge the tables from the tag nobody could then reproduce. Pin the TAG and not master. Master's single unreleased commit past v1.3.0 (d2ca8ef, "allow multiple identifiers before `:` in named_type") regresses multi-value return types — `f :: proc() -> (string, bool)` parses to an ERROR node — which is one of the most common signature shapes in real Odin. Measured across three pinned public Odin repositories (343 files): 68 files with parse errors at v1.3.0 against 111 at master. docs/grammars/tree-sitter-odin.md records the sha256, the parser.c and scanner.c hashes, the builder version, a reproducible rebuild recipe, and the measured parse health — including the fact that a CRLF working tree costs 1.7% of declarations to a v1.3.0 line-continuation gap. Refs colbymchenry#648, colbymchenry#1000
…ll scoping Registers `odin` across the five wiring points — LANGUAGES in src/types.ts, the wasm-file map / EXTENSION_MAP / VENDORED_WASM_LANGS / display name in grammars.ts, and the extractor map — and adds a 581-line extractor covering procedures with their calling conventions and `@(...)` attributes, procedure groups linked to their overloads, structs with fields, enums, unions, bit fields, `distinct` and proc-type aliases, `::` constants, imports with and without an alias, `foreign` blocks and `when` blocks. Two walker branches in tree-sitter.ts, both guarded on `this.language === 'odin'` and both there to stop a WRONG edge rather than to add one: - extractVariable: Odin declares positionally — `[attributes] NAME :: value`, with no `name:` field in the grammar and a comma-separated name list on both the `::` and the `:` form. The generic fallback takes every identifier child, so `Alias :: Other` minted a second symbol named after the right-hand side. - extractCall: Odin puts a call's package qualifier OUTSIDE the call node — `fmt.println(x)` is member_expression(`fmt`, call_expression(function: `println`)) — so the generic path read `println` alone and linked it to whatever same-named procedure the repo happened to define, which is the wrong-edge class colbymchenry#1079/colbymchenry#1107 fixed for same-named methods. The qualifier is re-attached as `pkg::callee`, matching the qualifiedName the package namespace already gives every top-level symbol. A `->` call is deliberately left bare: its receiver is a variable, not a package. Builtins (`len`, `append`, `max`) are suppressed at emit the way terraform's BUILTIN_HEADS and SCALA_BUILTIN_TYPES are. An Odin package IS a directory, so resolution/index.ts gains a third arm beside the two nix ones on the same `if`/`else if`: an unqualified `calls` ref can only bind inside its own package directory, because a genuine cross-package call carries its qualifier and resolves by qualified name. This is the same directory-scoping shape Terraform already has and that the Nix language PR (7f32513) established for a language's own scoping rules. A committed test asserts the gate costs neither of the two edges Odin genuinely has. Imports are recorded as symbols but deliberately not resolved to file edges: `import "core:fmt"` and `import "../shared"` name a directory, not a file. 28 extraction tests and 2 resolution tests, modeled on the Rust block. README and CHANGELOG gain their rows, both carrying the grammar's measured parse caveat; docs/grammars/tree-sitter-odin.md (previous commit) carries the provenance and the numbers. Odin routes to the wasm engine — the native Rust kernel has no Odin counterpart, and adding one is a clean follow-up rather than a prerequisite. Refs colbymchenry#648, colbymchenry#1000
|
@ivandrenjanin - I'm happy if you could take this over and work on it, as t the moment i have very limited time for this. Thank you Rainer PS: My code was created and run on macosx. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Odin support: a vendored
tree-sitter-odingrammar, an extractor, the registration points, and tests.It works
I indexed three public Odin repos with this branch. A
mainbuild finds nothing in any of them, since.odinis not a known extension yet.maine62a37195bf11565f57caAn Odin package is a directory, so a call into another package has to carry its qualifier. Every cross-directory call edge across the three resolves by qualified name, and none point at a builtin like
lenorappend.About #1000
@RainerXE proposed Odin back in June in #1000 and I built on his work. He picked the grammar and worked out the node types an extractor needs. I checked those against the parser and they are right.
I did not cherry-pick his code. The extractor is new, and the overlap is boilerplate the
LanguageExtractorinterface forces. HisextractStructfix is a real bug that is still onmain, and I left it for him.He got there first and I would rather he got the credit. Take anything useful here, it is MIT. I opened separately because #1000 has conflict markers committed and its registration edits got reverted by upstream merges, so no
.odinfile would actually be opened. That is not carelessness on his part: his head branch is his fork'smain, so every sync merges into the PR. Up to you which one you take.Not done
The agent A/B in
CLAUDE.mdstep 3. It needs a logged-inclaudeCLI and tmux, and the harness targets macOS/Linux. Steps 1 and 2 ran fine. So I have no retrieval-quality number against a without-codegraph arm. I will run it if you can point me at an environment.CRLF checkouts lose symbols. The grammar chokes on a trailing-backslash line continuation when the file has CRs, which is what
odinfmtemits on a wrapped line. Roughly twice as many files fail to parse on a Windows checkout as on a POSIX one, and a few lose their declarations outright. Two smaller grammar gaps sit in the doc. I worked around none of them, and the README and CHANGELOG rows say so.Windows only. I did not test macOS or Linux. The two things that break on a case-sensitive filesystem are fine: the wasm filename and the
./odinimport are both lowercase.Some tests already fail on Windows before this PR, all
EPERMfromfs.rmSyncin cleanup. Same names onmain. My branch adds passing tests and nothing else moves.Grammar
v1.3.099fd3217b82b1ba4f438120591ec2ed24108f782ab693d69b330d753188632fbsrc/parser.c, no regenerate, no patchI pinned the tag rather than master because master's one unreleased commit breaks multi-value returns, turning
f :: proc() -> (string, bool)into an error tree. Both binaries load cleanly, so you only catch it by diffing a corpus. Provenance and a rebuild recipe are indocs/grammars/tree-sitter-odin.md.Wasm only. The kernel has no Odin, so
codegraph-kernelis untouched.Testing
tsc --noEmitandnpm run buildare clean, and the wasm reachesdist/with its hash intact. The Odin tests pass. The fixture covers what those three repos do not reach on their own: calling conventions,whenblocks, unions, bit fields, and procedure groups.Twelve files in two commits. The wasm sits alone in the first one so the source diff stays readable. Nothing outside Odin, and no version bump.
Odin did surface a bug in
codegraph affected, but the fix changes Go, Rust and Python too, so it is not this PR's to make. I can send it separately.Refs #648, #1000.