Skip to content

Generate Term encoders for Rust types read from source - #1

Merged
dannote merged 6 commits into
masterfrom
syn-variant-fields
Sep 30, 2026
Merged

dannote merged 6 commits into
masterfrom
syn-variant-fields

Conversation

@dannote

@dannote dannote commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Generates Rustler Term encoders for Rust types read from source. This covers types owned by another crate that don't implement serde, such as vize_atelier_vapor's IR, which vize_ex currently encodes by hand.

RustQ.Rustler.Term.encoders_from_source/3

Starting from root types, it follows field and variant payload types through a RustQ.Syn index and builds one free function per reachable struct or enum:

pub(crate) fn encode_set_prop_ir_node<'a>(env: rustler::Env<'a>, value: &SetPropIRNode<'_>) -> rustler::Term<'a>
  • Structs encode as atom-keyed maps, and newtype structs as their field.
  • Unit variants encode as atoms, tuple variants as their payload, and named variants as maps. With tag:, data-carrying variants also carry the variant atom.
  • Option encodes None as nil, sequences and sets as lists, and maps as maps.
  • Policy is explicit: wrapper roles (for example allocator-aware Vec/Box), external: helpers, and per-type except:, key and variant renames, and with: helpers.
  • The traversal is closed. A reachable type that isn't indexed, a wrapper, a scalar, or external fails generation with a list of the unmapped types. The same applies to names defined in several sources and to generic types. An upstream change therefore shows up at generation time.
  • A tag: that would replace a payload field with the same key fails generation instead of silently overwriting it.
  • transparent: true encodes a struct with exactly one field as that field.
  • encoder_atoms_from_source/3 returns the atom declarations from the same source.

The feature is documented in a new "Encoders for types from Rust source" section of the Rustler generation guide.

Supporting changes

  • Syn metadata: RustQ.Syn.Enum.variant_shapes gives one RustQ.Syn.Variant per variant, with its kind, fields, and docs. lifetimes and type_parameters are added on Syn.Struct and Syn.Enum. All are defaulted, and variants is unchanged.
  • RustQ.Syn.Index.structs/1, struct/2, and struct!/2.
  • RustQ.Rust.AST.PatternBuilder.tuple/1, for the existing PatTuple node.
  • Fix: a literal atom in a typespec (required(:kind) => :if_node) is typed as Atom. It used to be emitted as a Rust type named if_node.
  • Fix: keyword field access (value.type), keyword struct literal and pattern fields, and keyword macro arguments now render as raw identifiers (r#type) in both renderers. Previously field access failed to render. The existing r#type spelling keeps working.

Verification

  • mix ci passes: 802 tests, credo, dialyzer, ExDNA, Reach, format, rust.fmt, rust.check, rust.clippy, rustq.gen --check, corpus, and templates.
  • Focused tests cover each encoding rule, the per-type options, atom derivation, Syn.Index input, and every error path. Direct render tests cover the keyword fix and PatternBuilder.tuple/1.
  • The public consumer fixture generates encoders for a small IR with Box, Vec, Option, HashMap, a keyword field, a newtype, and all variant kinds. Its package test now also runs cargo clippy -- -D warnings, which is clean.

First real consumer

vize_ex's Vapor IR encoders were generated with this branch, replacing its handwritten vapor_ir encoders. The generated code compiles clippy-clean with -D warnings. Across 18 templates covering every IR operation, the output kept every existing key and value, with 0 removed keys and 0 changed values, and it now also includes the IR fields the handwritten code dropped. The tag collision check and transparent: came out of that migration.

RustQ.Syn.Enum gains variant_shapes, a RustQ.Syn.Variant per variant
with its kind (unit, tuple, or named), fields, and doc comments, so
generators can follow enum payload types from real Rust source.
RustQ.Syn.Struct and RustQ.Syn.Enum also report lifetimes and
type_parameters, matching function and method metadata.

The new fields are defaulted; variants still lists variant names.
Field access, struct literal and struct pattern fields, and macro item
arguments built their identifiers with format_ident!, which cannot
produce a keyword, so value.type failed to render. They now go through
ident_from_part, which emits r#type, and the Elixir renderer escapes
macro arguments the same way. ident_from_part also accepts names that
are already spelled r#type.
RustQ.Rustler.Term.encoders_from_source/3 follows field and variant
types from root types through a RustQ.Syn index and builds one encoder
function per reachable struct or enum, so crates can encode types they
do not own and that do not implement serde. encoder_atoms_from_source/3
returns the matching atom declarations.

Wrapper roles, external helpers, the variant tag, and per-type renames
and exclusions are explicit options. A reachable type that is not
indexed, a wrapper, a scalar, or external fails generation with a list
of the unmapped types.

Also add Syn.Index.structs/1, struct/2, struct!/2 and
PatternBuilder.tuple/1. The public consumer fixture generates encoders
for a small IR and its package test now runs clippy.
@dannote dannote changed the title Describe enum variant payloads and item generics in Syn metadata Generate Term encoders for Rust types read from source Sep 29, 2026
Found while generating vize_ex's Vapor IR encoders:

- A variant tag that would replace a payload field with the same key
  (CreateComponentIRNode has its own kind field) now fails generation
  instead of silently overwriting the field. The public consumer
  fixture had this bug and now renames the field.
- transparent: true encodes a struct with exactly one field as that
  field, so IREffect { operations } can stay a list.
- Unmapped and unsupported types are reported once, with every type
  that uses them.
A literal atom such as :if_node in a map field was emitted as a Rust
type named after the atom (pub kind: if_node), which does not compile.
Lowercase atoms that are not Rust primitive names are Elixir literals,
so they now become Atom, like a union of atoms. Rust type names written
as atoms (u32, MyType) are unchanged.
@dannote
dannote merged commit dc458b3 into master Sep 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant