Internals

Copy Markdown View Source

How KiwiCodec is put together, for people changing it. Using the Rustler generator is covered in Native decoders.

Layout

  • lib/ is the codec: the schema parser and binary schema format (KiwiCodec.Schema.*), wire primitives (KiwiCodec.Wire.*), the DSL and the modules it generates (KiwiCodec.DSL, KiwiCodec.GeneratedModule.*), encoding and decoding of generated structs (KiwiCodec.Encoder, KiwiCodec.Decoder), and the schema interpreter. It has no dependency beyond varint.
  • codegen/ is the Rustler generator. Every module there is wrapped in if Code.ensure_loaded?(RustQ), because RustQ is optional: projects that depend on KiwiCodec get the generator only where they also depend on RustQ.
  • Tests mirror the source tree. test/package/ builds the package and compiles it in two consumer projects, one without RustQ and one generating Rust.

Rustler generator

The generator serves two goals at once:

  1. Keep generator code semantic and maintainable in Elixir.
  2. Keep generated Rust compact enough for large schemas, such as Figma's.

For repetitive schema decoders, those goals are served by a compact macro boundary rather than by fully expanding every decoder body.

Layers

The generator is organized as a small pipeline:

  1. Schema definitions are parsed and selected by KiwiCodec.RustlerGenerator.
  2. One generator per feature, Full, Sparse, Skip, and Records, turns each definition into a call to a compact Rust macro, such as kiwi_message_decoder!, kiwi_sparse_struct_decoder!, or kiwi_skip_message_decoder!, with a row per field. Field says how a field is read or skipped, and Name names every generated function.
  3. Helper modules define those macros and the functions they call, with RustQ defrustmacro and defrust typed against the reference decoder.rs beside KiwiCodec.RustlerGenerator.Decoder: Values for what full and sparse decoders share, Full.Helpers, Sparse.Helpers, and Skip.Helpers for each feature, and Sparse.Messages, Skip.Messages, or Sparse.SkipMessages for messages, depending on which features are generated together. Primitive defines the readers and skippers of each primitive type from one table of Decoder methods.
  4. KiwiCodec.RustlerGenerator.Splice includes each needed helper module whole, through its __rustq_items__/0.
  5. KiwiCodec.RustlerGenerator.Entrypoint emits a NIF per entrypoint calling kiwi_decode_binary with the definition's decoder.

This split keeps schema logic in Elixir while avoiding thousands of repeated expanded Rust function bodies.

Three pieces are shared by every schema and written as Rust under templates/: the record runtime, the runtime schema that skips unknown fields, and the atom cache. They are ordinary Rust APIs, documented for the Rust code that uses them, rather than repetitive decoder shapes. The record runtime and atom cache are named after the modules that provide them, templates/records.rs for records.ex, so RustQ knows their signatures and RustQ code elsewhere names the module rather than the file, and so is the runtime schema, templates/unknown_field_runtime_schema.rs for unknown_field_runtime_schema.ex, which UnknownFields includes in one mode.

Compact RustQ macro boundary

The macros selected by Splice are a compact RustQ-authored boundary. They are shared implementations for highly repetitive decoder shapes, but they should be authored through defrustmacro, defrust, type metadata, and RustQ AST helpers rather than through raw Rust heredocs or ad hoc token strings.

When changing full, sparse, or skip decoder generation:

  • Prefer semantic Elixir metadata and helpers first.
  • Keep compact schema-specific output as macro invocations when the expanded body would be repetitive.
  • Use RustQ AST or Rusty-Elixir helpers for local expression/arm/function generation where it remains compact, such as skip field arms used by downstream custom templates.
  • Declare a Decoder method in the reference decoder.rs before adding unwrap!, verbose propagation case expressions, or duplicate method metadata to a helper.
  • Do not replace compact decoder macro invocations with fully expanded Rust just to say the generator is "AST-backed" or "defrust-backed".

A good change should make the Elixir generator clearer without increasing large schema output size.

Tests

KiwiCodec.RustlerGenerator.NativeTest builds the generated decoders for every feature with the reference Decoder, loads them as NIFs, and checks what they return against KiwiCodec.decode/2 and KiwiCodec.SchemaInterpreter. A generator change that produces Rust that does not compile, or decodes differently, fails there; tests of the generated text are not needed for it.

Checklist

Before committing generator changes:

  1. Generate Rust for a schema that exercises full, sparse, and skip decoders, before and after the change, and compare the size.
  2. Run mix ci, which builds and runs the generated Rust.
  3. Generate Figler's decoders with this checkout, as Figler's schema is the largest consumer: set KIWI_CODEC_PATH in Figler, run mix rustq.gen, and check that its crate compiles. Restore Figler's generated files afterwards.