# Internals

How KiwiCodec is put together, for people changing it. Using the Rustler
generator is covered in [Native decoders](native-decoders.md).

## Layout

- `lib/` is the codec: the schema parser and binary schema format
  (`KiwiCodec.Schema.*`), wire primitives (`KiwiCodec.Wire.*`), the DSL and the
  modules it generates (`KiwiCodec.DSL`, `KiwiCodec.GeneratedModule.*`), encoding
  and decoding of generated structs (`KiwiCodec.Encoder`, `KiwiCodec.Decoder`),
  and the schema interpreter. It has no dependency beyond `varint`.
- `codegen/` is the Rustler generator. Every module there is wrapped in
  `if Code.ensure_loaded?(RustQ)`, because RustQ is optional: projects that
  depend on KiwiCodec get the generator only where they also depend on RustQ.
- Tests mirror the source tree. `test/package/` builds the package and compiles
  it in two consumer projects, one without RustQ and one generating Rust.

## Rustler generator

The generator serves two goals at once:

1. Keep generator code semantic and maintainable in Elixir.
2. Keep generated Rust compact enough for large schemas, such as Figma's.

For repetitive schema decoders, those goals are served by a compact macro
boundary rather than by fully expanding every decoder body.

### Layers

The generator is organized as a small pipeline:

1. Schema definitions are parsed and selected by `KiwiCodec.RustlerGenerator`.
2. One generator per feature, `Full`, `Sparse`, `Skip`, and `Records`, turns each
   definition into a call to a compact Rust macro, such as
   `kiwi_message_decoder!`, `kiwi_sparse_struct_decoder!`, or
   `kiwi_skip_message_decoder!`, with a row per field. `Field` says how a field
   is read or skipped, and `Name` names every generated function.
3. Helper modules define those macros and the functions they call, with RustQ
   `defrustmacro` and `defrust` typed against the reference `decoder.rs` beside
   `KiwiCodec.RustlerGenerator.Decoder`: `Values` for what full and sparse
   decoders share, `Full.Helpers`, `Sparse.Helpers`, and `Skip.Helpers` for
   each feature, and `Sparse.Messages`, `Skip.Messages`, or
   `Sparse.SkipMessages` for messages, depending on which features are
   generated together. `Primitive` defines the readers and skippers of each
   primitive type from one table of `Decoder` methods.
4. `KiwiCodec.RustlerGenerator.Splice` includes each needed helper module whole,
   through its `__rustq_items__/0`.
5. `KiwiCodec.RustlerGenerator.Entrypoint` emits a NIF per entrypoint calling
   `kiwi_decode_binary` with the definition's decoder.

This split keeps schema logic in Elixir while avoiding thousands of repeated
expanded Rust function bodies.

Three pieces are shared by every schema and written as Rust under `templates/`:
the record runtime, the runtime schema that skips unknown fields, and the atom
cache. They are ordinary Rust APIs, documented for the Rust code that uses
them, rather than repetitive decoder shapes. The record runtime and atom cache
are named after the modules that provide them, `templates/records.rs` for
`records.ex`, so RustQ knows their signatures and RustQ code elsewhere names the
module rather than the file, and so is the runtime schema,
`templates/unknown_field_runtime_schema.rs` for
`unknown_field_runtime_schema.ex`, which `UnknownFields` includes in one mode.

### Compact RustQ macro boundary

The macros selected by `Splice` are a compact RustQ-authored boundary. They are
shared implementations for highly repetitive decoder shapes, but they should be
authored through `defrustmacro`, `defrust`, type metadata, and RustQ AST helpers
rather than through raw Rust heredocs or ad hoc token strings.

When changing full, sparse, or skip decoder generation:

- Prefer semantic Elixir metadata and helpers first.
- Keep compact schema-specific output as macro invocations when the expanded
  body would be repetitive.
- Use RustQ AST or Rusty-Elixir helpers for local expression/arm/function
  generation where it remains compact, such as skip field arms used by
  downstream custom templates.
- Declare a `Decoder` method in the reference `decoder.rs` before adding
  `unwrap!`, verbose propagation `case` expressions, or duplicate method
  metadata to a helper.
- Do not replace compact decoder macro invocations with fully expanded Rust just
  to say the generator is "AST-backed" or "defrust-backed".

A good change should make the Elixir generator clearer without increasing large
schema output size.

### Tests

`KiwiCodec.RustlerGenerator.NativeTest` builds the generated decoders for every
feature with the reference `Decoder`, loads them as NIFs, and checks what they
return against `KiwiCodec.decode/2` and `KiwiCodec.SchemaInterpreter`. A
generator change that produces Rust that does not compile, or decodes
differently, fails there; tests of the generated text are not needed for it.

### Checklist

Before committing generator changes:

1. Generate Rust for a schema that exercises full, sparse, and skip decoders,
   before and after the change, and compare the size.
2. Run `mix ci`, which builds and runs the generated Rust.
3. Generate Figler's decoders with this checkout, as Figler's schema is the
   largest consumer: set `KIWI_CODEC_PATH` in Figler, run `mix rustq.gen`, and
   check that its crate compiles. Restore Figler's generated files afterwards.