Umbrella: parse once into Elements, and make every component consume them (#211) - #229
Draft
milyin wants to merge 5 commits into
Draft
Umbrella: parse once into Elements, and make every component consume them (#211)#229milyin wants to merge 5 commits into
milyin wants to merge 5 commits into
Conversation
Umbrella document for making every prebindgen component consume `Element`s instead of parsing captured Rust itself: the design and the rule it turns on, the measured size of the problem (202 classification sites, 113 registry map reads), the stage order L0–L5, and the completion criteria restated from #211. This file is the authority on stage state; the umbrella PR body mirrors it. Refs #211. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
|
Stage L0 is up as #227 — |
…227) * jnigen: derive a return expansion from a value form (#213) (#221) * jnigen: derive a return expansion from a value form (#213 Gap A) `expand_return!(T).fields(fields!(t_to_struct))` takes T's output fields from its value form — the struct gathering its own accessors — instead of restating them. Two of zenoh-flat-jni's five hand-written lists had already drifted from the struct they mirror; a derived list cannot. `.fields()` is `.field()` applied to each struct field, so it keeps the same rule: a field crosses by ITS OWN type's default output boundary. A field type with an `expand_return!` splices it (a KeyExpr field still crosses as its string, not as a handle), a declared data class inlines, a field behind Option/Vec stays one leaf. Adopting it therefore preserves the boundary shape a hand-written list already had. Per-field adjustments live on the `FieldsDecl`, keyed on the Rust field ident like `FunctionDecl::expand_param`: `.field(name, expand_return!(..))` replaces one field's decomposition, `.name(name, "kt")` renames its leaf. Naming a field the struct lacks is a hard error — that is the drift this declarator exists to catch. Core changes: - `UnfoldLeaf.path` becomes `Vec<PathStep>` (`Call` / `Field`, each carrying its own optionality) so one path can mix accessor calls and field reads. Behaviour-preserving for every existing producer. - `DeconRecord::Fields` + `FieldRecord`; the adapter walks the struct (it knows which are declared classes), core decides per field whether to splice, and rides the existing visited/Cycle guard. - `UnfoldPlan.root_call` hoists the value-form call to one local, so the struct is built once per delivery rather than once per field. - `Prebindgen::deconstructors` now takes `&Registry`, matching `value_struct_decons` — a value form's fields come off the indexed struct. Sum-typed fields (ReplyStruct.result) are not covered yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: a sum-typed field of a value form (#213 Gap A, sums) `ReplyStruct { result: ReplyResult, .. }` — a `sealed_class!` field of a value form now decomposes in place into its selector and one leaf group per alternative. A sum has no whole-value converter by construction, so this is the only shape in which it can cross at all. The user-facing callback still receives ONE typed `ZOutcome`: the tag and group slots collapse into a single parameter rebuilt by an inlined `when`, reusing the `GroupDesc` collapsing that a fixed-builder arg already uses. Handing the raw slots over would have defeated the `sealed_class!`. Generalizations, both behaviour-preserving for a sum in the whole-return position (its 20 existing tests are unchanged): - the selector leaf carries the sum's own type as its `out_ty`, so the emitter finds the enum to match on from the leaf rather than from `plan.source` — which names the CONTAINING value once a sum is a field; - `encode_sum_leaves` becomes `encode_sum_group`, taking one sum's leaf segment plus the expression to match on. `encode_plan_leaves` segments the leaf list and emits one match per sum instead of the whole plan being handed to the sum emitter; a whole-return sum is the degenerate case of one segment covering everything. `Vec<sum>` and `Option<sum>` fields are refused by name: the first has variable arity, the second would need a present flag beside its tag that an output leaf list cannot carry (the `fromParts` bridge's `PlanFieldKind::Sum` can, which is why a data-class field may be `Option<sum>`). Also restores examples/example-cbindgen goldens, which the previous commit picked up from an --all-features regeneration. The generator output is unchanged; only the committed artifact was wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * examples: restore example-cbindgen goldens to the plain-build variant An earlier `git add -A` in this branch swept in an --all-features regeneration, whose FEATURES guard reads "example-flat/internal example-flat/unstable" instead of "". `examples/regen-check.sh` builds with default features, so the committed artifact has to be the default-feature one — this is what CI checks. The generator output is unchanged either way; only the committed file was wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * covertest: exercise the derived value-form boundary on the JVM (#213) Library tests alone do not count as coverage in this repo, so `.fields()` gets a real round trip: `perftest_flat::ext::Report` is a handle whose output boundary is declared from its value form, with each field landing on a different rule of the expansion — summary a type with its own expand_return! ⇒ spliced into (count, total), NOT handed over as a handle taken Option<data class> ⇒ one leaf origin a non-optional data class ⇒ inlined into its fields outcome a sealed_class! ⇒ selector + one group per alternative, carrying a handle label a plain leaf `Test.kt`'s new section is itself the assertion: the callback signature would not compile if any field had been derived wrongly. It also pins the ownership contract for a handle reached through a value form and a sum group — live inside the callback, still live after, the receiver's to close. 47 sections pass on a real JVM. Adds the Gap B unit test the issue asked for: a handle-payload sum in DATA-CLASS FIELD position, the one position return/callback coverage did not reach. It works — and the test pins two consequences that were previously unstated: the container is NOT AutoCloseable (a sum payload is the receiver's to close, unlike a plain handle field, which cascades), and a sum field pushes its parent onto the whole-value fromParts bridge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: address review on #221 — three value-form defects P1 — a single-leaf value form passed a borrow to an owned converter. One leaf makes core pick `Delivery::Return`, whose reach is composed separately in `emit/wrapper.rs`'s `is_convert` path. That path rendered a `Field` step as `&(expr).field` and returned it, so a plain field leaf — whose `out_ty` is the field type as written — got `&F` where its converter takes `F`, and a non-`Copy` field additionally borrowed out of the temporary the value-form call returned. It now clones the reached place, the same treatment `encode_plan_leaves` gives a `LeafSource::Field` leaf; an identity leaf stays borrowed, since its converter IS the borrowed-opaque clone. P2 — a per-field override did not validate its declared type. `.field("key_expr", expand_return!(ZBytes)...)` was accepted for a `ZKeyExpr` field whenever both were declared handles, and an override silently outlived an upstream field-type change — the exact drift `.fields()` exists to catch. The declared key is now compared against the peeled field type and names both, matching the target checks on the per-function expansion APIs. P2 — nested value forms were not hoisted. `root_call` only searched the declaration's top-level records, so a field splicing a child whose own boundary is also derived rebuilt that child once per child leaf, breaking the stated "called once per delivery" contract. Replaced by `UnfoldPlan.hoists: Vec<Vec<PathStep>>` — the path prefixes to bind once, recorded where `flatten` descends and therefore outermost-first. Each is composed from the longest already-bound prefix of itself, and each leaf reaches off the innermost hoist it sits under: let __vf0 = z_outer_to_struct(&arg); let __vf1 = z_inner_to_struct(&(&__vf0).inner); This also removes the single-value-form special case rather than adding a second one beside it. Three regression tests, one per finding. The only generated-output change is the `__vf` -> `__vf0` rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: validate nested value-form field shapes * jnigen: consuming value forms — move the fields instead of cloning them `.fields(fields!(f))` now accepts a value form that takes its receiver BY VALUE. Such a form destroys the object into its parts, so the generated code moves the value in and moves each field OUT into its leaf — the clones the borrowing form pays disappear entirely. This is what the hot receive path wants and what zenoh itself recommends: `From<Sample> for SampleFields` exists, in zenoh's words, because it "allows deconstructing a sample to fields without cloning, which is more efficient than using getter methods". Every callback hands its value over owned (`impl Fn(Sample)`), so there is nothing to preserve — the borrowing form clones fields out of a value it is about to drop. Measured on covertest's `Report`: six clones removed from the callback body, `report_into_struct(__cb_arg0)` moved in, every field moved out. Consuming-ness is INFERRED from the accessor's signature, so it cannot drift from it, and both forms stay usable side by side. Because a consuming form moves the value, two shapes are refused at declaration time rather than emitted as Rust that cannot compile downstream: a sibling record (`.field_self()` or another `.field()` would read a moved value), and a form reached through another value form (it would move a field out from under the parent's other leaves). A `&T`-returning function clones once up front and consumes the clone, so one declaration still serves owned and borrowed returns alike. Two supporting changes: - The reach derivation is now SHARED (`reach_leaf_flat`) between the multi-leaf encoder and the single-leaf `Delivery::Return` shortcut in emit/wrapper.rs. Deriving it twice is what let them drift into the P1 defect; the shortcut also now refuses an optional intermediate step explicitly instead of composing code that cannot type-check. - Reaches project the leading run of plain field steps DIRECTLY (`&v.a.b`) instead of through a borrow of the base (`&(&v).a.b`). The two name the same value, but the second borrows the base as a whole, which the borrow checker rejects once a sibling leaf has moved another field out — so without this, field moves compiled only while the borrowing leaves happened to be declared first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: `.fields_into()` — declare the consuming value form, and let it nest `6133f91` taught `.fields(fields!(f))` to accept a by-value accessor and INFERRED consuming-ness from its signature. That reads the decision off the wrong thing. Giving the value away is a boundary decision — the same one `.field_self()` makes, which is exactly why the two cannot coexist — not a property of which function happened to be named. So the collision surfaced as a resolve-time error phrased as a restriction on `.fields()`, when it is really two declarators to pick between. Now the decl says which it wants: .field_self() the value itself, whole .fields(fields!(to_struct)) a copy of its parts .fields_into(fields!(into_)) the value itself, as its parts `.fields_into(..)` must be the decl's only record — a `.field_self()` or a sibling `.field(..)` would read a value that is gone — and that is now a panic in the declarator, in BOTH orders, rather than an `UnfoldError` found a resolve later. The declared flag and the accessor's receiver are cross-checked when the records are flattened, so intent still cannot drift from the signature; naming the wrong one of a `to_struct`/`into_struct` pair is an error that says which declarator the accessor belongs to. The nesting refusal is GONE. Its stated reason — "it would move a field out from under the parent's other leaves" — does not hold: a hoisted value form is an owned struct, its fields are disjoint, and `project_leading_fields` (same commit) already stopped leaves from borrowing the base as a whole. So a nested consuming form is handed the parent's field BY MOVE: let __vf0 = z_outer_to_struct(&__cb_arg0); let __vf1 = z_inner_into_struct(__vf0.inner); // moved, not cloned … __vf0.tag … // sibling leaf, still fine `compose_step` borrows (`&(e).f`), so the field run to that field is projected in the hoist loop instead of going through it. A nested form reached through an accessor CALL holds a borrow with nothing to give up, so it clones once and consumes the clone — the same fallback a borrowed root already takes. That was the one place an available `_into_struct` went unused for no reason. Verified: 438 lib tests (three retargeted, five new — both collision orders, both signature-mismatch directions, and the nested move under a borrowing AND a consuming parent), covertest-kotlin's 47 JVM sections, regen-check byte-clean. The generated output for covertest is unchanged — same accessor, same moves; only the declaration that names it moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: address review on #221 — consuming ownership in two more places Two review findings, both cases where `.fields_into(..)` promised a move and the emitter did not deliver one. [P1] The single-leaf `Delivery::Return` shortcut never consulted the plan's hoists. It composed its reach straight off the raw value, so a one-field value form declared with `.fields_into(..)` emitted (&(myflat::z_one_into_struct(&__cvsrc)).label).clone() — `&ZOne` handed to a by-value receiver, ill-typed in the consumer's crate before you even reach the pointless clone. Every consuming test so far produced a MULTI-leaf callback plan and went through `encode_plan_leaves`, so nothing covered it. The hoist loop is now `bind_hoists`, shared by both paths, and `reach_leaf_flat` takes the rebased path plus its hoist's `consuming` flag. The shortcut binds the same `__vfN` locals as the multi-leaf encoder and reaches the leaf off the innermost one. That is the same fix that was applied to the reach itself in `6133f91` and for the same reason: two derivations of one question drift. [P2] The identity branch computed `consuming` and then returned before using it. Only a handle at the owned ROOT (empty path) moved; a handle FIELD always took the clone-via-converter arm: ZChild_to_jlong_...(&mut env, &__vf0.child) despite the parent form having given its value away — a preserved clone, and a `Clone` bound the handle type need not have. The branch now computes the owned PLACE (the root, or a plain-field run under a consuming hoist) and boxes it, `Box::into_raw(Box::new(__vf0.child))`. Both regressions reproduce the reviewer's exact shapes and both fail without the corresponding fix (verified by stashing each). Sum payloads, which the P2 comment also flagged, are NOT fixed here: filed as #228. `encode_sum_group` matches by reference and clones every payload kind through one chain, so moving means reworking that emitter's ownership model — the selector reads the same matched value, and an owned handle payload wants the identity branch's box rather than the borrowed-opaque converter. Not an addendum to this PR. Verified: 440 lib tests, covertest-kotlin's 47 JVM sections, regen-check byte-clean (neither shape occurs in covertest, which is why its goldens do not move — the unit regressions are what pin them). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: decide leaf ownership in the plan, not in each emitter Two more review findings on #221, both the same defect wearing a different hat: an identity (handle) leaf under a consuming value form was still reached as a borrow, so the borrowed-opaque converter cloned it — and demanded a `Clone` the handle type need not have. * A value form whose SOLE field is a handle takes the single-leaf `Delivery::Return` shortcut. `bind_hoists` called the by-value accessor correctly, then the shortcut returned `&__vf0.child`, because its consuming case only covered `LeafSource::Field`. * An `Option<Handle>` field was excluded by the previous fix's plain-field test, leaving `match &(&__vf0).child { Some(__n0) => …clone… }` — an ordinary optional handle field, not the sum limitation of #228, and the commonest shape there is (`SampleStruct.attachment`). Patching each emitter would have been a third special case for one question. The question belongs to the PLAN: `place_is_owned` now decides, where an identity leaf's `out_ty` is chosen, whether the value at that path is the plan's to give away — the root of an owned plan, or a field of a form that CONSUMED its value, reached by a movable run of steps. An owned `out_ty` IS that statement, and it already selects the owning converter, so every emitter follows one decision instead of re-deriving it. `steps_are_movable` (plan.rs) is that run: field reads only, with an `Option` allowed on the LAST one — a `None` arm still hands the whole `Option` over by value, while an `Option` in the middle must be unwrapped and so can only be borrowed through. The resolver and both emitters read the same predicate; two readings would drift, and the disagreement is a borrow handed to an owning converter. Emitters then just project the place: * `reach_leaf_flat` moves whenever the leaf owns its `out_ty` — field and identity leaves alike. It keeps requiring a plain-field run, since return delivery has no `None` arm for a trailing `Option`. * The nullable identity branch matches the `Option` BY VALUE and boxes the `Some` payload, instead of matching a borrow of it. Both regressions reproduce the reviewer's shapes and fail without the fix (verified by stashing it). 442 lib tests, covertest's 47 JVM sections, regen-check byte-clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: a nullable sole leaf is a callback delivery, not a return `single_return` chose `Delivery::Return` on leaf COUNT alone. A value form whose only field is an `Option<Handle>` therefore landed on the flat return path, which has no `None` arm and whose `convert_out_ty` names the leaf's own type rather than an optional of it — so it composed &(&__vf0).child into `ZChild_to_jlong(.., __out)`, typed for `ZChild`. The downstream crate does not compile. Making `out_ty` owned in 421531e addressed move-vs-clone; it says who frees the handle, not whether there is one. Absence is a DELIVERY question. Callback delivery already has the arm — the leaf crosses as a boxed `Long` or JVM null — so a nullable leaf goes there, which is one condition on `single_return` rather than teaching the shortcut to match and map a trailing option it has no way to represent in its return type. Nullability here only ever comes from an `Option` with something DECOMPOSED below it (a `.field_self()` handle, a nested value form); a plain leaf's own `Option` rides its converter and leaves the leaf non-nullable. So no shape that returns today stops returning — regen-check is byte-identical and covertest's 47 sections are unchanged. Regression reproduces the reviewer's shape and fails without the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: an owned root identity moves on the flat return path too The flat return path asked the wrong question. It tied the move to the rebased hoist's `consuming` flag, but "a consuming form gave it to me" is only ONE of the two ways a leaf owns what it reaches. A plain `-> ZChild` return under the type-level `expand_return!(ZChild).field_self()` — the declaration that exists so the same boundary can be spliced as a value-form field — has no hoist at all, so `consuming` was false and the path emitted let __cvsrc = myflat::z_root_child_make(); { &__cvsrc } into the OWNING `ZChild_to_jlong`, whose argument is `ZChild`. Same mismatch inside the `map` closure of an `Option<ZChild>` return. For an identity leaf the plan already states ownership — that is what `place_is_owned` decides and what selected the owning converter — so the emitter reads it off `out_ty` instead of re-deriving it. A field leaf keeps asking the enclosing form, since its `out_ty` is the field type as written and owned either way. That predates this PR: the previous shape of this path composed `&base` for an empty path regardless. The callback emitter has always treated the owned root as an owned place; now both do. Regression covers the plain and the `Option` return and fails without the fix. 444 lib tests, covertest's 47 JVM sections, regen-check byte-clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * jnigen: rename `.fields_into()` to `.fields_self_into()` Puts the declarator squarely in the `field_self` family it belongs to, which is the whole point of it being its own declarator: `.field_self()` hands the value over whole, `.fields_self_into(..)` hands *the value itself* over as its parts, and `.fields(..)` hands over a copy of its parts. `self` is what the first two share and what makes them mutually exclusive. Mechanical: the method, the two panic messages, the doc links, the covertest declaration and its coverage-table row. Generated output is unchanged — regen-check byte-clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * Parse the record stream into elements that keep their syntax `Language` turns a captured `(syn::Item, SourceLocation)` stream into `Element`s: a closed, destination-neutral classification paired, at every level, with the exact syntax it was built from — the item, each parameter, field, variant and type. The pairing is the point. Issue #211 asks that adapters stop re-reading captured Rust, and the natural reading of that — a syn-free semantic model — makes the model responsible for reconstructing Rust too, because the generated glue is itself a destination artifact. That pressure is what turns a language-neutral IR back into a second `syn`: a delimiter, a lifetime and a literal's base all have to be modelled so they can be re-emitted. Keeping the original slice costs nothing and removes the pressure, so the classification stays small: Element::Enum → Variant { tag, discriminant: Option<i64>, fields, syntax } `B()` is a unit *group* and still spells `E::B()`, because `Variant::spell` reads the delimiters off `syntax`. `= 0x07` reaches a C header as `0x07` while Kotlin gets the number 7. Neither is a modelled fact. The rule for consumers is therefore: **classify off `kind`, spell off `syntax`.** #224's boundary ledger measures exactly that without adaptation — it counts variant mentions of `syn::Type` / `syn::Expr`, so `quote!(#slice)` is invisible to it and `matches!(ty, syn::Type::Reference(_))` is not. It is ported here and seeded at 202 sites, the population the adapter migrations pay down. Acceptance is preserved, not expanded. An item the language cannot express becomes `Element::Unsupported`, carrying its diagnosis: the pipeline has always scanned a signature only once an adapter declares it, and a source crate may mark items no binding uses. Only a duplicate name — which no declaration can disambiguate — fails the parse. Nothing consumes elements yet; `Registry::from_elements` is the next step. Ported from the #215 branch: the array-length subgrammar (#212), the type grammar and its acceptance tests, enum tag/discriminant numbering (#226), the ledger (#224). Refs #211. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make the element model logical, not Rust-shaped `TypeKind` still named Rust type constructors where it should have named concepts, and the identity of a nominal type was a `syn::Path` sitting inside the classification — a position the boundary ledger cannot see. The test a variant has to pass is whether a *destination* language would act on the distinction; if only Rust can tell, it is spelling, and the syntax slice already carries it. Twelve variants become ten: * `Slice` folds into `Sequence`. `Vec<T>` and `[T]` are one concept — a run of `T` — and ownership is already the `Ref` layer's fact, so a second variant encoded it twice. This is what the pipeline does anyway: one `Shape::Iterable` covers both, and jnigen rewrites a `&[T]` input into the `Vec<_>` pattern. * `Boxed` goes. `Box<T>` **is** `T`: owned either way, and nothing outside Rust can tell. It classifies as what it wraps, and the `Box` survives where it matters — in the syntax generated Rust spells. * `Ptr` goes. No source crate writes a raw pointer, neither adapter has a selection arm for one, and accepting it *widened* acceptance, which this stage was not supposed to do. * `Str` covers `str`, so `&str` is a borrowed string rather than a reference to a nominal type nothing can resolve. It is the most common non-scalar parameter in the whole ecosystem, and both adapters already special-case it by name. * `Named` carries a `TypeId` — a name — instead of a `syn::Path`. The same test applied to the elements: a function's return is a `Type`, unit when elided, because no consumer distinguishes that from `-> ()` (eight of them normalize one to the other on the spot). A struct's fields are `Option<Vec<Field>>` — a product, or opaque — because named/unnamed/unit were three Rust shapes where `Variant` already modelled the same idea as a field list plus delimiters read off the syntax. `spell.rs` now holds everything that turns an element back into Rust tokens, so `element.rs` describes structure alone, and `Struct::spell` joins `Variant::spell` as the dual that makes the shapes unnecessary. Two things move to where they belong: `Language::parse` normalizes before lowering (`ty.rs` already assumed it had), and the callback grammar `extract_fn_trait_args` lives in the language rather than the registry — one ledger site paid down, 202 to 201. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Delete the passthrough element A `#[prebindgen]` crate marks the items that cross the boundary; the supporting code around them belongs to the consumer. The proc-macro already enforces that — marking a `use`, `mod`, `impl` or `macro_rules!` is a compile error at the mark site — so the variant's own doc listed items that could never reach it. What actually reached it was one thing: the `const _` feature guard, which is not a source item at all. `CfgFilter` synthesizes it and prepends it to the stream, so `Passthrough` existed to carry an item prebindgen itself wrote. It is a const, so it is modelled as one, and `Element::name` returns `None` for `_` — which is the real fact, and the one that lets several sources' guards coexist in the flat namespace. `write.rs` already had that rule for consts (`*ident == "_"` bypasses the declaration gate), dead until now because `const _` never reached the consts map. That leaves `union` and a type alias, the two kinds the macro accepts and the frontend does not model. Neither is written by any source crate in the ecosystem. They become `Unsupported` with a diagnosis naming the kind, rather than being copied verbatim into generated code that would reference source types by bare name — so the mark site and the frontend now disagree about exactly two kinds, and disagree loudly instead of silently. `Unsupported::name` becomes optional, since an item kind may have no identifier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Give every node one Origin: its syntax, and where that syntax came from The classification is now logical, but its other half was still ad-hoc. `syntax` sat on nine node types as nine separate fields; `location` sat on the five item types only, because a captured record is per-item and a component has none of its own. That asymmetry had a cost. The one semantically load-bearing part of a location — the crate name — was reachable at item level only, so it got copied downward by hand, under a third field name, with drifting meaning: `ConstId.origin` is the crate a const was *declared* in, while `TypeId.origin` was the crate of the item *using* the type. The latter was also part of `TypeId`'s derived `Eq`, so `Sample` referenced from two source crates compared unequal — one type with two identities, three lines under a doc calling the name "the whole address". The two facts are orthogonal and neither derives from the other. `syn` tokens normally carry spans, but the proc-macro serializes each item as a string into JSONL and `build.rs` re-parses it, so every span in a slice points into an anonymous buffer; `SourceLocation::from_span` captures file/line/column while real rustc spans still exist, precisely because they cannot survive the trip. So every node now carries `Origin<S> { syntax: S, location: Rc<SourceLocation> }` — item, parameter, field, variant, type, and the array extent, which had no syntax at all and now spells its own length. Generic, so the typed slices survive; `Rc` because the model holds `syn` and is `!Send` regardless, the call `TypeKey` already made. One captured record is one item, so an item and every node lowered out of it share one allocation, which is both the honest answer to "where is this field" and the cheap one. With provenance arriving on its own, `item_crate: Option<&str>` stops being threaded through six lowering functions, `TypeId` is a name alone, and `ConstId.origin` becomes `ConstId.crate_name` — a crate that belongs to a *different* item, not this node's provenance. The rule, now stated where it can be read: a reference carries a name, the declaration carries the origin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * A variant's position is an index, not a tag `Variant.tag: i32` and `Field.index: usize` were one fact under two names: the ordinal of a child within its parent's ordered list. Sum versus product is already carried by *which* list it is — `Enum::variants` or `Struct::fields` — not by the number. The defence for keeping them apart was that a tag is transmitted while an index is only used to address a field. That defence was made of adapter behaviour: `i32` because cbindgen writes `c_int` and jnigen writes `jint`. Deciding a frontend field's shape from two generators' wire types is exactly the coupling this module exists to prevent, and it is the same test that stripped `Boxed` and `Slice` — a fact earns its shape from what the source means, not from what one adapter does with it. Transmitting the position to say which alternative is live is one destination's choice; another may send a name. The signedness had no defence at all: a declaration-order position is `0..N-1`. So `Variant.index: usize`, matching `Field.index`, and both documented as the same fact for the same reason — a node handed out on its own still knows where it sits. What remains genuinely distinct is `Variant::discriminant`: a position is where the source *put* a variant, a discriminant is the value Rust *assigns* it, and the two are independent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Address review: extent identity, callback returns, i64::MIN Three correctness fixes before this becomes the model later stages consume. **`ArrayExtent` had an equality that was neither identity it could have been.** It compared `value` and `source`, so `[u8; A]` differed from `[u8; 4]` when `A == 4` — one Rust type reported as two — while `[u8; 4]` equalled `[u8; 0x04]`, whose retained syntax differs. So it was not type identity and not spelling identity, and its own doc claimed the first while the code did neither. There is no single equality that could be right, because the extent answers three different questions, so it now provides none and each consumer projects what it needs: `value` for type and converter identity, `origin.syntax` for a C declaration's spelling at that occurrence, `const_id()` for which consts must reach the header. A regression pins all three apart — same value with different const dependency, same value with different spelling, same value with different const. The doc also records what a converter table will need: `value` being the identity means occurrences share one converter with differing spellings, so a canonical spelling must be chosen deliberately rather than inherited from whichever occurrence populated the entry. **The callback grammar silently dropped a return type.** `extract_fn_trait_args` read `ParenthesizedGenericArguments::inputs` and never `output`, so `impl Fn() -> u8 + Send + Sync + 'static` was accepted as `Callback { args: [] }`. `TypeKind::Callback` has no slot for a return and the grammar's own error text says a callback returns `()`, so the fact was lost — silently, which is worse than refusing. A non-unit return is now refused, a written `-> ()` still accepted, both with tests. No source crate in the ecosystem writes a returning callback, so nothing real narrows. The helper predates this PR, but making it the authoritative frontend classifier is what would have made the loss irreversible for every later consumer. **`i64::MIN` was not a discriminant.** `int_literal` parsed the magnitude as `i64` before applying the sign, so `-9223372036854775808` — valid Rust — failed at the digits. The magnitude is now parsed as `i128` and range-checked after negation, with a regression at the bottom of the range and one step past it. Along the way, `is_unit_type` becomes the language's one answer to "is this `()`", used by both the type lowering and the callback check. `types_util::is_unit` could not serve: it is gated behind `unstable-cbindgen`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Address review: async, variadic, generic binders, and a ledger hole Three more shapes the frontend accepted but could not represent, and one hole in the check that is supposed to catch exactly this class of thing. **`async fn` was the dangerous one.** `Function` has a direct return, so `pub async fn ping() {}` lowered as a function returning `()` — a generated wrapper would call it, drop the future, and export a function whose body never runs. A **C-variadic** tail was dropped from the signature just as quietly. Both are now `ItemError`s. **A type or const generic parameter is refused.** The elements have no generic binder, so a `T` in a field or parameter lowered as `TypeKind::Named` — an ordinary reference into the flat namespace, indistinguishable from a real item called `T`, which loses the scoping every downstream resolver needs. Modelling binders and substitution is the other option; refusing is the right one, because no destination language can express an uninstantiated parameter, and the source crates already write concrete types per instantiation. The diagnosis says so. Two things are deliberately *not* generic binders, both tested. A lifetime parameter: lifetimes are spelling and the spelling already travels, the same call `lower_type` makes for a lifetime argument. And `impl Trait` in argument position — Rust calls it an anonymous type parameter, but `syn` does not desugar it into the binder list, so the callback form every callback-taking source function uses is untouched. **The boundary ledger could be evaded.** `is_cfg_test` treated any predicate containing the ident `test` as test-only, so a classifier under `#[cfg(not(test))]` or `#[cfg(any(test, feature = "x"))]` was skipped — in a production build. It now matches the exact predicate `cfg(test)` and counts everything it cannot prove test-only, which is the safe direction for a check whose job is to stop a classifier hiding. `cfg(all(test, ..))` is genuinely test-only and is counted anyway; nothing in the tree writes one, and widening it later should be a deliberate edit with a ledger diff attached. The count does not move: every `cfg` on an item in the tree is either exactly `cfg(test)` or mentions no `test` at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Let Language read a source directory, not just a stream A build script's whole prebindgen preamble was two steps and a binding it did not otherwise want: let source = prebindgen::Source::new(zenoh_flat::PREBINDGEN_OUT_DIR); let registry = Registry::from_items(source.items_all())?; `Language` now folds the first step in, so naming the directory is enough: let elements = Language::new() .source(zenoh_flat::PREBINDGEN_OUT_DIR) .parse()?; That is five of the six consumer build scripts — zenoh-flat-jni, zenoh-flat-c, perftest-c, perftest-kotlin, example-cbindgen — which use nothing of `Source` but `new` and `items_all`. Reading a stream is kept, as the general case rather than the only one: `items()` takes any `(syn::Item, SourceLocation)` iterator, so everything a `Source` can express still composes — a group selection, a renamed dependency (covertest-kotlin's `crate_name` override, the sixth build script), several sources at once. `source()` is sugar over it. The other four knobs on `Source`'s builder — group selection and feature/target filtering — are reachable this way and were not mirrored, because no build script in the workspace calls them. The feeders accumulate and `parse` consumes, rather than each input being parsed as it arrives. That is forced, not stylistic: the rules that make a parse fail are whole-stream — one flat namespace, one const index an array length may reach into, one set of source modules to normalize against — so every input must be in hand before any of it is classified. A test now pins both directions of that: a length in one feeder resolving a const from another, and a duplicate name across feeders still failing. `Language` and `Element` join `Registry` in the `core` facade, since they are what a build script names; the rest of the element model stays in `core::language`, where an adapter reaches for it. The four doc examples on `Language` are now real doctests rather than `ignore` blocks — `Source::init_doctest_simulate` was already there to make that possible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Split the two enum shapes: a Variant is not an Enum `Element::Enum` covered both a payload-carrying enum and a fieldless one, on the theory that the second is the degenerate first. They are two entities, and the evidence is in how they are numbered. A sum's alternatives are identified by **position**: cbindgen states it outright — "the mirror carries no explicit discriminants, so its tags are declaration order `0..N`" — and jnigen's sum emission mentions `discriminant` exactly zero times against eleven uses of the position. A fieldless enum's members are identified by the **value Rust assigns**: a C header re-states each `= expr`, and a Kotlin `enum class` entry is `NAME(7)`, with position only a fallback when the discriminant is not a literal. So one model covering both carried a field dead in each direction — and worse than dead on the sum side, because Rust *does* assign a discriminant to a payload alternative and using it would be wrong. The unified model invited exactly that mistake. Element::Variant(Variant { alternatives: Vec<Alternative> }) // a sum Element::Enum(Enum { values: Vec<EnumValue> }) // C-style `Alternative` carries `index` and `fields` and no discriminant; `EnumValue` carries `index` and `discriminant` and no fields. `discriminant_values` belongs to `Enum` alone now. `is_unit` and `first_payload_variant` are gone: the first was the classification, which `lower_enum` now makes once, and the second existed to name an offender to an adapter that only accepts fieldless enums — such an adapter matches `Element::Enum` and never sees the other shape. Both shapes still spell delimiters off their own syntax, because `A`, `B()` and `C {}` are fieldless alike and Rust demands the delimiters wherever the last two are named — so `spell` is on `Alternative` and `EnumValue`, over the one `spell::fields`. `enum E {}` and an all-empty-group enum are `Enum`; one field anywhere makes the item a `Variant`, and a sum may still mix empty and payload-carrying alternatives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…232) * Rename the module to flat: these are the flat API's elements `core::language` modelled one thing and was named for another. What it parses is the **flat API** — the single flat namespace a `#[prebindgen]` crate exports — so `Language` becomes `Flat` and `api/core/language/` becomes `api/core/flat/`. Mechanical, and separated from the model changes that follow so those arrive as a readable diff. The boundary ledger's skipped-path constant and header move with the directory; the count does not change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Element is a function, a type, or a constant `Element` mixed two levels: `Function | Struct | Variant | Enum | Const` set type declarations beside functions and constants, when the kinds a binding distinguishes are a function, a type, and a constant. Types now group under `Element::Type`, and the type *reference* — which held the name `Type` — becomes `TypeRef`, so a declaration and a use site stop sharing a word. `Opaque` becomes the entity for a type whose contents do not cross, and it arrives two ways: * `#[prebindgen] pub type X = path;` — this **reverses** #227, where a marked alias was `Unsupported`. It is now how a handle enters the flat API deliberately: a foreign or crate-private type gets a name here without any claim about its contents. That is what makes the API closable, and it is the prerequisite for requiring references to resolve. * a marked tuple struct, whose fields no adapter has ever crossed — unchanged acceptance, now named for what it always meant. So `Struct::fields` drops its `Option`. `None` was the opaque case; an empty list now means the source wrote a struct with no fields, which is a different thing. `MaybeUninit<T>` joins the grammar as `TypeKind::Uninit`. It is a boundary concept — an out-parameter whose slot the caller supplies and the callee fills — and cbindgen already models it as exactly that, so this moves a classification out of the adapter and into the frontend, per #211. It is also the one foreign generic that no alias could name, a generic alias being a generic binder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Flat resolves its references and answers by name Two changes that belong together, because the first is what makes the second decidable. **The model is addressed by name, not iterated.** `FlatBuilder` collects and `build` hands over a `Flat` — `function(name)`, `declared_type(name)`, `constant(name)`, `element(name)`, plus iterators over each kind. Names are unique across the whole model, so a name is a complete address, and that is what every later stage wants: an adapter asks what a declared name *is* rather than scanning a list. L1 carried this as a checklist bullet; it is really a property of the model. Two types rather than one, because a half-built model should not be the same type as a resolved one — `Source::builder()` sets the precedent. **References resolve at parse time.** A third pass walks every `TypeRef` — through `Option`, `Vec`, `&`, `Result`, arrays, callback arguments and generic arguments alike — and an item naming a type the flat API does not declare becomes `Element::Unsupported` with `ItemError::UnresolvedType`. Deferred, not fatal, like every other refusal: an item no binding declares stays harmless. This is what a marked type alias bought. A dangling name previously surfaced far downstream as an unresolved *converter*, from whichever adapter happened to look first — the "one fact, several authorities" #211 exists to end. Note the two remain distinct: resolution here says a name denotes something, while an adapter's resolver still decides whether it supplied a converter for it. A path-qualified name gets its own diagnosis, since `#[prebindgen] pub type foreign::Option = ..` is not a spelling that exists — marked items live in one flat namespace of bare names. Also: `Item::Type` no longer reaches the registry's passthrough. An opaque declaration states something about the API's surface and is not code to copy into the binding; its target is routinely crate-private, so re-emitting it would not compile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Close the example flat APIs, and assert they stay closed Every type a marked signature named had to become a declaration for resolution to mean anything. Two idioms, chosen by what the type actually is rather than by its Rust shape: **A handle gets a marked alias.** `Storage`, the three callback handlers, `Token`, `TokenGc`, `Summary`, `Archive`, `Report`, `EscapeProbe`, `StorageError`, and example-flat's `Calculator` move into a private `handles` module, with `#[prebindgen] pub type X = handles::X;` at the top level. The alias is transparent, so every signature still says `Storage`. `Error` in both crates was already an alias and only needed the attribute — which is exactly the shape zenoh-flat's 26 zenoh re-exports will take. **A public newtype stays a marked struct.** `Millis`, `Celsius`, `Percent` and `Label` are not handles: they cross by `convert!`, and covertest-helpers both constructs them and reads `.0`. Hiding them behind an alias broke that downstream, which is the useful signal — a type alias names the type, not the tuple-struct constructor, and the constructor lives in the value namespace where the struct is defined. In-crate construction of the relocated handlers is qualified `handles::PayloadHandler(..)` for the same reason. Marking these as structs rather than aliases matters for a second reason: a marked struct enters `registry.structs`, and `write.rs` emits `on_struct` for any declared type there — so marking the *handles* as structs would have changed generated output. The alias route is invisible to the registry, which is why the goldens hold. **And the closure is asserted, not assumed.** covertest-kotlin's build script now runs `Flat` over both sources and fails if anything is unsupported. It is the right place: only there do the helper crate's references to perftest-flat's types resolve, since it cannot mark them itself. Verified by deliberately unmarking `Storage` — the build fails naming all twelve referencing functions and the fix. Generation is byte-identical (`examples/regen-check.sh`) and the JVM covertest passes all 47 sections. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Record L0.5 in the stage map The model is now indexed and resolved, which takes two bullets off L1 — elements indexed by name, and the entry point that shares one parser — and adds a prerequisite L0 did not have: the flat API has to be closed for resolution to mean anything. Also records what is left open: zenoh-flat and its two consumers are separate repos whose 28 unmarked types need the same treatment, and `Cow<'_, [u8]>` has no alias spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Take a slice, not a Vec reference, in the resolution pass `clippy::ptr_arg` under CI's no-default-features run: the pass only mutates elements in place, so a slice is the honest signature. My local checks used --all-features only; CI runs three clippy configurations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * An out-parameter is a mode of borrowing, not a type `TypeKind::Uninit` wrapped a type, but uninitialized-ness is a property of the **borrow**: my own doc said `MaybeUninit` is "only meaningful behind a `&mut`", which is the argument against modelling it as a type at all. So `Ref` carries the mode, and the `MaybeUninit` is absorbed into it: Ref { mode: RefMode, inner: Box<TypeRef> } enum RefMode { Shared, Exclusive, Out } `&T`, `&mut T`, `&mut MaybeUninit<T>` — one axis, three values, and `inner` is always the borrowed *value's* type. One variant fewer than the `mutable` flag plus a wrapper, and the combinations that mean nothing at a boundary can no longer be written down: uninitialized storage owned, returned or in a field promises nothing a destination language can use, and `&MaybeUninit<T>` promises a readable `T` that may not be one. Both are refused, each naming why. `Out` rather than `Uninit` because it names the boundary role every destination language has — C's `T *out` — which is the fact an adapter acts on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Address review: transitive closure, generic aliases, goldens, real index Four findings, all valid; two were mine in this PR. **Refusal was not transitive.** `resolve_references` snapshotted the initial declarations and validated everything against that fixed set, so refusing a type stranded its dependents: pub struct Broken { pub field: Missing } // refused pub fn use_broken(value: Broken) {} // survived anyway `Flat::resolve` then returned `None` for `use_broken`'s parameter, contradicting the one invariant the model promises. It now runs to a fixed point: each round drops the declarations it refused, and stops when a round refuses nothing. Chains of any length collapse, in either declaration order, because the declared set only ever shrinks — which is also why it terminates. Regressions cover the direct case both ways round, a four-link chain both ways round, a sound chain that must be left alone, and the invariant itself: every `Named` reachable from a surviving element resolves. **A generic type alias bypassed the binder refusal.** The `Item::Type` arm built an `Opaque` without calling `reject_generic_params`, so `pub type Handle<T> = hidden::Handle<T>;` was accepted as one declaration that `Handle<u8>` then resolved against — losing exactly the scoped-parameter distinction every other item kind refuses, and contradicting this PR's own argument that `MaybeUninit` needed grammar support *because* a generic alias is a binder. Type and const parameters are now refused; a lifetime binder stays accepted, as on every other kind. **The aarch64 goldens carried unrelated all-features output.** `git add -A examples` in the migration commit swept in pre-existing working-tree drift — `unstable_field`, `calculator_reset`, a non-empty feature guard — which is exactly the state 95fd753 had reverted, because committed aarch64 goldens represent a plain build. Restored from the base, and verified: a plain `cargo build --release -p example-cbindgen` on arm64 reproduces the base files byte-for-byte. CI is x86_64 and cannot see this pair, so it needed catching by hand. My "byte-identical" claim was wrong for that reason, not for the model changes. **`Flat` was not actually indexed.** It stored only a `Vec` and `element()` did `iter().find`, so every typed accessor and `resolve()` scanned — quadratic once later stages resolve in a loop, and not the "indexed by name" criterion L0.5 claims. Now a `HashMap<String, usize>` beside the elements: positions, so there is one copy of each element and source order stays available for iteration. Built after resolution, since refusing an item changes its kind but never its name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Brings in #221, #231 and #233. Three things worth knowing about the resolution. **Most conflicts were one change arriving twice.** #227's branch had been rebased onto #221 before it was squash-merged here, so the squash absorbed #221's diff — `git diff 225954a 0901651` over jnigen is empty. Merging main then replayed #221 as its own commit, conflicting with its own absorbed copy in nine files. For each, `git diff 225954a 989010e` showed this branch had added nothing beyond that copy, so main's side was taken wholesale: main is #221 plus #231 and #233 on top. **`Ledger` needed declaring.** #231 added it as an unmarked handle struct, which is exactly the drift the closure guard exists to catch — and it caught it on its first merge, naming all four referencing functions. It now takes the same treatment as every other handle here: the definition sits in the private `handles` module behind `#[prebindgen] pub type Ledger = handles::Ledger;`. `Report` keeps main's new `#[derive(Clone)]`, moved onto the definition, since the top-level name is only its alias. **Generated artifacts are regenerated**, because the merged tree's committed copies were a mix of both sides. The Kotlin and Rust output is the union of the two feature sets, and the boundary ledger is reseeded — `api/core/unfold.rs` 16 → 18, as #231/#233 added two classification sites there. Also fixed two strings the `language` → `flat` rename left stale: the ledger's own drift message and header still named `core::language`. Note for #231's author: `covertest-kotlin/build.rs` says "`Report` is not `Clone`, so cloning it here would not compile", while `ext.rs` now derives `Clone` on it and explains why it must. One of the two comments is stale on main; taken verbatim here rather than edited, since a merge should not quietly rewrite either side's prose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in #234, `Registry::builder().source(dir)`. Two conflicts, both from the same cause: #234 removed the `Source` bindings from `covertest-kotlin`'s `main()`, and this branch had added the flat-API closure guard right there, built from those same bindings. `FlatBuilder` gains `source_named` — the follow-up #234 flagged, now needed rather than merely tidy. The guard reads its two directories directly, so the two builders stay shape-identical and no `Source` survives in that build script: Flat::builder() .source(perftest_flat::PREBINDGEN_OUT_DIR) .source_named(cov_helpers::PREBINDGEN_OUT_DIR, "cov_helpers") .build() That does read each directory twice, once for the guard and once for the registry. It is a build script and the cost is a second JSONL parse, and it goes away at L1 when the registry consumes `Flat` instead of re-indexing the stream — which is what the shared shape was for. `lib.rs`'s conflict was two export lists growing in parallel; both sides' names belong. The feature-coverage table at the top of covertest's build script now names `source_named` instead of the `Source::builder().crate_name()` it replaced. 519 tests, 18 doctests, clippy clean on all three configurations, generation byte-identical, and the JVM covertest still passes all 48 sections — including the renamed second source, which is the path this merge touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This body mirrors
docs/language-integration.mdon this branch, which is the one place stage state is edited. Change the doc, then re-sync this body — never the other way round.Parse once, consume elements everywhere
Umbrella for making every component of prebindgen consume
core::language'sElements instead of parsing captured Rust itself. Draft, and long-lived — it carries the map, not the code; every stage lands as its own PR againstmain.#211 remains the authority on the invariants and the frontend/adapter boundary. This document does not restate them; it records the design this program follows, what has landed, and the order.
The design
An
Elementis two things at once, and the pairing is the whole point:TypeKind,StructFields, the variant list —that says what the source means, in terms every destination language shares;
field, variant and type.
Why the syntax rides along
The predecessor design (#215)
built a syn-free semantic model and kept hitting one wall: the generated Rust
glue is itself a destination artifact, and it is the only consumer that needs
syntax fidelity. Each time it did, the answer was to model the syntax —
DiscriminantSource::Explicit(syn::Expr),syn::Member,syn::Lifetime,to_syn(), and finally aVariantShapewhose only job was to make generatedRust spell
E::B()instead ofE::B. A model that carries no syntax has tobecome lossless to serve that consumer, which is how a language-neutral IR
turns back into a second
syn.Carrying the original slice costs nothing and removes the pressure, so the
classification stays small and genuinely neutral:
B()vsBVariant::syntax(viaVariant::spell)= 0x07vs= 7Variant::syntax.discriminantVariant::discriminantNAME(7),jintdecodeFoo<'a, T>Type::syntaxFoowith one type argument"TypeKind::Named[u8; TAG_LEN]— spelling / number / const identityType::syntax/ArrayExtent::value/ExtentSource::ConstThe rule
This is mechanically measured, and needed no new mechanism:
core::language::boundary(ported from#224) counts variant mentions
of watched syn enums per file, so
quote!(#slice)is invisible to it whilematches!(ty, syn::Type::Reference(_))is counted. The committed ledger is thescoreboard for this whole program.
Size of the problem
Seeded by L0 at 202 classification sites outside
core::language, plus113 reads of the registry's
syn-keyed item maps:api/core(types_util40,unfold15,registry13,expand4)api/lang/cbindgenapi/lang/jnigenNot every site must go: some inspect types the adapter itself synthesized —
wire types, converter signatures — which is legitimately the adapter's business.
Separating the two populations is not a document to write up front; it is each
entry's fate as it comes off the ledger, with a stated reason in the PR that
moves it.
Stages
Language+Element+ the ledgerRegistryconsumes elementsapi/corestops classifying source syntaxCbindgenconsumes elementsJniGenconsumes elements (the long pole — 105 sites)synL0 — the parser — done (#227)
Language::parseover any(syn::Item, SourceLocation)stream — the seamRegistry::from_itemsoccupies, so multi-source composition is unchangedElement=Function | Struct | Enum | Const | Unsupported | Passthrough,every element and component carrying its syntax slice
Type { kind, syntax }; lowering total over the accepted grammarArrayExtent, ported from One frontend for array lengths, and a typed source model that carries extents to C (#210, first step of #211) #212checked_addcases a reconstruction loses (empty delimiters,
0x07, lifetimes, docs)and the component
Acceptance is preserved, not expanded. An item the language cannot express
becomes
Element::Unsupportedcarrying its diagnosis, because the pipeline hasalways scanned a signature only once an adapter declared it, and a source crate
may mark items no binding uses. Only a duplicate name — which no declaration can
disambiguate — fails the parse. Tuple-struct fields stay unmodelled for the same
reason.
L1 —
Registryconsumes elementsThe seam that makes the direction real. Adapters must not need touching.
Registry::from_elements(Vec<Element>);from_itemsbecomesLanguage::parse+from_elements, so both entry points share one parserfunctions/structs/enums/consts/passthroughmaps arerebuilt from each element's retained
syntax— a projection, not a secondsource of truth
scan_fn_signature's receiver / parameter-pattern /impl Traitguardsare deleted: the diagnosis is already on
Element::Unsupported, anddeclaring such an item is what raises it
ScanError's per-item variants map ontoItemError, so one authorityproduces the message
(
examples/regen-check.sh)L2 —
api/corestops classifying source syntaxtypes_util— 40 sites, the largest single file.normalize_type,immediate_pattern_children,match_pattern, theis_*predicatesregistry::immediate_subtype_positions— near-duplicate ofimmediate_pattern_children, and the two already diverge onType::Pathunfold(15) andexpand(4) read element typesTypeKeyderivable from aTypeso a lookup stops routing through aspelling
the PR as adapter-synthesized
L3 —
Cbindgenconsumes elementsbuilder(8),trait_impl(6),emit(5),mod(5),convert(1)Variant::spell, not fromre-deriving delimiters
Variant::syntax, and the number comesfrom
Variant::discriminantL4 —
JniGenconsumes elementsThe long pole. Split by area, each PR independently green.
emit/names(17),jni/builder(13),jni/trait_impl(11),emit/wrapper(11),emit/flat_input(10),render(8),selector(7),and the rest
classify.rs— a whole classifier with zero watched sites, so theledger cannot see it: it must be migrated on its own merit
prim_array_ofreadsArrayExtentinstead of re-matchingType::ArrayL5 — close the seam
The public contract stops being
syn, which is what stops the population fromgrowing back.
Registry's public item maps stop being the adapter-facing contract —relates to #92
Prebindgen::post_process_item(&mut syn::Item)— the hook that letqualification live in an adapter in the first place
ConverterImpl::function/TypeEntry::functionassyn::ItemFn;prerequisites/local_functionsreturning raw itemsNiches { value: syn::Expr, matches: syn::Expr }— a semantic fact carriedas raw expression syntax
WATCHEDbeyondType/Expr—Item,Fields,FnArg,ReturnType,GenericArgument,Pat— one enum at a time, eachaddition a regenerated ledger whose diff is the decision
classification, ident-name classification, helper delegation)
Completion criteria
#211's, restated for this design:
Language::parse.CbindgenandJniGentake every source fact from an element.has reached the irreducible set, and every remaining entry is documented as
inspecting adapter-synthesized types.
diagnostics naming item and component.
syntaxslice, never byreconstructing one from a classification.
Relationship to #215
#215 is superseded. Its four merged PRs are not lost: L0 ports the
array-length subgrammar (#212), the type grammar and its acceptance tests, the
enum tag/discriminant numbering (#226) and the boundary ledger (#224). What is
dropped is the syn-free model itself —
SourceType::to_syn,DiscriminantSource,VariantShape,NamedArg::Lifetime— because carrying thesource's own slice does that job without a modelling cost.
The
source-frontendbranch stays in place as the reference. Nothing depends onit, and it is not a base for anything here: every stage of this program targets
main.Review protocol
Each stage PR states its own exit:
examples/regen-check.sh. A diff is a bug.that cause is a bug.