Skip to content

Drive client-go as the oracle, and state the first gap against it - #4

Merged
eduralph merged 10 commits into
mainfrom
parity-go-runner
Jul 13, 2026
Merged

Drive client-go as the oracle, and state the first gap against it#4
eduralph merged 10 commits into
mainfrom
parity-go-runner

Conversation

@eduralph

@eduralph eduralph commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What & why

The harness could only ever say "client-rust fails." It could never say "and client-go succeeds" — which is the claim that makes a gap a gap, and the claim every upstream issue needs. Even d6, our strongest finding, only demonstrates client-rust failing; it never demonstrates that anyone does better.

The Go half was fully anticipated and entirely unbuilt. pins.toml already pinned client-go as "The ORACLE"; check-pins.sh already branched on a go/go.mod that "arrives with the Go runner, Phase 2"; provenance.sh already promised a ledger-check that did not exist; gate-verdict.sh already called itself "a deliberately small stand-in for the parity ledger". This builds that.

scenarios/*.json          declarative; no conditionals, no loops
        │
        ▼
crates/parity-runner  ──NDJSON──▶  crates/rust-driver  → tikv-client   (the SUBJECT)
  owns the total order──NDJSON──▶  go/driver           → client-go     (the ORACLE)
        │
        ▼
results/traces/*.json ──▶ projection ──▶ diff ──▶ ledger.toml ──▶ XDIVERGE / XCONVERGE

Both clients wrap the same protobuf, and that is what makes a cross-language diff defensible rather than a fudge: tikv-client's Error::KeyError and client-go's ErrWriteConflict are thin wrappers over the same kvrpcpb messages, so the vocabulary they are compared in is not invented — it is the server's.

Nothing is normalized at capture. An observation keeps three layers: the canonical class (always compared), the kvproto (presence compared), and the client's own error taxonomy in native (never compared by default, always kept as evidence). Canonicalize the question, never the evidence — a projection can be widened later; evidence discarded at capture is gone.

That layering is load-bearing, not decoration. G-0001's root cause is a native-taxonomy bug — check_txn_status matches Error::ExtractedErrors but only ever receives Error::MultipleKeyErrors — and both map to the same canonical class. A class-only diff would call the two clients identical and the bug would be invisible.

The orphan factory

client-go exports CommitterProbe from a non-test file, so the Go driver can prewrite a secondary whose primary is never written — finding 1's orphan — in three commands with no races. Both runs share that one setup and change only the READER, so any divergence is attributable to the reader alone.

gate::d6 cannot claim this: it must build the orphan through the public API with a region split and a racing txn, so "I failed to BUILD the orphan" and "the client failed to RESOLVE it" are the same red. That conflation is exactly the WRONG-FAILURE hazard gate-verdict.sh exists to catch, and it is why d6 needs a PRECONDITION FAILED panic.

Result at the pinned revisions

step · field oracle (client-go) subject (client-rust)
reader-final-read · class not_found — read succeeds txn_not_found — read errors
reader-final-read · native.type (none — it succeeded) MultipleKeyErrors ← the bug itself
reader-locks-after · locks.count 0 — orphan resolved 1 — orphan remains forever

Kind of change

  • Gate/harness code
  • Cluster / tooling (Makefile, CI, scripts/)
  • Docs (README.md)

Checklist

  • make check is green (now also go vet / go mod verify / gofmt)
  • Cluster-backed changes verified against a real TiKV/PD cluster (make ledger, PARITY_STRICT=1)
  • Each new test names the finding it verifies
  • The harness carries no workaround for a client-rust bug. prewrite_only on the Rust driver answers unsupported rather than emulating the missing probe surface — and that gap (client-rust exposes no importable test probes; its mocks are #[cfg(test)]) is itself a finding worth filing
  • README.md updated

Evidence

All four verdict rules were exercised for real, not merely written. A comparator that only ever reports divergence is worth nothing:

rule how it was provoked outcome
XDIVERGE at the pin, gap open pass ✅
XCONVERGE applied findings/fix-check-txn-status-wrapper.patch (#544) — the divergence vanished entirely hard fail, as it must ✅
WRONG DIVERGENCE fired unprompted when a driver change moved a field the ledger still declared hard fail ✅
REFUSE dirty client-rust; deleted traces; unverified oracle; stale evidence from an older harness refused to settle ✅

That XCONVERGE run is the important one: it proves the gap is real, that the fix closes it, and that the harness is not stuck-red.

Two bugs found in the existing harness

  • provenance.sh stamped client-go by re-reading pins.toml — recording what the harness believed, not what it built. The exact hole its own header complains about for Rust, and not theoretical: Go's MVS resolved kvproto above go.mod during this very build. It now asks go list -m.
  • Nothing asserted the ../client-go sibling was at the pinned rev. It was, by luck.

Declared deviations

Both land in hello.config and therefore in every trace, rather than in a comment nobody reads six months from now:

  • The Go driver disables PD's router client. The pinned client-go resolves regions via a streaming QueryRegion RPC that the pinned PD (v8.5.5 — client-rust's own CI pin) does not implement, so left on, every region lookup hangs forever. Disabling it falls back to the classic GetRegion path, which is the one client-rust uses; for a parity harness that is more faithful, not less. Bumping the cluster instead would re-state every wyrd-gate claim — a reviewed decision, not something to do in passing. Worth an explicit ack.
  • The Rust driver begins every txn with drop_check(Warn): a dropped uncommitted Transaction panics by default, which would make abandon — the crash we are modelling — unimplementable.

Review

Ten passes of codex review; 27 findings, all fixed. Most clustered on one theme, and it is the theme this repo exists to defend against: an absent check reading as a passed one (a typo'd projection path that compared a sentinel to itself; deleted traces that skipped the oracle's identity proof; a scenario comparing a run with itself). Two were serious: the harness was pinning the subject but not the instrument (a modified projection or ledger could manufacture the declared divergence), and provenance was re-derived at adjudication time, so stale evidence from a dirty tree could be certified by a later clean checkout. Both are closed and both attacks are verified as refused.

Not done, deliberately

The README's hand-written verdict is still hand-written and still stale. Generating it from the ledger is the next step, along with the capability inventory (Checksum, LockNoWait, fair locking) now that unsupported is a first-class observation.

🤖 Generated with Claude Code

Base automatically changed from workspace-conversion to main July 13, 2026 21:23
eduralph and others added 10 commits July 13, 2026 23:25
@eduralph @claude
Drive client-go as the oracle, and state the first gap against it
9fe0792 
The harness could only ever say "client-rust fails." It could never say "and
client-go succeeds" -- which is the claim that makes a gap a gap, and the claim
every upstream issue needs. Even d6, the strongest finding, only demonstrates
client-rust failing; it never demonstrates that anyone does better.

The Go half was fully anticipated and entirely unbuilt: pins.toml already pinned
client-go as "The ORACLE", check-pins.sh already branched on a go/go.mod that
"arrives with the Go runner, Phase 2", provenance.sh already promised a
ledger-check that did not exist, and gate-verdict.sh already called itself "a
deliberately small stand-in for the parity ledger". This builds that.

  crates/parity-proto   the protocol + the layered observation + the projection.
                        Depends on NEITHER client.
  crates/rust-driver    drives tikv-client   (the SUBJECT)
  go/driver             drives client-go     (the ORACLE), at the pinned module
  crates/parity-runner  the comparator. Links neither client, enforced in its
                        Cargo.toml: the thing deciding a verdict must not be able
                        to reach for the crate it is adjudicating.

BOTH CLIENTS WRAP THE SAME PROTOBUF, and that is what makes a cross-language diff
defensible rather than a fudge. tikv-client's Error::KeyError and client-go's
ErrWriteConflict are thin wrappers over the same kvrpcpb messages, so the
vocabulary they are compared in is not invented -- it is the server's.

Nothing is normalized at capture. An observation keeps three layers: the canonical
class (always compared), the kvproto (presence compared), and the client's OWN
error taxonomy in `native` (never compared by default, always kept as evidence).
The projection decides what a claim looks at. Canonicalize the QUESTION, never the
EVIDENCE: a projection can be widened later; evidence discarded at capture is gone.

That layering is load-bearing, not decoration. G-0001's root cause IS a
native-taxonomy bug -- check_txn_status matches Error::ExtractedErrors but only
ever receives Error::MultipleKeyErrors -- and both map to the same canonical class.
A class-only diff would call the two clients identical and the bug would be
invisible.

THE ORPHAN FACTORY. client-go exports CommitterProbe from a non-test file, so the
Go driver can prewrite a secondary whose primary is never written -- finding 1's
orphan -- in three commands with no races. Both runs of the scenario share that one
setup and change only the READER, so any divergence is attributable to the reader
alone. d6 cannot claim this: it must build the orphan through the public API with a
region split and a racing txn, so "I failed to BUILD the orphan" and "the client
failed to RESOLVE it" are the same red. That conflation is exactly the WRONG-FAILURE
hazard gate-verdict.sh exists to catch, and it is why d6 needs a PRECONDITION panic.

Result at the pinned revisions -- client-go resolves the orphan; client-rust cannot:

    reader-final-read   class       oracle=not_found   subject=txn_not_found
    reader-final-read   native.type oracle=<absent>    subject=MultipleKeyErrors
    reader-locks-after  locks.count oracle=0           subject=1

ledger.toml declares each gap FIELD BY FIELD, and ledger-check.sh generalizes
gate-verdict.sh from "a substring in stdout" to "a field in a trace": XDIVERGE /
XCONVERGE / WRONG DIVERGENCE / NEW GAP. All four rules were exercised for real, not
merely written:

  XDIVERGE          at the pin, gap open.
  XCONVERGE         applying findings/fix-check-txn-status-wrapper.patch (#544)
                    makes the divergence vanish ENTIRELY -- so the comparator is not
                    stuck-red -- and ledger-check then hard-fails, as it must.
  WRONG DIVERGENCE  fired for real when a driver change moved a field the ledger
                    still declared. It refused to accept a divergence it had not
                    predicted rather than shrug and call it confirmed.
  REFUSE            a dirty client-rust cannot settle a claim.

Also fixed, both found while building this:

  * provenance.sh stamped client_go by re-reading pins.toml -- recording what the
    harness BELIEVED, not what it BUILT. That is the exact hole its own header
    complains about for Rust, and Go's MVS can resolve above go.mod (it did: tidy
    raised kvproto). It now asks `go list -m` what was actually selected, and
    refuses an off-pin or `replace`d oracle. The Go driver independently reports
    `replaced` from its own debug.ReadBuildInfo(), so pins.toml's "an oracle you
    can accidentally edit is not an oracle" is now a machine check made from inside
    the binary that ran.
  * nothing asserted the ../client-go sibling was at the pinned rev. It was, by
    luck. check-pins.sh now checks, and also rejects a `replace`, a tracked
    go.work, a missing go.sum, and a go directive that disagrees with the pin.

DECLARED DEVIATIONS, in hello.config and therefore in every trace rather than in a
comment nobody reads six months from now:

  * the Go driver disables PD's router client. The pinned client-go resolves regions
    via a streaming QueryRegion RPC that the pinned PD (v8.5.5 -- client-rust's own
    CI pin) does not implement, so left on, every region lookup hangs. Disabling it
    falls back to the classic GetRegion RPC, which is the path client-rust uses; for
    a parity harness that is more faithful, not less. Bumping the cluster instead
    would re-state every wyrd-gate claim, which is a reviewed change, not something
    to do in passing.
  * the Rust driver begins every txn with drop_check(Warn): a dropped uncommitted
    Transaction PANICS by default, which would make `abandon` -- the crash we are
    modelling -- unimplementable.

The two verdict mechanisms stay separate, and that is correct for now. `verdict`
adjudicates the wyrd M4 contract, which has no Go counterpart (client-go has no
MetadataStore, so diffing it against the oracle would be meaningless). `ledger`
adjudicates parity. They answer different questions.

Not done, and deliberately: the README's hand-written verdict is still hand-written
and still stale. Generating it from the ledger is the next step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Verify the cluster, and close three ways the comparator could lie
e8af54d 
Codex review of the parity mechanism. All four findings were real; two of them
could produce a WRONG VERDICT rather than merely being untidy.

CERTIFYING AGAINST AN UNVERIFIED SERVER (P1). provenance.sh stamped the cluster by
copying pins.toml's image digests -- recording what the harness BELIEVED, not what
it EXERCISED. This is precisely the hole the previous commit closed for client-go,
reintroduced one field lower down, and it has teeth: $PD_ADDRS points anywhere and
`make ledger` does not depend on `cluster-up`, so a strict run against a
hand-started TiKV of some other version would have been stamped with the pinned
digests and published as a pinned run.

That is not a technicality. Every behavioural claim in the ledger is a claim about
a CLIENT talking to a SPECIFIC TiKV -- lock resolution, prewrite residue and
conflict shapes are server behaviour as much as client behaviour. Certifying one
against an unknown server certifies nothing.

provenance.sh now ASKS the running cluster (PD's HTTP API, the same ground truth
crates/harness already uses for region layout) and records PD's version and every
Up store's version. Off-pin or unidentifiable -> refuse under PARITY_STRICT, exactly
as an off-pin client-rust does. ledger-check refuses such a result too. `make ledger`
now depends on the cluster being reachable rather than assuming it. Verified: strict
against an absent PD exits 1; strict against the pinned cluster exits 0 and stamps
observed_pd_version=v8.5.5, observed_tikv_versions=8.5.5.

A TYPO THAT COMPARED NOTHING WHILE LOOKING LIKE IT COMPARED SOMETHING (P2). An
unknown `also_compare` path projected the same sentinel string on BOTH sides, so
diff() saw equality, reported nothing, and the comparison the claim explicitly asked
for never happened -- while the scenario still passed its ledger check on its other
declared divergences. A claim that examined nothing looked exactly like a claim that
held. That is the vacuous pass this entire harness exists to make impossible, and it
was sitting inside the projection.

Opt-in paths are now a closed set, validated at scenario LOAD (before a single
command runs), and the sentinel branch panics rather than silently comparing equal.

A FALSE DIVERGENCE MANUFACTURED BY THE HARNESS (P2). proto_type() walked the JSON
recursively, so it descended into the PAYLOAD: Rust's TxnNotFound (which carries
start_ts) rendered as `key_error.txn_not_found.start_ts` while Go's rendered as
`key_error.txn_not_found` -- because client-go leaves that KeyError untyped and has
no proto to lift fields from. Two clients reporting THE SAME MESSAGE would diverge
purely because one attached a field. It now extracts exactly container.message, and
payload values remain reachable through the opt-in `proto` path.

SCAN_LOCKS ASYMMETRY (P2). Rust's scan_locks takes a per-region BATCH SIZE and does
not page within a region; client-go's probe pages to exhaustion. Passed through
naively, the two drivers answered different questions and would have invented a
lock-count divergence out of harness semantics. The protocol field is renamed
batch_size (what it actually is), its contract is stated -- a driver MUST return
every lock in [start, end) -- and the Rust driver now pages, exactly as client-go's
own probe does. Truncating Go's result to match Rust's cap would have been the wrong
fix: it HIDES locks, and seeing the residue is the entire point of the observation.

The ledger's proto.type row is re-stated for the corrected extraction. Had I
forgotten, WRONG DIVERGENCE would have caught it -- it did exactly that during the
previous commit's development, which is the rule earning its keep.

make ledger is green end-to-end at the pinned revisions: XDIVERGE on all 6 declared
fields, cluster verified as PD v8.5.5 / TiKV 8.5.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Do not lose locks when paging, and namespace binary keys
0f5607a 
Second codex pass. Both findings were real; the first was a bug I introduced in the
previous commit's own fix, which is a fair reminder that a subtle correctness
argument deserves a test rather than a comment.

LOSING LOCKS WHILE PAGING. My scan_locks pagination advanced the cursor past the
GREATEST key in each page. But scan_locks merges per-region results, each capped at
batch_size (retry_multi_region -> Collect), so a LOW region can be truncated while a
HIGH one is complete -- and the greatest key in the merged page then belongs to the
high region. Advancing past it steps over the low region's unreturned tail
permanently and silently.

For this observation that is the worst failure available. Its entire job is "what
lock residue was left behind", so dropping locks would turn client-rust's
un-resolved orphan into an apparent 0-lock success and quietly CLOSE A REAL GAP --
the harness reporting parity precisely where the bug is.

The safe cut is the batch_size-th SMALLEST key, and it is provably complete: let `u`
be any lock in range that was not returned. Then u's region was truncated, so it
returned exactly batch_size locks, all with keys below `u` (a region yields its
lowest keys first), and all of them are in this page. Hence at least batch_size
returned keys sort below `u`, so u > page[batch_size - 1]. Everything at or below
that key is complete; everything above is dropped and re-read next pass.

The cut is now a pure function with the proof in its doc comment and, more to the
point, tests: a simulated multi-region cluster over region-layout x batch-size,
including the exact shape that defeated the old code (low region truncated, high
region's key is the max).

BINARY KEYS WERE NOT NAMESPACED. KeyArg::substitute applied the {P} token to text
and passed b64 through UNCHANGED -- so a scenario using a binary key would send
byte-identical keys to BOTH runs, on a SHARED cluster, where the oracle run
deliberately leaves residue. The subject would trip over the oracle's orphaned lock
and the harness would manufacture a divergence, or hide one, entirely out of itself.
No scenario does this yet, but binary keys are not exotic: a 0xFF-boundary key
cannot be written as text, and gate::d4 already tests exactly that boundary.

Keys and values now substitute differently, and the runner's match is the only place
that knows which field is which, so it is the only place that can get it right:

  as_key    text: {P} -> prefix.  binary: prefix BYTES prepended.
  as_value  text: {P} -> prefix.  binary: UNTOUCHED -- values are opaque bytes that
            must round-trip byte-identically (the store's CAS is value-equality over
            a whole record; prefixing one would corrupt the property under test).

make ledger remains green end-to-end: XDIVERGE on all 6 declared fields, cluster
verified as PD v8.5.5 / TiKV 8.5.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Bind provenance to the evidence, and close the last two isolation esc…
90d9a50 
…apes

Third codex pass. All three findings were real, and two of them were escapes I
opened myself in the previous commit -- which is what happens when you fix an
isolation hole and forget that the fix has to survive the projection too.

PROVENANCE RE-DERIVED AT ADJUDICATION TIME DESCRIBES THE WRONG MOMENT.
ledger-check.sh re-stamped provenance by running provenance.sh, then adjudicated
whatever happened to be sitting in results/. So traces produced against a dirty or
off-pin client-rust could be left there, the checkout restored to the pin, and a
strict ledger-check would mint fresh, admissible-looking provenance and settle the
stale evidence with it. The trace never has to lie -- the adjudicator does it on
its behalf.

Evidence must carry the conditions it was gathered under, or it is not evidence.
The runner now reads results/provenance.json once, before anything runs, and stamps
it into every trace and every divergence report. ledger-check reads the provenance
IN THE ARTIFACT and refuses on that -- plus it re-checks the recorded revisions
against pins.toml, so a pin that MOVED after a result was gathered also invalidates
it.

Verified by running the attack: produce results with the #544 fix applied (dirty,
advisory), restore client-rust to the pin, then run a strict ledger-check. It
refuses -- "produced with strict:false" -- because the artifact remembers, even
though the tree no longer does.

A TEXT KEY WITHOUT {P} WAS NOT NAMESPACED. `replace` is a no-op when the token is
absent, so both runs used the same cluster key. That made the isolation guarantee
depend on a scenario author remembering five characters in a JSON file -- and
forgetting them would not look like a mistake, it would look like a DIVERGENCE:
the oracle's deliberately-abandoned lock poisoning the subject's read.

The invariant now holds BY CONSTRUCTION rather than by inspection. `{P}` says WHERE
the prefix goes, never WHETHER: text with the token substitutes it, text without it
gets the prefix prepended, binary gets the prefix bytes prepended. No key form can
escape its run prefix, and a test asserts exactly that over every form.

A NAMESPACED BINARY KEY LEAKED ITS PREFIX INTO THE DIFF. Prepending prefix BYTES
(the previous commit's own fix) makes a binary key invalid utf-8, and the redactor
only stripped the prefix on the utf-8 path -- so it fell through to base64-encoding
the key WITH the run prefix still in it. Two runs get different prefixes by
construction, so the oracle's and the subject's otherwise-identical locks would
have encoded differently and diverged for a reason that was purely an artifact of
the harness. Redaction now happens on BYTES, before any utf-8 decision, and the
regression test does the full round trip: namespace one binary key into two runs,
and the projection must call them equal.

make ledger green end-to-end: XDIVERGE on all 6 declared fields, cluster verified
as PD v8.5.5 / TiKV 8.5.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
An absent check must never read as a passed one
6144b8a 
Fourth codex pass. Three findings, and all three are the SAME species: a check that
does nothing, quietly, and is therefore indistinguishable from a check that passed.
That is the exact failure this harness exists to eliminate, so they matter more than
their severity label suggests -- and two of them were mine.

AN UNVERIFIED ORACLE WAS RECORDED AS AN ON-PIN ORACLE. When `go list -m` could not
run (no toolchain, no module), provenance.sh copied the PINNED version into the
resolved field and left matches_pin=true. So a strict run with no Go at all recorded
a perfectly on-pin-looking oracle it had never once looked at, and ledger-check --
which never consulted `verified` -- admitted it. The file that exists to stop the
harness recording what it BELIEVES rather than what it BUILT was doing precisely
that. Unverified is now not-on-pin, and strict refuses it. check-pins.sh already had
the words for this: "an unverifiable invariant is not a satisfied one."

DELETING THE TRACES SKIPPED THE ORACLE'S IDENTITY PROOF -- AND THE SKIP LOOKED LIKE
A PASS. ledger-check's go-driver check globbed results/traces/*.json and looped. With
no traces the loop body never ran and the script exited happily, so a divergence
artifact could be adjudicated with NO proof of which client-go binary produced it --
the ReadBuildInfo() attestation that makes "an oracle you can accidentally edit is
not an oracle" a machine check rather than a comment. It now demands, per ledger
entry, the traces for exactly the runs its verdict is computed from, and a `go`
binding in each. Verified: delete results/traces, keep the divergence artifact, and
a strict ledger-check refuses.

A SCENARIO COULD COMPARE A RUN WITH ITSELF. `compare: ["oracle","oracle"]` passed
validation, and both lookups resolve to the same trace -- an empty diff by
construction. For an `expect = "agrees"` entry that is a free green: the claim holds
without the subject having been exercised at all. A comparison that cannot fail is
not a comparison. Rejected at load, along with duplicate run names (which would make
`compare` ambiguous about which trace to adjudicate).

make ledger green end-to-end: XDIVERGE on all 6 declared fields.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Pin the instrument, not just the subject
846af6c 
Fifth codex pass.

THE HARNESS IS THE INSTRUMENT, AND NOTHING WAS PINNING IT. Strict mode checked that
client-rust, client-go and the cluster were all on-pin and clean -- and then let a
locally modified runner, driver, projection, scenario or ledger produce the evidence.
A tweak to the projection can change the observed divergence; a tweak to a scenario
or the ledger can MANUFACTURE the declared one; and ledger-check would report success
throughout. Pinning what is MEASURED while leaving what MEASURES uncontrolled is not
a verified result. It is a verified subject and an unverified instrument.

provenance.sh has recorded harness.rev and harness.dirty since the beginning. Nothing
ever read them. Evidence must now come from a COMMITTED harness, and its revision is
recorded either way, so any claim can be reproduced against the exact instrument that
produced it. Iterating on the harness therefore cannot settle the ledger -- which is
the point, and is the same rule already applied to a dirty client-rust. Verified
against this very commit: with the tree dirty, a strict `make ledger` refuses.

DISTINCT RUN NAMES WERE NOT A DISTINCT COMPARISON. The previous pass rejected
compare: ["oracle","oracle"], but two runs can be named differently and still bind
every role to the SAME driver -- comparing all-Go against all-Go. The diff is empty
for a reason that has nothing to do with either client, and an `expect = "agrees"`
claim passes without client-rust ever being exercised. The self-comparison hole
wearing a different hat. The compared runs must now actually contrast something.

MALFORMED BASE64 DECODED TO EMPTY BYTES. `bytes()` cannot fail (it is called deep
inside the drivers), so `{"b64":"!!!"}` silently became "". Both drivers would then
agree on a prefix-only key or an empty value, the diff would be clean, and the ledger
would certify a scenario nobody wrote -- a green for a test that never happened. Every
byte-string argument is now validated at scenario LOAD, before a command is dispatched.

Also fixed, and it was mine: the binding check used `.expect("checked above")` on a
run lookup whose existence check actually lived BELOW it, so `compare: ["oracle",
"ghost"]` would have PANICKED instead of reporting a scenario error. The lookup now
reports. A comment that says "checked above" is worth exactly as much as the check
being above.

make ledger green end-to-end from a clean tree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Strict base64, a harness revision that must match, and no handshake s…
4366131 
…teps

Sixth codex pass. Three findings, all follow-ons from my own fixes.

BASE64 WAS PERMISSIVE, AND I HAND-ROLLED IT. "A" (a lone sextet), "AA=A" (padding in
the middle), "AB" (unpadded) and any non-zero residual bits all decoded to Some(empty
or truncated) instead of being rejected. So the KeyArg::validate added last pass --
whose entire purpose is to stop a scenario typo becoming a key nobody wrote -- waved
them straight through. Both drivers would then agree on that key, the diff would be
clean, and the ledger would certify a test that never happened.

Validation is now by ROUND TRIP, which is the cheapest way to be exactly right:
canonical base64 has precisely one encoding for any byte string, so if re-encoding the
decoded bytes does not reproduce the input character for character, the input was not
canonical. Quartet length, padding position, padding count and residual bits are all
covered by that single check, with no separate rules to get subtly wrong.

A CLEAN HARNESS IS NOT NECESSARILY *THIS* HARNESS. Last pass required harness.dirty ==
false. But results/ is gitignored, so artifacts survive a checkout -- and an OLD,
perfectly clean runner, projection or scenario could then be adjudicated against
TODAY's ledger. The instrument that produced the evidence and the instrument judging
it must be the same one, or pinning the instrument means nothing. ledger-check now
compares the recorded harness.rev against HEAD.

A `hello` STEP PASSED EVERY STATIC CHECK AND THEN ABORTED THE RUN. It deserializes as
a valid Command, so scenario validation accepted it -- but the driver answers a
handshake with a `hello`, not an observation, so execution died with "driver returned
no observation". A statically valid scenario must not be able to fail for a protocol
reason its author cannot see. Rejected at load. (The scenario tests were themselves
using {"op":"hello"} as a filler step, which is how it went unnoticed.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Keep multi-error cardinality, and judge contrast by what actually runs
ecc8abd 
Seventh codex pass.

THE MAPPING TABLE CONTRADICTED ITS OWN RULE. Its comment says a multi-key error must
never be flattened, because "a client that reports 1-of-3 where the other reports
3-of-3 is a VISIBLE difference" -- and then the homogeneous arm collapsed exactly
that. One write conflict and three write conflicts projected identically, so a client
silently dropping or duplicating per-key errors was invisible.

The collapse itself is REQUIRED and stays: Rust's MultipleKeyErrors([conflict]) has to
compare against Go's lone ErrWriteConflict, or every single-key conflict would diverge
on calling convention alone. What was missing is that the CARDINALITY should survive
it. Observation now carries error_count, and `errors.count` is an opt-in projection
path -- opt-in for the same reason native.type is: client-go's ExtractKeyErr surfaces
ONE error where client-rust surfaces all of them, so diffing counts by default would
flag every multi-key error as divergent for a reason that is about Go's calling
convention rather than about TiKV.

CONTRAST WAS JUDGED BY THE BINDING MAP, NOT BY WHAT EXECUTES. The previous pass
required the two compared runs to have different bindings. But the maps can differ on
a role that never executes anything: bind an unused `ghost` role to rust, route every
real step through go, and the maps differ while the comparison still contrasts nothing
-- an inevitable empty diff certifying an `agrees` claim without client-rust ever
running. Contrast is now computed over the roles that ACTUALLY EXECUTE STEPS, and a
binding for an undeclared role (the trick that made the maps differ) is rejected
outright.

batch_size: 0 HAD ASYMMETRIC HARNESS SEMANTICS. It deserializes happily, but the Rust
driver cannot page with it (driver_error -> the run is inadmissible) while the Go
driver ignores the hint and scans fine. The same protocol command must not mean two
different things to the two drivers; that is a divergence manufactured by us rather
than observed in a client. Rejected at scenario load, with the driver's guard kept as
defense in depth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Emit cardinality from both drivers, and keep the retry promise
19c45fe 
Eighth codex pass.

ONLY HALF THE PROTOCOL LEARNED ABOUT error_count. Last pass added it to the Rust
driver and not the Go one, so a claim opting into `errors.count` would compare Go's
`<absent>` against Rust's "1" and diverge even when both clients surfaced exactly ONE
error — a false divergence manufactured by an unset field. Worse, absent is
indistinguishable from "we did not look", which throws away the very fact the field
exists to state: client-go's ExtractKeyErr collapses N per-key errors into one, and
client-rust surfaces all of them. The Go driver now says so explicitly.

THE VERDICT AND THE ADMISSIBILITY CHECK DISAGREED ABOUT WHICH FILE THEY WERE JUDGING.
The runner names its artifacts after the scenario's JSON `name`; the admissibility
section read that, but the verdict section derived the path from the scenario's
FILENAME stem. The moment the two differ, a perfectly fresh result is reported MISSING.
Both halves now read the same field.

class.rs PROMISED A RETRY THAT NOTHING IMPLEMENTED. Its doc says a region error is "a
CLUSTER event, not client behaviour: a trace carrying one is inadmissible and the
runner retries the scenario". The runner did not — it failed the whole invocation, so
a region split or leader change under a run would fail CI as if a client had
misbehaved.

An inadmissible run now records WHY it was inadmissible, and only a cluster-movement
cause is retried (bounded, 3 attempts, loud when exhausted — a cluster that will not
settle is a real problem, and unlimited retries would just convert a broken cluster
into a slow one). Every other cause stays fatal, and deliberately so: an unmapped
error, a driver failure or a failed precondition is a genuine defect, not weather.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph @claude
Make the lock-kind map total, ignore obsolete artifacts, pin the Go c…
3715141 
…ompiler

Ninth codex pass.

THE LOCK-KIND MAP WAS NOT TOTAL, AND THE HARNESS INVENTED A DIVERGENCE. Go's switch
had no arm for Op_Rollback, so it rendered the operation as a bare "op_3" while Rust
named it "rollback". Both clients returned the IDENTICAL protobuf value; the
difference was manufactured entirely by us, in the one field (`locks[].kind`) that
findings 1 and 2 both turn on. The map is now total and agrees arm for arm, and the
`op_N` fallback is kept deliberately so that a proto which grows an operation makes
BOTH drivers say `op_N` and agree, rather than one of them guessing at a name.

ONE OBSOLETE ARTIFACT REFUSED EVERY FUTURE RUN. ledger-check globbed
results/divergence.*.json. results/ is gitignored, so a scenario that gets renamed or
deleted leaves its artifact behind -- and being stale it carries an older harness
revision, which the previous pass (correctly) made grounds for refusal. The result:
one dead file would wedge every subsequent `make ledger`, however fresh and valid the
real results were. The rule was technically right and practically useless.

A ledger entry names its scenario and the scenario names its artifact, so those are
the files judged. Nothing else on disk can speak.

A `go` DIRECTIVE IS A MINIMUM, NOT A PIN. Go builds happily with any newer release, so
go.mod does not imply which compiler produced the oracle -- and provenance recorded
`go version` without ever comparing it to pins.toml. A build done with a different
toolchain than the pin names was certified as coming from the pinned world. The Rust
side never had this hole (rust-toolchain.toml makes cargo FETCH the pinned channel);
nothing was doing the equivalent job for Go. Now checked, and refused under strict.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eduralph
eduralph merged commit cc7931b into main Jul 13, 2026
3 checks passed
@eduralph
eduralph deleted the parity-go-runner branch July 13, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@eduralph