feat(mutation): domain-specific operator packs via universalmutator (#141) - #143
Open
vincent-k2026 wants to merge 9 commits into
Open
feat(mutation): domain-specific operator packs via universalmutator (#141)#143vincent-k2026 wants to merge 9 commits into
vincent-k2026 wants to merge 9 commits into
Conversation
First implementation step of issue #141: regex mutation packs that complement the cargo-mutants gate with the semantic faults it cannot express. Three packs, scoped to the files whose invariants they guard: - trie: level-base and bit-width arithmetic of the two-tier trie (STARTING_NODE_ID off-by-one, 8/24/40 bit-width confusion, fanout shift direction, ceil-rounding in the subtree-root climb) - canon: deletion/reversal of sort+dedup canonicalization on the proof path - statement deletion is not in cargo-mutants' repertoire - endian: to/from_le_bytes flips in the consensus hash and the 12-byte metadata codec (the issue #127 bug class) scripts/spec_mutation.sh generates mutants restricted to production lines (never inside #[cfg(test)]), then classifies each as invalid, killed, or survived; survivors fail the run and are reported as diffs. The Spec Mutation workflow runs it on demand (workflow_dispatch) and uploads the report. Local generation across all packs currently yields 59 mutants (50 trie, 3 canon, 6 endian). Families C/D/F from the issue are staged for after the first survivor triage; known gaps are documented in mutants/operators/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mutation testing - PASSNothing to test: no viable mutants were generated. |
4 tasks
Performance Benchmark ComparisonCompared Detailed Comparison
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a85c5d47f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
universalmutator exits 0 when a file yields no mutants and nonzero only for real invocation errors (missing source, unreadable rules or lines file), so tolerating its exit code could silently produce a partial manifest and a false PASS. Verified empirically against 1.14.1; the earlier local failure this guarded against was actually the since-fixed grep pipeline, not mutate. Raised by review on PR #143. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Troublor
reviewed
Jul 9, 2026
flyq
reviewed
Jul 10, 2026
Address review feedback on PR #143: - endian.rules / trie.rules / canon.rules: anchor rules to call sites and add word boundaries so they can't match identifier substrings (`serialize_to_le_bytes`) or prefixes (`TRIE_WIDTH_BITS`); make the comparator-flip name-agnostic instead of hardcoding `a`/`b`. (Troublor) - spec_mutation.sh: drop comment-only mutants (universalmutator rewrites inside `///` docs → false SURVIVED), reported not silently discarded. (flyq #1) - spec_mutation.sh: cap lints to allow for the mutant build so a lint-only breakage (unused_mut under CI's -D warnings) isn't miscounted invalid. (flyq #2) - spec_mutation.sh: wrap nextest in a per-mutant timeout so a looping mutant is killed at 900s instead of hanging the job for hours. (flyq #3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c8889119a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Adopt the mega-evm spec-gate architecture per PR #143 review (Troublor): - Replace the regex packs + standalone spec_mutation.sh / spec-mutation.yml with the unified scripts/umutate.py driver (universalmutator --comby), scored by the SAME gate as cargo-mutants (scripts/mutation_gate.py) via the caught/missed/timeout/unviable results contract. - Express trie/canon/endian as comby structural packs under mutants/operators/<pack>/{manifest.toml,<pack>.rules}. Structural matching removes the regex-fragility class by construction: method/path-call templates can't hit an identifier substring, :[[word]] holes can't swallow a compound receiver; the one prefix-collision case (TRIE_WIDTH vs TRIE_WIDTH_BITS) is pinned with a boundary hole. - Fold everything into .github/workflows/mutation.yml: a diff-scoped blocking spec-gate on PRs (near-instant when no operator site changed), a nightly per-pack sweep, and an orphan-suppressions check that spans both engines. Verified locally with universalmutator 1.14.1 + comby 1.8.1: plan generates 57 mutants across the three packs (48 trie / 6 endian / 3 canon) with no identifier corruption and the TRIE_WIDTH boundary respected; a full endian run classified all 6 mutants as caught by the suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mutation testing - PASSNothing to test: no viable mutants were generated. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a879c9529
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Self-review fixes on the comby migration: - umutate.py: separate an unviable bucket. Comby operators produce genuinely non-compiling mutants (e.g. '<< BUCKET_SLOT_BITS ==> << BUCKET_ID_BITS' in a file that doesn't import BUCKET_ID_BITS), which the ported driver folded into 'caught' — inflating the score and regressing vs the original spec_mutation.sh invalid/killed split. Packs opt in with a 'build_cmd' (cargo check); a mutant that fails it is unviable and skips the test run. Verified: the BUCKET_ID_BITS mutant now classifies unviable while a viable '% (TRIE_WIDTH - 1)' mutant still compiles and is caught by a test. - mutation.yml: drop the redundant 'Warm test binaries' step from both spec-gate jobs (the lazy baseline/build already compile; it only wasted a full build on the common zero-mutant PR). cargo-mutants jobs keep theirs for copy_target seeding. Add a file-level header documenting the two engines + hygiene. - Reword stale 'spec-gate' phrasing (salt's packs are trie/canon/endian; the gate is the CI job) and document build_cmd in the operators README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58f305bbf9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
setup-rust-toolchain defaults RUSTFLAGS to "-D warnings". A valid, testable mutant can trip a warning — e.g. a canon sort-deletion leaves its `let mut` binding unused — which would fail the umutate build_cmd (`cargo check`) and be miscounted unviable instead of exercised. Set rustflags: "" on the spec-gate and spec-gate-sweep jobs so warnings stay non-fatal; genuine type/scope errors (the real unviable mutants) still fail. Verified: the entries.sort_unstable_by_key deletion now compiles and is caught by a test rather than dropped as unviable. (flyq #6 — mutants inside doc comments — needs no handling: comby is comment-aware and does not match tokens inside //, ///, //! by construction.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mutation.yml: pip -> pipx for universalmutator. ubuntu-24.04 runners block plain 'pip install' into the system env (PEP 668, runner-images#10897), which would fail the job before it produced results. - mutation.yml: add 'set -o pipefail' to the orphan-suppressions universe step. The default step shell lacks pipefail, so a failing 'umutate.py plan' piped into grep exited 0 and the orphan check would run on a partial universe. - endian pack: drop 'state/ahash/convert.rs' as a dead target. Its hash readers go through zerocopy::transmute! / .convert(), which these to/from_*_bytes rules can't match, so it generated zero mutants and overstated coverage. (The re-review also flagged unviable classification, already fixed in the prior commit via build_cmd.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
flyq
reviewed
Jul 13, 2026
vincent-k2026
pushed a commit
that referenced
this pull request
Jul 13, 2026
…e (flyq #143) Two config/doc inconsistencies from re-review, both resolved at the root in favour of the deliberate comby-only, line-independent design. 1. The manifest `mode` key was dead config. `generate_mutants` hardcodes `--comby` and nothing ever reads `pack["mode"]`; the comby migration (8a879c9) replaced the regex packs outright, so a `mode = "regex"` would silently do nothing. Remove the key from all three manifests and drop the `mode` / "regex mode" wording from the README (and a stale comment in umutate_setup.sh). `match` stays a regex file-content filter — that one is real and still used. 2. The triage policy advertised a `line = <n>` suppression pin that no schema field backs; `load_suppressions` ignores unknown keys and matching strips the locator on purpose. Implementing a line pin would fight the deliberate model (suppressions.toml matches by `(file, mutant-text)` precisely so they survive line shifts), so document the knobs that exist instead: a `kind = "line"` entry matched by locator-stripped text, and `kind = "function"` with a `pattern` regex. Verified: py_compile clean; load_packs loads all 3 packs with no `mode` key; `umutate.py plan --packs endian` still generates the 6 mutants; `mutation_gate.py report` suppresses a survivor by text regardless of line, and `orphans` matches across differing lines yet flags absent text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e (flyq #143) Two config/doc inconsistencies from re-review, both resolved at the root in favour of the deliberate comby-only, line-independent design. 1. The manifest `mode` key was dead config. `generate_mutants` hardcodes `--comby` and nothing ever reads `pack["mode"]`; the comby migration (8a879c9) replaced the regex packs outright, so a `mode = "regex"` would silently do nothing. Remove the key from all three manifests and drop the `mode` / "regex mode" wording from the README (and a stale comment in umutate_setup.sh). `match` stays a regex file-content filter — that one is real and still used. 2. The triage policy advertised a `line = <n>` suppression pin that no schema field backs; `load_suppressions` ignores unknown keys and matching strips the locator on purpose. Implementing a line pin would fight the deliberate model (suppressions.toml matches by `(file, mutant-text)` precisely so they survive line shifts), so document the knobs that exist instead: a `kind = "line"` entry matched by locator-stripped text, and `kind = "function"` with a `pattern` regex. Verified: py_compile clean; load_packs loads all 3 packs with no `mode` key; `umutate.py plan --packs endian` still generates the 6 mutants; `mutation_gate.py report` suppresses a survivor by text regardless of line, and `orphans` matches across differing lines yet flags absent text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vincent-k2026
force-pushed
the
krabat/feat/spec-mutation-operators
branch
from
July 13, 2026 07:49
d548145 to
b04648e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b04648eae0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
flyq
reviewed
Jul 16, 2026
…spec-gate subtrie.rs Address flyq/codex review on #143: - canon pack: add proof/subtrie.rs so its bucket_ids.dedup() is covered by the spec-gate (deleting that dedup can no longer pass with zero canon mutants) - umutate.py: give the baseline its own larger BASELINE_TIMEOUT and report a cold-build timeout distinctly from a real test failure (no more misleading 'fix the failing tests') - umutate.py: flush caught/missed/timeout/unviable after each source file so a job wall-clock timeout keeps partial results instead of uploading an empty dir - mutation.yml: warm test binaries in spec-gate and spec-gate-sweep so the baseline is not charged the workspace cold build
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First implementation step of #141 — the semantic mutation layer complementing the
cargo-mutantsgate:mutants/operators/{trie,canon,endian}.rules— families A, E and B from the issue, scoped per pack (the selection filter: what soundness or determinism invariant would a surviving mutant violate?). Generic operators are deliberately excluded; thecargo-mutantsgate owns those.scripts/spec_mutation.sh— generates mutants with universalmutator (--onlyour rules,--linesrestricted to production code so mutants never land in#[cfg(test)]), then classifies each as invalid / killed / survived. Survivors fail the run and are reported as diffs.Spec Mutationworkflow —workflow_dispatchwith apacksinput; uploadsreport.md+survivors.txtand writes the step summary.Validated locally
Generation across all three packs yields 59 mutants (50 trie, 3 canon, 6 endian), each confirmed to be exactly the intended semantic fault (spot-checked diffs:
STARTING_NODE_ID[level+1]→[level],>>MIN_BUCKET_SIZE_BITS→>>(MIN_BUCKET_SIZE_BITS+1),MAX_SUBTREE_LEVELS-1→-2, sort/dedup deletion,le↔beflips). The kill-analysis loop needs cargo and runs in CI (~a few hours for all packs; killed mutants stop at the first failing test).Notes
endianpack finds nothing inconvert.rsuntil fix(hasher): make the deterministic hasher endian-independent #142 lands (it currently usestransmute); after that it guards the new explicit-LE reads.gh workflow run "Spec Mutation"and triage the survivors — each is either a missing test or a documented equivalence.Refs #141.
🤖 Generated with Claude Code