docs: full README sweep — re-score the arena, correct the install path, document the hook + env vars - #117
Merged
Merged
Conversation
…l path, document the hook and env vars
askalf
enabled auto-merge (squash)
August 1, 2026 21:35
sprayberry-reviewer
approved these changes
Aug 1, 2026
sprayberry-reviewer
left a comment
Collaborator
There was a problem hiding this comment.
Automated review from the Sprayberry Labs fleet code reviewer.
Verdict: No blocking issues — clean docs/data sweep, approving.
What I checked
- Full diff (
gh pr diff 117): README/SECURITY/CONTRIBUTING/INTEGRATING/CHANGELOG prose,arena/results.json+ related JSON regeneration, fourexamples/*-warden→examples/*-redstampdirectory renames, and two one-line source fixes (src/daemon-cli.mjs,package-lock.json). - CI check-rollup (
gh pr checks 117): all 7 checks pass, includingarena(regeneration workflow itself green) and both Windows/Ubuntu test matrices. - Cross-referenced the two code fixes against source:
src/daemon-cli.mjs:22now defaults to'claude-sonnet-5', matchingsrc/judge.mjs:32'smodel = 'claude-sonnet-5'default — the two callers were previously out of sync (claude-sonnet-4-6vsclaude-sonnet-5); this change unifies them correctly.package-lock.jsonversion bump0.7.2→0.7.3now matchespackage.json's"version": "0.7.3".
- Spot-checked the example dir renames:
examples/langgraph-redstamp/verify_audit.mjsandexamples/autogen-redstamp/verify_audit.mjscomments were updated to reference the new path, andpackage.jsonnames (langgraph-redstamp-example) were updated to match.redstamp-hookis a real bin entry (package.json→src/cc-hook.mjs), so the new "Wiring the Claude Code hook" section documents a real, existing CLI.
Findings
None rise to blocking. This PR is pure documentation/data correction plus two small numeric alignments — no new logic paths, no control-flow changes, no new inputs to validate.
Minor (non-blocking): The PR body itself flags an unresolved citation gap (README line 19, "~180k stars" claim) and stale publish.yml/npm-drift.yml language as deliberately out of scope for this PR — consistent with what's in the diff, no action needed here.
What's good
- The arena re-score is backed by a live CI run (
arenacheck passes in this PR itself), not hand-edited numbers. - The two code fixes (judge model default, lockfile version) are genuine, verified inconsistencies caught incidentally and fixed correctly.
- Renames are consistent — package names, in-file path comments, and the README's
native/redstamp-fast→native/warden-fastcorrection all line up with the actual repo layout.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full sweep of the README and every doc it contradicts, verified claim by claim against the live repo rather than edited by eye. Three parallel audits (numbers, CLI/API surface, install/links) plus first-hand re-runs of the bench and the arena.
The big one: the arena scoreboard was scored on a corpus that no longer exists
arena/corpus.jsongrew to 291 samples / 25 families on Jul 27, butarena/RESULTS.md,arena/results.json, andarena/README.mdstill published figures computed on the retired 245-sample / 19-family corpus. The competitor comparison — the repo's strongest evidence — was stale.Re-ran the arena workflow on master (run 30719172862, all live competitors pinned as usual) and committed the regenerated tables. redstamp scores better on the larger corpus, not worse:
Competitor rows moved too (Pipelock 7.0% → 6.1%, AEGIS 4.9%/59.4% → 4.2%/53.9%, mcp-firewall 9.8%/100% → 8.5%/100%).
README was understating the product
npm run benchhere: 291 samples / 25 families, 165/165 blocked (100% recall), 0/82 over-flagged (100% precision).allow.SECURITY.md— fixed there too.Install section was actively wrong as of today
npm i @askalf/redstampgets the stub, not redstamp.redstamp/redstamp-mcp/redstamp-serveCLIs that the rest of the README assumes were never on PATH. The verify-first,-groute is now the primary block.INTEGRATING.mdsaid "until@askalf/redstampis published, depend on a path";CONTRIBUTING.mdsaid releases "publish to npm via OIDC trusted publishing". Both false — corrected to the signed-tarball reality.CHANGELOG.mdgains an Unreleased entry recording the npm decision, since it is the fact most likely to be re-litigated.Dead links and undocumented surface
examples/{langgraph,openai-agents,autogen}-redstampwhile the directories were still*-warden. Renamed the directories (the last non-deliberatewardenpaths in the tree) and fixed the stale path comments and package names inside them. Deliberate legacy naming — the "Formerly warden" note,WARDEN_env prefix,warden*CLI aliases — left untouched.native/redstamp-fastdoesn't exist; corrected tonative/warden-fast.'☠ pipe remote script to shell (RCE)'→ the classifier actually emits'☠ pipe remote download to an interpreter (RCE)'.redstamp-hookwas never mentioned anywhere, despite two sections telling you to wire a Claude Code PreToolUse hook — the primary deployment shape had no documented command. Added a wiring section with the realsettings.jsonblock.init --global,check --policy,verify --audit,redstamp-serve --no-taint) and an environment-variable table — 13 real vars were undocumented, includingWARDEN_TOKEN, which the README described in prose without ever naming. Every var in the table was grepped against source; none are invented.Two code fixes found along the way
src/daemon-cli.mjsdefaulted the judge toclaude-sonnet-4-6whilesrc/judge.mjsdefaults toclaude-sonnet-5— a library caller and a daemon user got different models on the same security decision. Unified on sonnet-5 (the measured choice). No production impact: the fleet setsWARDEN_JUDGE_MODELexplicitly.package-lock.jsonwas pinned at0.7.2whilepackage.jsonis0.7.3, so CI'snpm ciran against a lock a version behind. Realigned.Verified
npm test— 294/294 pass.npm run benchandnpm run bench:redosrun first-hand; every number in the README traced to its output.Not fixed here (flagged)
publish.ymlandnpm-drift.ymlstill carry "awaiting an npm content-scan allowlist" language; the allowlist was declined, so that premise is stale. Separate PR.