Skip to content

docs: full README sweep — re-score the arena, correct the install path, document the hook + env vars - #117

Merged
askalf merged 1 commit into
masterfrom
docs/readme-full-sweep
Aug 1, 2026
Merged

docs: full README sweep — re-score the arena, correct the install path, document the hook + env vars#117
askalf merged 1 commit into
masterfrom
docs/readme-full-sweep

Conversation

@askalf

@askalf askalf commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Full sweep of the README and every doc it contradicts, verified claim by claim against the live repo rather than edited by eye. Three parallel audits (numbers, CLI/API surface, install/links) plus first-hand re-runs of the bench and the arena.

The big one: the arena scoreboard was scored on a corpus that no longer exists

arena/corpus.json grew to 291 samples / 25 families on Jul 27, but arena/RESULTS.md, arena/results.json, and arena/README.md still published figures computed on the retired 245-sample / 19-family corpus. The competitor comparison — the repo's strongest evidence — was stale.

Re-ran the arena workflow on master (run 30719172862, all live competitors pinned as usual) and committed the regenerated tables. redstamp scores better on the larger corpus, not worse:

published (245-corpus) re-scored (291-corpus)
recall (block) 96.5% 100.0%
precision 100% 100%
under-gate 1/35 1/44

Competitor rows moved too (Pipelock 7.0% → 6.1%, AEGIS 4.9%/59.4% → 4.2%/53.9%, mcp-firewall 9.8%/100% → 8.5%/100%).

README was understating the product

  • Bench: claimed "245-sample corpus across 19 attack families … 97% deterministic recall". Real, confirmed by running npm run bench here: 291 samples / 25 families, 165/165 blocked (100% recall), 0/82 over-flagged (100% precision).
  • The "remaining ~3% is the evasion bucket" paragraph described a gap that no longer exists (obfuscation family is 5/5). Replaced with what the residue actually is: 1 of 44 risky samples under-gates to allow.
  • ReDoS: claimed "every pattern under 1ms". The committed budget is 25 ms and the worst case moves run to run (1.14 ms / 1.51 ms on two machines). Now cites the deterministic budget instead of a number that rots.
  • Same stale sentence lived in SECURITY.md — fixed there too.

Install section was actively wrong as of today

  • "Not yet on npm" promised a future that will never arrive. Now states plainly that npm holds a deprecated pointer stub, why, and that npm i @askalf/redstamp gets the stub, not redstamp.
  • The first install command was the unverified one, and it wasn't global — so the redstamp / redstamp-mcp / redstamp-serve CLIs that the rest of the README assumes were never on PATH. The verify-first, -g route is now the primary block.
  • Git installs demoted and explicitly labelled no attestation.
  • INTEGRATING.md said "until @askalf/redstamp is published, depend on a path"; CONTRIBUTING.md said releases "publish to npm via OIDC trusted publishing". Both false — corrected to the signed-tarball reality. CHANGELOG.md gains an Unreleased entry recording the npm decision, since it is the fact most likely to be re-litigated.

Dead links and undocumented surface

  • Three of four framework example links were 404s: README pointed at examples/{langgraph,openai-agents,autogen}-redstamp while the directories were still *-warden. Renamed the directories (the last non-deliberate warden paths in the tree) and fixed the stale path comments and package names inside them. Deliberate legacy naming — the "Formerly warden" note, WARDEN_ env prefix, warden* CLI aliases — left untouched.
  • native/redstamp-fast doesn't exist; corrected to native/warden-fast.
  • A documented example verdict was simply wrong: '☠ pipe remote script to shell (RCE)' → the classifier actually emits '☠ pipe remote download to an interpreter (RCE)'.
  • redstamp-hook was never mentioned anywhere, despite two sections telling you to wire a Claude Code PreToolUse hook — the primary deployment shape had no documented command. Added a wiring section with the real settings.json block.
  • Added the missing CLI surface (init --global, check --policy, verify --audit, redstamp-serve --no-taint) and an environment-variable table — 13 real vars were undocumented, including WARDEN_TOKEN, which the README described in prose without ever naming. Every var in the table was grepped against source; none are invented.

Two code fixes found along the way

  • src/daemon-cli.mjs defaulted the judge to claude-sonnet-4-6 while src/judge.mjs defaults to claude-sonnet-5 — a library caller and a daemon user got different models on the same security decision. Unified on sonnet-5 (the measured choice). No production impact: the fleet sets WARDEN_JUDGE_MODEL explicitly.
  • package-lock.json was pinned at 0.7.2 while package.json is 0.7.3, so CI's npm ci ran against a lock a version behind. Realigned.

Verified

  • npm test294/294 pass.
  • npm run bench and npm run bench:redos run first-hand; every number in the README traced to its output.
  • Arena figures come from the CI re-run artifact, not hand-editing.

Not fixed here (flagged)

  • README line 19 cites OpenClaw "~180k stars … tens of thousands of instances exposed" with no in-repo source. Left as-is, but it wants a citation.
  • publish.yml and npm-drift.yml still carry "awaiting an npm content-scan allowlist" language; the allowlist was declined, so that premise is stale. Separate PR.

@askalf
askalf enabled auto-merge (squash) August 1, 2026 21:35

@sprayberry-reviewer sprayberry-reviewer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review from the Sprayberry Labs fleet code reviewer.

Verdict: No blocking issues — clean docs/data sweep, approving.

What I checked

  • Full diff (gh pr diff 117): README/SECURITY/CONTRIBUTING/INTEGRATING/CHANGELOG prose, arena/results.json + related JSON regeneration, four examples/*-wardenexamples/*-redstamp directory renames, and two one-line source fixes (src/daemon-cli.mjs, package-lock.json).
  • CI check-rollup (gh pr checks 117): all 7 checks pass, including arena (regeneration workflow itself green) and both Windows/Ubuntu test matrices.
  • Cross-referenced the two code fixes against source:
    • src/daemon-cli.mjs:22 now defaults to 'claude-sonnet-5', matching src/judge.mjs:32's model = 'claude-sonnet-5' default — the two callers were previously out of sync (claude-sonnet-4-6 vs claude-sonnet-5); this change unifies them correctly.
    • package-lock.json version bump 0.7.20.7.3 now matches package.json's "version": "0.7.3".
  • Spot-checked the example dir renames: examples/langgraph-redstamp/verify_audit.mjs and examples/autogen-redstamp/verify_audit.mjs comments were updated to reference the new path, and package.json names (langgraph-redstamp-example) were updated to match. redstamp-hook is a real bin entry (package.jsonsrc/cc-hook.mjs), so the new "Wiring the Claude Code hook" section documents a real, existing CLI.

Findings

None rise to blocking. This PR is pure documentation/data correction plus two small numeric alignments — no new logic paths, no control-flow changes, no new inputs to validate.

Minor (non-blocking): The PR body itself flags an unresolved citation gap (README line 19, "~180k stars" claim) and stale publish.yml/npm-drift.yml language as deliberately out of scope for this PR — consistent with what's in the diff, no action needed here.

What's good

  • The arena re-score is backed by a live CI run (arena check passes in this PR itself), not hand-edited numbers.
  • The two code fixes (judge model default, lockfile version) are genuine, verified inconsistencies caught incidentally and fixed correctly.
  • Renames are consistent — package names, in-file path comments, and the README's native/redstamp-fastnative/warden-fast correction all line up with the actual repo layout.

@askalf
askalf merged commit 5c5ccdc into master Aug 1, 2026
7 checks passed
@askalf
askalf deleted the docs/readme-full-sweep branch August 1, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants