Add the attack-changes workflow: adversarial pass before a PR - #95
Merged
Conversation
…re a PR Two ad-hoc rounds of this pattern found 16 real defects in changes that already had a green unit lane - a fail-open freshness gate, a hasher that stopped failing closed, a label encoding that emitted invalid JSON, and a cache that could forge an upgrade instruction. None were reachable from the happy path the tests exercised, and the Linux VM would have caught approximately none of them: it tests platform divergence, not hostile input. The workflow scopes the branch diff into attackable areas, runs one attacker per area that must produce a paste-able repro, then has an independent skeptic re-run every claim and drop anything not reproducible or already present on the base. It carries the traps that have actually produced false results here: vacuous execution behind an early-return guard, tripwires written to a stream production discards, $( ) swallowing errexit, and a function body ending in printf instead of a pipeline silently losing its error handling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes reusable the pattern that found 16 real defects this week in changes that already had a green unit lane.
Why
Every one of those defects lived in an input nobody constructed — a label containing a newline or
0x1f, a broken hasher, a crafted cache file, a 2⁶³ timestamp. The unit lane passed throughout, because a green suite only covers the path you happened to exercise. The Linux VM would have caught approximately none of them either: it tests platform divergence (awk implementations, hashers, iptables) and is excellent at that, but these were failure-path and hostile-input defects. Different tool, different bug class.Among what it surfaced: a fail-open freshness gate (a stale box reporting as current),
_sha256silently losing its error propagation in a refactor, a label encoding that emitted invalid JSON for an entirels --jsondocument, and an update cache that could forge an upgrade instruction telling the user what to run.How it works
Scopethe branch diff into attackable areas → oneAttackagent per area, each required to produce a paste-able repro and to test the base so a pre-existing condition isn't reported as a regression → an independentConfirmskeptic re-runs every claim and drops what isNOT_REPRODUCIBLEorWORKS_AS_INTENDED."I could not break it" is an explicitly valid result, so it doesn't manufacture findings.
Traps it carries
Every agent is handed the failure modes that have actually produced false results in this repo:
check_update_noticebails unless it can derive an X.Y.Z, andgit describe --tags --alwaysreturns a bare SHA on a tagless checkout (which is what CI produces)./dev/null; the test then passes for the wrong reason.$(...)disables errexit — a failure inside a command substitution is discarded, and an empty result compares equal to an empty expected value: a fail-open gate that looks fine.pipefail; one ending inprintfalways returns 0, so a refactor can silently delete error handling.Verification
node --checkparses,build-check+lintclean,test-unit402/402. Docs-and-tooling only — no launcher slice is touched, sobin/sluiceis unchanged.