Skip to content

Add the attack-changes workflow: adversarial pass before a PR - #95

Merged
Pyronewbic merged 1 commit into
mainfrom
chore/attack-changes-workflow
Jul 21, 2026
Merged

Add the attack-changes workflow: adversarial pass before a PR#95
Pyronewbic merged 1 commit into
mainfrom
chore/attack-changes-workflow

Conversation

@Pyronewbic

Copy link
Copy Markdown
Owner

Makes reusable the pattern that found 16 real defects this week in changes that already had a green unit lane.

Why

Every one of those defects lived in an input nobody constructed — a label containing a newline or 0x1f, a broken hasher, a crafted cache file, a 2⁶³ timestamp. The unit lane passed throughout, because a green suite only covers the path you happened to exercise. The Linux VM would have caught approximately none of them either: it tests platform divergence (awk implementations, hashers, iptables) and is excellent at that, but these were failure-path and hostile-input defects. Different tool, different bug class.

Among what it surfaced: a fail-open freshness gate (a stale box reporting as current), _sha256 silently losing its error propagation in a refactor, a label encoding that emitted invalid JSON for an entire ls --json document, and an update cache that could forge an upgrade instruction telling the user what to run.

How it works

Scope the branch diff into attackable areas → one Attack agent per area, each required to produce a paste-able repro and to test the base so a pre-existing condition isn't reported as a regression → an independent Confirm skeptic re-runs every claim and drops what is NOT_REPRODUCIBLE or WORKS_AS_INTENDED.

"I could not break it" is an explicitly valid result, so it doesn't manufacture findings.

Traps it carries

Every agent is handed the failure modes that have actually produced false results in this repo:

  • Vacuous execution — a function behind an early-return guard may never run, so every assertion "passes". check_update_notice bails unless it can derive an X.Y.Z, and git describe --tags --always returns a bare SHA on a tagless checkout (which is what CI produces).
  • Swallowed output — a stderr tripwire never fires when production redirects stderr to /dev/null; the test then passes for the wrong reason.
  • $(...) disables errexit — a failure inside a command substitution is discarded, and an empty result compares equal to an empty expected value: a fail-open gate that looks fine.
  • Pipeline vs printf — a body ending in a pipeline propagates status under pipefail; one ending in printf always returns 0, so a refactor can silently delete error handling.

Verification

node --check parses, build-check + lint clean, test-unit 402/402. Docs-and-tooling only — no launcher slice is touched, so bin/sluice is unchanged.

…re a PR

Two ad-hoc rounds of this pattern found 16 real defects in changes that already had a green unit lane - a fail-open freshness gate, a hasher that stopped failing closed, a label encoding that emitted invalid JSON, and a cache that could forge an upgrade instruction. None were reachable from the happy path the tests exercised, and the Linux VM would have caught approximately none of them: it tests platform divergence, not hostile input.

The workflow scopes the branch diff into attackable areas, runs one attacker per area that must produce a paste-able repro, then has an independent skeptic re-run every claim and drop anything not reproducible or already present on the base. It carries the traps that have actually produced false results here: vacuous execution behind an early-return guard, tripwires written to a stream production discards, $( ) swallowing errexit, and a function body ending in printf instead of a pipeline silently losing its error handling.
@Pyronewbic
Pyronewbic enabled auto-merge (squash) July 21, 2026 04:14
@Pyronewbic
Pyronewbic merged commit 548a3e8 into main Jul 21, 2026
9 checks passed
@Pyronewbic
Pyronewbic deleted the chore/attack-changes-workflow branch July 21, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant